Bonferroni Correction

In a single hypothesis testing problem, we the [[type I error]] Types of Errors in Statistical Hypothesis Testing We all make mistakes. The question is, what kind of mistakes. : Rejecting the one null hypothesis when it is actually true. Given a threshold $\alpha$, we can find out the interval $\Gamma$ that leads to a probability of rejecting the hypothesis $p\leq\alpha$ (single-sided).

In a [[multiple comparisons problem]] Multiple Comparison Problem In a multiple comparisons problem, we deal with multiple statistical tests simultaneously. Examples We see such problems a lot in IT companies. Suppose we have a website and would like to test if a new design of a button can lead to some changes in five different KPIs (e.g., view-to-click rate, click-to-book rate, …). In multi-horizon time series forecasting, we sometimes choose to forecast multiple future data points in one shot. To properly find the confidence intervals of our … , type I error is ambiguous. Suppose we have $m$ hypotheses $H_1, \cdots, H_i, H_m$. If we define the type I error as committing at least one type I error, this is the FamilyWise Error Rate (FWER).

In such a problem, we look for intervals $\Gamma^\alpha_i$ that lead to $p_i\leq \alpha/m$, where $m$ is the number of hypotheses. These intervals will together make sure our jointed type I error greater than $\alpha$.