In a single hypothesis testing problem, we the [[type I error]] Types of Errors in Statistical Hypothesis Testing We all make mistakes. The question is, what kind of mistakes. : Rejecting the one null hypothesis when it is actually true. Given a threshold $\alpha$, we can find out the interval $\Gamma$ that leads to a probability of rejecting the hypothesis $p\leq\alpha$ (single-sided).
In a [[multiple comparisons problem]] Multiple Comparison Problem In a multiple comparisons problem, we deal with multiple statistical tests simultaneously. Examples We see such problems a lot in IT companies. Suppose we have a website and would like to test if a new design of a button can lead to some changes in five different KPIs (e.g., view-to-click rate, click-to-book rate, …). In multi-horizon time series forecasting, we sometimes choose to forecast multiple future data points in one shot. To properly find the confidence intervals of our … , type I error is ambiguous. Suppose we have $m$ hypotheses $H_1, \cdots, H_i, H_m$. If we define the type I error as committing at least one type I error, this is the FamilyWise Error Rate (FWER).
In such a problem, we look for intervals $\Gamma^\alpha_i$ that lead to $p_i\leq \alpha/m$, where $m$ is the number of hypotheses. These intervals will together make sure our jointed type I error greater than $\alpha$.
- Contributors to Wikimedia projects. Bonferroni correction. In: Wikipedia [Internet]. 22 Feb 2022 [cited 18 Apr 2022]. Available: https://en.wikipedia.org/wiki/Bonferroni_correction
- Contributors to Wikimedia projects. Multiple comparisons problem. In: Wikipedia [Internet]. 11 Apr 2022 [cited 18 Apr 2022]. Available: https://en.wikipedia.org/wiki/Multiple_comparisons_problem
L Ma (2020). 'Bonferroni Correction', Datumorphism, 04 April. Available at: https://datumorphism.leima.is/cards/statistics/bonferroni-correction/.