Multiple Comparison Problem

In a multiple comparisons problem, we deal with multiple statistical tests simultaneously.

Examples

We see such problems a lot in IT companies. Suppose we have a website and would like to test if a new design of a button can lead to some changes in five different KPIs (e.g., view-to-click rate, click-to-book rate, …).

In multi-horizon time series forecasting, we sometimes choose to forecast multiple future data points in one shot. To properly find the confidence intervals of our predictions, one approach is the so called conformal prediction method. This becomes a multiple comparisons problem because we have to tell if we can reject at least one true null hypothesis. In a paper by Zeni et al¹, the confidence interval has to be adjusted due to a specific definition of type I error for the family of hypotheses [[FWER]] Bonferroni Correction Bonferroni correction is very useful in a multiple comparison problem .