Multiple Comparison Problem
In a multiple comparisons problem, we deal with multiple statistical tests simultaneously.
We see such problems a lot in IT companies. Suppose we have a website and would like to test if a new design of a button can lead to some changes in five different KPIs (e.g., view-to-click rate, click-to-book rate, …).
In multi-horizon time series forecasting, we sometimes choose to forecast multiple future data points in one shot. To properly find the confidence intervals of our predictions, one approach is the so called conformal prediction method. This becomes a multiple comparisons problem because we have to tell if we can reject at least one true null hypothesis. In a paper by Zeni et al1, the confidence interval has to be adjusted due to a specific definition of type I error for the family of hypotheses [[FWER]] Bonferroni Correction Bonferroni correction is very useful in a multiple comparison problem .
- Contributors to Wikimedia projects. Multiple comparisons problem. In: Wikipedia [Internet]. 11 Apr 2022 [cited 18 Apr 2022]. Available: https://en.wikipedia.org/wiki/Multiple_comparisons_problem
- Zeni2020 Zeni G, Fontana M, Vantini S. Conformal Prediction: a Unified Review of Theory and New Challenges. arXiv [cs.LG]. 2020. Available: http://arxiv.org/abs/2005.07972
L Ma (2020). 'Multiple Comparison Problem', Datumorphism, 04 April. Available at: https://datumorphism.leima.is/cards/statistics/multiple-comparison-problem/.