Classification of multiple hypothesis tests

The following table defines the possible outcomes when testing multiple null hypotheses. Suppose we have a number m of null hypotheses, denoted by: H₁, H₂, ..., H_m. Using a statistical test, we reject the null hypothesis if the test is declared significant. We do not reject the null hypothesis if the test is non-significant. Summing each type of outcome over all H_i  yields the following random variables:

	Null hypothesis is true (H0)	Alternative hypothesis is true (HA)	Total
Test is declared significant	V	S	R
Test is declared non-significant	U	T	${\displaystyle m-R}$
Total	${\displaystyle m_{0}}$	${\displaystyle m-m_{0}}$	m

• m is the total number hypotheses tested
• ${\displaystyle m_{0}}$ is the number of true null hypotheses, an unknown parameter
• ${\displaystyle m-m_{0}}$ is the number of true alternative hypotheses
• V is the number of false positives (Type I error) (also called "false discoveries")
• S is the number of true positives (also called "true discoveries")
• T is the number of false negatives (Type II error)
• U is the number of true negatives
• ${\displaystyle R=V+S}$ is the number of rejected null hypotheses (also called "discoveries", either true or false)

In m hypothesis tests of which ${\displaystyle m_{0}}$ are true null hypotheses, R is an observable random variable, and S, T, U, and V are unobservable random variables.

Multiple testing correction

Multiple testing correction refers to making statistical tests more stringent in order to counteract the problem of multiple testing. The best known such adjustment is the Bonferroni correction, but other methods have been developed. Such methods are typically designed to control the family-wise error rate or the false discovery rate.

If m independent comparisons are performed, the family-wise error rate (FWER), is given by

${\displaystyle {\bar {\alpha }}=1-\left(1-\alpha _{\{{\text{per comparison}}\}}\right)^{m}.}$

Hence, unless the tests are perfectly positively dependent (i.e., identical), ${\displaystyle {\bar {\alpha }}}$ increases as the number of comparisons increases. If we do not assume that the comparisons are independent, then we can still say:

${\displaystyle {\bar {\alpha }}\leq m\cdot \alpha _{\{{\text{per comparison}}\}},}$

which follows from Boole's inequality. Example: ${\displaystyle 0.2649=1-(1-.05)^{6}\leq .05\times 6=0.3}$

There are different ways to assure that the family-wise error rate is at most ${\displaystyle \alpha }$. The most conservative method, which is free of dependence and distributional assumptions, is the Bonferroni correction ${\displaystyle \alpha _{\mathrm {\{per\ comparison\}} }={\alpha }/m}$. A marginally less conservative correction can be obtained by solving the equation for the family-wise error rate of ${\displaystyle m}$ independent comparisons for ${\displaystyle \alpha _{\mathrm {\{per\ comparison\}} }}$. This yields ${\displaystyle \alpha _{\{{\text{per comparison}}\}}=1-{(1-{\alpha })}^{1/m}}$, which is known as the Šidák correction. Another procedure is the Holm–Bonferroni method, which uniformly delivers more power than the simple Bonferroni correction, by testing only the lowest p-value (${\displaystyle i=1}$) against the strictest criterion, and the higher p-values (${\displaystyle i>1}$) against progressively less strict criteria.^[5] ${\displaystyle \alpha _{\mathrm {\{per\ comparison\}} }={\alpha }/(m-i+1)}$.

For continuous problems, one can employ Bayesian logic to compute ${\displaystyle m}$ from the prior-to-posterior volume ratio. Continuous generalizations of the Bonferroni and Šidák correction are presented in.^[6]

Assessing whether any alternative hypotheses are true

A normal quantile plot for a simulated set of test statistics that have been standardized to be Z-scores under the null hypothesis. The departure of the upper tail of the distribution from the expected trend along the diagonal is due to the presence of substantially more large test statistic values than would be expected if all null hypotheses were true. The red point corresponds to the fourth largest observed test statistic, which is 3.13, versus an expected value of 2.06. The blue point corresponds to the fifth smallest test statistic, which is -1.75, versus an expected value of -1.96. The graph suggests that it is unlikely that all the null hypotheses are true, and that most or all instances of a true alternative hypothesis result from deviations in the positive direction.

A basic question faced at the outset of analyzing a large set of testing results is whether there is evidence that any of the alternative hypotheses are true. One simple meta-test that can be applied when it is assumed that the tests are independent of each other is to use the Poisson distribution as a model for the number of significant results at a given level α that would be found when all null hypotheses are true. If the observed number of positives is substantially greater than what should be expected, this suggests that there are likely to be some true positives among the significant results.

For example, if 1000 independent tests are performed, each at level α = 0.05, we expect 0.05 × 1000 = 50 significant tests to occur when all null hypotheses are true. Based on the Poisson distribution with mean 50, the probability of observing more than 61 significant tests is less than 0.05, so if more than 61 significant results are observed, it is very likely that some of them correspond to situations where the alternative hypothesis holds. A drawback of this approach is that it overstates the evidence that some of the alternative hypotheses are true when the test statistics are positively correlated, which commonly occurs in practice. . On the other hand, the approach remains valid even in the presence of correlation among the test statistics, as long as the Poisson distribution can be shown to provide a good approximation for the number of significant results. This scenario arises, for instance, when mining significant frequent itemsets from transactional datasets. Furthermore, a careful two stage analysis can bound the FDR at a pre-specified level.^[14]

Another common approach that can be used in situations where the test statistics can be standardized to Z-scores is to make a normal quantile plot of the test statistics. If the observed quantiles are markedly more dispersed than the normal quantiles, this suggests that some of the significant results may be true positives.