Classification of multiple hypothesis tests

The following table defines the possible outcomes when testing multiple null hypotheses. Suppose we have a number m of null hypotheses, denoted by: H₁, H₂, ..., H_m. Using a statistical test, we reject the null hypothesis if the test is declared significant. We do not reject the null hypothesis if the test is non-significant. Summing each type of outcome over all H_i  yields the following random variables:

	Null hypothesis is true (H0)	Alternative hypothesis is true (HA)	Total
Test is declared significant	V	S	R
Test is declared non-significant	U	T	${\displaystyle m-R}$
Total	${\displaystyle m_{0}}$	${\displaystyle m-m_{0}}$	m

• m is the total number hypotheses tested
• ${\displaystyle m_{0}}$ is the number of true null hypotheses, an unknown parameter
• ${\displaystyle m-m_{0}}$ is the number of true alternative hypotheses
• V is the number of false positives (Type I error) (also called "false discoveries")
• S is the number of true positives (also called "true discoveries")
• T is the number of false negatives (Type II error)
• U is the number of true negatives
• ${\displaystyle R=V+S}$ is the number of rejected null hypotheses (also called "discoveries", either true or false)

In m hypothesis tests of which ${\displaystyle m_{0}}$ are true null hypotheses, R is an observable random variable, and S, T, U, and V are unobservable random variables.

The Bonferroni procedure

• Denote by ${\displaystyle p_{i}}$ the p-value for testing ${\displaystyle H_{i}}$
• reject ${\displaystyle H_{i}}$ if ${\displaystyle p_{i}\leq {\frac {\alpha }{m}}}$

The Šidák procedure

• Testing each hypothesis at level ${\displaystyle \alpha _{SID}=1-(1-\alpha )^{\frac {1}{m}}}$ is Sidak's multiple testing procedure.
• This procedure is more powerful than Bonferroni but the gain is small.
• This procedure can fail to control the FWER when the tests are negatively dependent.

Tukey's procedure

• Tukey's procedure is only applicable for pairwise comparisons.
• It assumes independence of the observations being tested, as well as equal variation across observations (homoscedasticity).
• The procedure calculates for each pair the studentized range statistic: ${\displaystyle {\frac {Y_{A}-Y_{B}}{SE}}}$ where ${\displaystyle Y_{A}}$ is the larger of the two means being compared, ${\displaystyle Y_{B}}$ is the smaller, and ${\displaystyle SE}$ is the standard error of the data in question.
• Tukey's test is essentially a Student's t-test, except that it corrects for family-wise error-rate.

Holm's step-down procedure (1979)

• Start by ordering the p-values (from lowest to highest) ${\displaystyle P_{(1)}\ldots P_{(m)}}$ and let the associated hypotheses be ${\displaystyle H_{(1)}\ldots H_{(m)}}$
• Let ${\displaystyle k}$ be the minimal index such that ${\displaystyle P_{(k)}>{\frac {\alpha }{m+1-k}}}$
• Reject the null hypotheses ${\displaystyle H_{(1)}\ldots H_{(k-1)}}$. If ${\displaystyle k=1}$ then none of the hypotheses are rejected.

This procedure is uniformly more powerful than the Bonferroni procedure.^[8] The reason why this procedure controls the family-wise error rate for all the m hypotheses at level α in the strong sense is, because it is a closed testing procedure. As such, each intersection is tested using the simple Bonferroni test.

Hochberg's step-up procedure

Hochberg's step-up procedure (1988) is performed using the following steps:^[9]

• Start by ordering the p-values (from lowest to highest) ${\displaystyle P_{(1)}\ldots P_{(m)}}$ and let the associated hypotheses be ${\displaystyle H_{(1)}\ldots H_{(m)}}$
• For a given ${\displaystyle \alpha }$, let ${\displaystyle R}$ be the largest ${\displaystyle k}$ such that ${\displaystyle P_{(k)}\leq {\frac {\alpha }{m-k+1}}}$
• Reject the null hypotheses ${\displaystyle H_{(1)}\ldots H_{(R)}}$

Hochberg's procedure is more powerful than Holms'. Nevertheless, while Holm’s is a closed testing procedure (and thus, like Bonferroni, has no restriction on the joint distribution of the test statistics), Hochberg’s is based on the Simes test, so it holds only under non-negative dependence.

Dunnett's correction

Charles Dunnett (1955, 1966) described an alternative alpha error adjustment when k groups are compared to the same control group. Now known as Dunnett's test, this method is less conservative than the Bonferroni adjustment.

Scheffé's method

Resampling procedures

The procedures of Bonferroni and Holm control the FWER under any dependence structure of the p-values (or equivalently the individual test statistics). Essentially, this is achieved by accommodating a `worst-case' dependence structure (which is close to independence for most practical purposes). But such an approach is conservative if dependence is actually positive. To give an extreme example, under perfect positive dependence, there is effectively only one test and thus, the FWER is uninflated.

Accounting for the dependence structure of the p-values (or of the individual test statistics) produces more powerful procedures. This can be achieved by applying resampling methods, such as bootstrapping and permutations methods. The procedure of Westfall and Young (1993) requires a certain condition that does not always hold in practice (namely, subset pivotality).^[10] The procedures of Romano and Wolf (2005a,b) dispense with this condition and are thus more generally valid.^[11][12]

Harmonic mean p-value procedure

The harmonic mean p-value (HMP) procedure^[13][14] provides a multilevel test that improves on the power of Bonferroni correction by assessing the significance of groups of hypotheses while controlling the strong-sense family-wise error rate. The significance of any subset ${\textstyle {\mathcal {R}}}$ of the ${\textstyle m}$ tests is assessed by calculating the HMP for the subset,

$${\displaystyle {\overset {\circ }{p}}_{\mathcal {R}}={\frac {\sum _{i\in {\mathcal {R}}}w_{i}}{\sum _{i\in {\mathcal {R}}}w_{i}/p_{i}}},}$$

where ${\textstyle w_{1},\dots ,w_{m}}$ are weights that sum to one (i.e. ${\textstyle \sum _{i=1}^{m}w_{i}=1}$). An approximate procedure that controls the strong-sense family-wise error rate at level approximately ${\textstyle \alpha }$ rejects the null hypothesis that none of the p-values in subset ${\textstyle {\mathcal {R}}}$ are significant when ${\textstyle {\overset {\circ }{p}}_{\mathcal {R}}\leq \alpha \,w_{\mathcal {R}}}$^[15] (where ${\textstyle w_{\mathcal {R}}=\sum _{i\in {\mathcal {R}}}w_{i}}$). This approximation is reasonable for small ${\textstyle \alpha }$ (e.g. ${\textstyle \alpha <0.05}$) and becomes arbitrarily good as ${\textstyle \alpha }$ approaches zero. An asymptotically exact test is also available (see main article).