Hierarchical Testing

A multiple testing strategy organising hypotheses into a prespecified sequence, so later tests are only conducted if earlier, higher-priority tests succeed.

Source: Dmitrienko, Tamhane & Bretz 2009

Hierarchical testing arranges a trial's hypotheses into a fixed sequence set in advance, testing them one after another and stopping as soon as one fails. Ordering them this way controls the overall risk of a false positive without splitting the significance level across them, because each test is reached only if every earlier, higher-priority one succeeded. This lets several endpoints be claimed in turn while keeping the family-wise error rate in check. Testing hypotheses in a preset order is its method. Friedman and colleagues (2015) describe this strategy.

Source: Friedman et al. 2015

Hierarchical testing controls the overall type I error by fixing the order of hypotheses in advance and testing each only if all earlier ones were rejected, so that the full significance level can be used at each step without inflating the overall error. Because later tests occur only after earlier successes, no adjustment to the individual significance levels is needed. So hierarchical testing controls the error rate through the prespecified sequence and the stopping rule, which ensures that the chance of any false positive across the sequence stays at the intended level, allowing each hypothesis to be tested at the full level while preserving overall control.

Source: Dmitrienko, Tamhane & Bretz 2009

Hierarchical testing tests hypotheses in a strict prespecified sequence, each tested only if the previous was significant, while gatekeeping generalises this to ordered families of hypotheses with gates between them, offering more flexibility in how error is allocated across the families. Hierarchical testing is essentially the simplest gatekeeping structure. So hierarchical testing and gatekeeping are related strategies, with hierarchical, or fixed-sequence, testing a linear ordering of individual hypotheses and gatekeeping a broader framework of ordered families, both using prespecified priorities to control multiplicity, and the choice depends on whether a simple sequence or a more complex structure of hypothesis families is required.

Source: Dmitrienko, Tamhane & Bretz 2009

Hierarchical testing is used when hypotheses can be ranked by importance in advance and it is acceptable to test lower-priority ones only if higher-priority ones succeed, as with a primary endpoint followed by ordered secondary endpoints in a trial. So hierarchical testing is used to structure the testing of multiple endpoints or hypotheses by priority, allowing confirmatory conclusions on secondary hypotheses when the primary ones are significant, which is common in clinical trials, since it controls multiplicity while reflecting the relative importance of the hypotheses and avoids the loss of power that adjusting all tests equally would cause.

Source: Dmitrienko, Tamhane & Bretz 2009

The limitations of hierarchical testing include that once a hypothesis in the sequence fails to reach significance, testing stops and all later hypotheses cannot be formally concluded, even if their evidence is strong; and that the ordering must be fixed in advance and cannot be changed by the data. So hierarchical testing is used with awareness that its strict sequence can prevent conclusions on later hypotheses if an earlier one fails, regardless of their own results, which is a trade-off for its simplicity and error control, and this is why the ordering is chosen carefully in advance and more flexible gatekeeping structures are sometimes preferred when several hypotheses matter.

Source: Dmitrienko, Tamhane & Bretz 2009