Hierarchical Testing

Hierarchical Testing of Ordered Trial Endpoints

Hierarchical testing is a common way for a confirmatory trial to claim more than one benefit. The endpoints are ranked in the protocol, each is tested at the full significance level in that order, and the first non-significant result closes the sequence, so nothing ranked below it can be claimed whatever its p-value. The method keeps the chance of any false claim of benefit at the planned level without dividing that level between endpoints. For health economists the hierarchy fixes which effects of a medicine count as demonstrated, while the economic model often needs endpoints, such as quality of life, that sit low in the order.

Why several endpoints need a shared error budget

A single hypothesis test at a significance level of 0.05 accepts a 5% risk of a type I error, rejecting a true null hypothesis. A trial that tests several endpoints, doses or populations has several chances to make that error. The EU guidance on multiplicity for joint clinical assessments, adopted by the HTA Coordination Group on 10 June 2024 under Regulation (EU) 2021/2282, gives the arithmetic for independent tests when every null hypothesis is true.

$$\text{FWER}_{\text{weak}} = 1 - (1 - \alpha)^{k}$$

where $\text{FWER}_{\text{weak}}$ is the probability of rejecting at least one of the $k$ null hypotheses when all of them are true, $\alpha$ is the significance level used for each test and $k$ is the number of independent tests. With $k = 3$ and $\alpha = 0.05$, 0.95 cubed is 0.857, and subtracting it from 1 leaves 0.143, nearly three times the nominal level.

The same guidance defines the family-wise error rate in the strong sense as the probability of wrongly rejecting at least one true null hypothesis, whichever and however many of the others are true, and notes that strong control implies weak control. In its section on trials with more than two arms, the 2002 EMA Points to Consider on multiplicity states, as a general rule, that strong control through closed test procedures is a minimal prerequisite for confirmatory claims. The ICH E9 guideline asks that any adjustment, or the reason none is needed, be set out in the analysis plan. Bonferroni-type methods split $\alpha$ across endpoints. Hierarchical testing spends it one endpoint at a time instead. Regulators sort endpoints into primary, secondary and exploratory families, each with its own role in a confirmatory trial.

The fixed-sequence rule and why it controls error

The FDA guidance Multiple Endpoints in Clinical Trials, issued in October 2022, describes the fixed-sequence procedure as testing endpoints in a predefined order, all at the same level $\alpha$, and moving to the next only after a success on the previous one. It requires two things, prospective specification of the sequence and no further testing once the sequence breaks. Written for $m$ hypotheses in their planned order, the rule is as follows.

$$H_k \ \text{is rejected} \iff p_j \le \alpha \ \text{for every} \ j = 1, \dots, k$$

where $H_k$ is the null hypothesis ranked $k$th in the protocol, $p_j$ is the p-value for the hypothesis ranked $j$th, $\alpha$ is the overall significance level and $k$ is any rank up to $m$.

The error control follows from the order. Suppose some null hypotheses are true and let $H_{k^*}$ be the first true one in the sequence. Every false rejection needs the procedure to reach and reject $H_{k^*}$, because nothing below it can be tested otherwise.

$$\text{FWER} = P(V \ge 1) \le P(p_{k^*} \le \alpha) \le \alpha$$

where $\text{FWER}$ is the strong family-wise error rate, $V$ is the number of true null hypotheses rejected, $P$ denotes probability and $p_{k^*}$ is the p-value of the first true null hypothesis in the sequence, which a valid test keeps at or below $\alpha$ with probability at most $\alpha$.

The EMA Points to Consider describes the same strategy for primary variables ranked by clinical relevance and says no reduction or splitting of $\alpha$ is necessary, adding that confidence intervals consistent with the procedure can be derived. It places these methods among the closed testing procedures that control the family-wise error rate. In the graphical form of the FDA guidance, the three hypotheses of a fixed sequence start with local levels of $\alpha$, 0 and 0, and each rejection passes the whole $\alpha$ along the arrow to the next hypothesis.

Choosing the order and what it costs

The order is a design decision with consequences for every later claim. The FDA guidance says endpoints are usually grouped hierarchically, often by clinical importance, while also weighing how often the endpoint events are expected and the anticipated drug effects. The EMA document allows a natural order, such as time or seriousness, or one that reflects the investigator's interests, but requires it to be prespecified in the protocol. The price falls on the type II error of the lower ranks. The EMA notes that type II errors are inflated for hypotheses with lower ranks, and the FDA guidance warns that with many secondary endpoints the important hypotheses further down a hierarchy might never get tested. Its own example has a first p-value of 0.59 and a second of 0.001, so the second endpoint cannot be declared significant, although a Bonferroni test would have shown an effect. The FDA adds that the best order often cannot be known in advance.

Some orders are logical. The FDA guidance notes that non-inferiority and then superiority on the same endpoint are naturally ordered, so a non-inferiority trial with one endpoint can test both without adjustment, while any further endpoint brings the multiplicity question back.

Gatekeeping, Holm and graphical procedures

Hierarchical testing is the simplest member of a wider family. The FDA guidance groups the Bonferroni, Holm and Hochberg procedures as methods that assume no hierarchy among the hypotheses, and contrasts them with methods that mix partial alpha allocation with hierarchies. It calls the Bonferroni and Holm procedures assumption-free, since they hold whatever the correlation between endpoints.

Gatekeeping procedures extend the ordering to families of hypotheses, usually primary and secondary objectives. Dmitrienko and Tamhane review serial, parallel and tree gatekeeping procedures. In the FDA account a serial gatekeeper opens the secondary family only if every primary endpoint succeeds, and a parallel gatekeeper opens it when at least one does. Bretz and colleagues describe gatekeeping, fixed-sequence and fallback procedures as sequentially rejective, weighted Bonferroni-type tests built on the closed test principle, and they represent each one as a directed, weighted graph. The graph makes a strategy that would otherwise need many intersection tests easier to communicate.

Three neighbouring designs answer different questions. Co-primary outcomes require every primary endpoint to succeed, so there is only one path to success. A composite endpoint merges events into one measure and avoids the multiple-endpoint problem. Alpha spending controls error over repeated looks at one endpoint during an interim analysis plan, and the two are often combined in a group sequential design.

Worked example with three oncology endpoints

All numbers in this example are illustrative. A randomised trial of a new cancer medicine prespecifies three two-sided tests at $\alpha = 0.05$ in this order, H1 progression-free survival, H2 overall survival and H3 time to deterioration in health-related quality of life.

1. Planning the chance of each claim. The marginal statistical power at the full level is 0.90 for H1, 0.70 for H2 and 0.80 for H3. Treating the three tests as independent, the probability of claiming H1 is 0.90, of claiming H2 is 0.90 × 0.70 = 0.63 and of claiming H3 is 0.63 × 0.80 = 0.504. Positive correlation between survival endpoints would usually make the later figures higher.

2. The Bonferroni alternative. Splitting $\alpha$ equally gives each test 0.05 / 3 = 0.0167, with a two-sided critical value of 2.394 in place of 1.960. For H1, power of 0.90 at 0.05 implies a standardised effect of 1.960 + 1.282 = 3.242, so power at the Bonferroni level is $\Phi$(3.242 minus 2.394) = $\Phi$(0.848) = 0.80. The same steps give 0.54 for H2 and 0.66 for H3. The hierarchy protects the primary endpoint and buys a higher chance of the overall survival claim (0.63 against 0.54), at the cost of a lower chance for the quality of life claim (0.50 against 0.66).

3. The results. The trial reports p = 0.004 for H1, p = 0.062 for H2 and p = 0.009 for H3.

4. Applying the hierarchy. H1 is rejected because 0.004 is at most 0.05. H2 is not rejected because 0.062 exceeds 0.05, so the sequence breaks. H3 cannot be claimed although its p-value of 0.009 is small.

5. Other procedures on the same data. Bonferroni rejects H1 and H3, since 0.004 and 0.009 are both at most 0.0167. Holm compares 0.004 with 0.0167, then 0.009 with 0.05 / 2 = 0.025, then 0.062 with 0.05, and also rejects H1 and H3. Had the protocol ranked quality of life second, the order H1, H3, H2 would have rejected H1 and H3, but choosing that order after seeing the results would void the error control.

6. Interpretation. Under the planned hierarchy only progression-free survival is a confirmed benefit. Overall survival and quality of life remain estimated effects with uncertainty, and the 0.009 is a nominal p-value. The quality of life data still inform the economic model, although a time-to-deterioration result does not itself supply the preference-based utility values the model needs.

What a broken hierarchy means for HTA and economic models

Regulators read the hierarchy as a list of permitted claims. The EMA document states that no confirmatory claims can be based on variables ranked lower than or equal to the first variable whose null hypothesis could not be rejected. It adds that a change in a secondary variable that is a direct consequence of the change in the primary variable cannot be a separate labelling claim.

HTA bodies use the hierarchy more as context. The EU guidance for joint clinical assessments requires assessors to report the alpha level of each test, whether comparison-wise or family-wise error control was applied, the multiplicity procedure if one was used and whether each analysis was prespecified. For outcomes under multiplicity control it asks for the first data cut at which the null hypothesis was rejected, if that differs from the latest prespecified cut. It also notes that multiplicity adjustment cannot be planned in a confirmatory sense in an evidence synthesis such as a network meta-analysis, because the data are already observed, so the only practical approach there is to take multiplicity into account when interpreting the results.

An economic evaluation works on a different logic. Claxton argued that decisions should rest on mean net benefit whether or not differences are statistically significant, with the distribution of net benefit relevant to whether more information is needed. On that view the model keeps the unconfirmed estimates with their full uncertainty in probabilistic sensitivity analysis, and an endpoint that fell below a broken link should not be set to zero effect. What changes is the weight a committee gives the evidence and the case for further research through value of information analysis.

Building the rule in a spreadsheet

Counting the p-values below 0.05 ignores the sequence and, in the example, would report two significant endpoints in place of one. With the p-values in their planned order in cells B2, B3 and B4, a correct column puts =B2<=0.05 in C2, =AND(C2,B3<=0.05) in C3 and =AND(C3,B4<=0.05) in C4, so each cell can be TRUE only if every cell above it is TRUE.

Misreadings and limits

Most errors come from treating the sequence as flexible or the nominal p-values as equal. Others arise when a regulatory testing rule is carried over into the economic decision, where it does not belong.

1. Reordering after the results. The order belongs in the protocol, and changing it once results are known removes the error control.
2. Claiming below a break. A small p-value after a failed test is nominal and supports no confirmatory claim.
3. Reading a missed claim as no effect. An endpoint that was never formally tested may still carry a real and decision-relevant effect.
4. Ignoring the power cost. Lower-ranked endpoints lose power when earlier ones fail, so trials should be sized with the whole sequence in mind.
5. Assuming the method covers the economic question. Error control protects regulatory claims and leaves the expected cost-effectiveness question to the decision model.

Sources

• Bretz F, Maurer W, Brannath W, Posch M. A graphical approach to sequentially rejective multiple test procedures. Statistics in Medicine. 2009;28(4):586-604. https://doi.org/10.1002/sim.3495
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/S0167-6296(98)00039-3
• Committee for Proprietary Medicinal Products. Points to consider on multiplicity issues in clinical trials (CPMP/EWP/908/99). European Medicines Agency; 19 September 2002. https://www.ema.europa.eu/en/documents/scientific-guideline/points-consider-multiplicity-issues-clinical-trials_en.pdf
• Dmitrienko A, Tamhane AC. Gatekeeping procedures with clinical trial applications. Pharmaceutical Statistics. 2007;6(3):171-180. https://doi.org/10.1002/pst.291
• Food and Drug Administration. Multiple Endpoints in Clinical Trials. Guidance for industry, October 2022. https://www.fda.gov/media/162416/download
• International Council for Harmonisation. ICH E9 Statistical principles for clinical trials. Step 4, 5 February 1998. Section 5.6. https://database.ich.org/sites/default/files/E9_Guideline.pdf
• Member State Coordination Group on Health Technology Assessment. Guidance on reporting requirements for multiplicity issues and subgroup, sensitivity and post hoc analyses in joint clinical assessments. Adopted 10 June 2024. https://health.ec.europa.eu/publications/guidance-reporting-requirements-multiplicity-issues-and-subgroup-sensitivity-and-post-hoc-analyses_en