Alpha Spending

Alpha Spending Functions, Early Stopping and the Economic Model

Alpha spending, also called error spending, lets a randomised trial look at its accumulating results several times while holding the overall chance of a false claim of benefit at the level set when the trial was designed. Health economists usually meet it at the end of the chain, when a confirmatory trial planned as a group sequential design stops at an interim look, and the health technology assessment inherits a smaller dataset, shorter follow-up and an effect estimate that tends to be too large. A spending function turns one error budget into a set of stopping boundaries, and the Lan-DeMets O'Brien-Fleming type and Pocock type functions spend that budget at different rates, which an illustrative overall survival example compares. Early stopping also has consequences for effect estimates, for immature survival data and for the economic evaluation built on them.

Why repeated looks need an error budget

A fixed design tests the treatment effect once, at a chosen significance level. Each interim analysis adds another chance to reject a true null hypothesis, so testing every look at the conventional level raises the overall probability of a type I error. The FDA guidance on adaptive designs says so for repeated testing at the conventional one-sided 0.025 level, and ICH E9 states that the procedures chosen should always keep the overall probability of type I error controlled.

If a trial with four equally spaced looks tested each one at a one-sided 0.025 level (a critical value of 1.96), the overall false positive probability, computed numerically from the joint distribution of the four test statistics described below, would be about 0.063 rather than 0.025. For a health system, a false positive at this stage means adopting and paying for a technology that brings no health gain, with the displaced care as its opportunity cost.

Spending functions and the information clock

The spending approach measures trial progress on an information scale rather than in calendar time. The information fraction $t$ is the information available at an analysis divided by the information expected at the planned end; for a comparison of means it is the number of patients analysed over the target sample size, and for survival outcomes it is approximated by the number of deaths observed over the number expected by the end of the trial, $D$, which itself has to be estimated while the trial is running. A spending function $\alpha(t)$ rises from zero to the overall significance level as information accrues.

$$\alpha(0) = 0, \quad \alpha(1) = \alpha, \quad t_k = \frac{I_k}{I_{max}}$$

where $\alpha(t)$ is the cumulative type I error that may have been used by information fraction $t$, $\alpha$ is the overall one-sided significance level, $I_k$ is the statistical information at analysis $k$ and $I_{max}$ is the information planned for the final analysis.

The boundary $c_k$ for analysis $k$ is then set so that the probability, under the null hypothesis, of crossing any boundary up to and including analysis $k$ equals the cumulative spend.

$$P_0\left(Z_1 \ge c_1 \ \text{or} \ \dots \ \text{or} \ Z_k \ge c_k\right) = \alpha(t_k), \qquad \text{Corr}(Z_l, Z_k) = \sqrt{t_l / t_k} \ \text{for} \ l < k$$

where $Z_k$ is the standardised test statistic using all data up to analysis $k$, $c_k$ is its critical value, $P_0$ denotes probability when there is no treatment effect, and $t_l$ and $t_k$ are the information fractions at analyses $l$ and $k$. The increment $\alpha(t_k) - \alpha(t_{k-1})$ is the extra error available at analysis $k$. Because each boundary depends only on the current and earlier information fractions, neither the number of analyses nor their timing has to be fixed in advance.

DeMets and Lan trace the method to the Beta-Blocker Heart Attack Trial, whose monitoring board met on calendar dates rather than at equal numbers of events. The FDA guidance warns that choosing analysis times in response to accumulating results can inflate the type I error, and asks for a targeted number of analyses, an approximate schedule and a decision framework based on information independent of the treatment effect. ICH E9 asks the protocol to describe the schedule of interim analyses or the considerations governing it, naming flexible alpha spending function approaches as an example.

O'Brien-Fleming type and Pocock type spending

Lan and DeMets proposed two specific functions, one to approximate the O'Brien-Fleming boundary and one to approximate the Pocock boundary. Each is written so that it spends exactly the overall significance level by the end of the trial, whatever the number of looks. The O'Brien-Fleming type function is defined as follows.

$$\alpha_{OF}(t) = 2 - 2\Phi\left(\frac{\Phi^{-1}(1 - \alpha/2)}{\sqrt{t}}\right)$$

where $\alpha_{OF}(t)$ is the cumulative error spent by information fraction $t$, $\Phi$ is the standard normal cumulative distribution function, $\Phi^{-1}$ is its inverse and $\alpha$ is the overall one-sided significance level. The Pocock type function is defined as follows.

$$\alpha_{P}(t) = \alpha \ln\left(1 + (e - 1)t\right)$$

where $\alpha_{P}(t)$ is the cumulative error spent by information fraction $t$, $\ln$ is the natural logarithm and $e$ is the base of natural logarithms. Both equal $\alpha$ at $t = 1$.

The two shapes trade early stopping against final power. The O'Brien-Fleming type spends almost nothing early, so stopping at an early look needs very persuasive evidence, and the final critical value stays close to the fixed-design value, so the maximum sample size rises little. The Pocock type spends more evenly, gives nearly constant boundaries and a higher chance of stopping early, but its final critical value is well above the fixed-design value, which needs a larger maximum sample size to keep the planned statistical power. The gsDesign technical manual reports that the Pocock type function reproduces the original Pocock bounds closely, while the O'Brien-Fleming type approximation is less exact.

Worked example with four looks at overall survival

The figures below are illustrative. A two-arm trial with 1:1 randomisation compares a new cancer drug with standard care on overall survival, tested with the log-rank test at a one-sided 0.025 level. The final analysis is planned at 400 deaths, with interim efficacy analyses at 100, 200 and 300 deaths, so the information fractions are 0.25, 0.50, 0.75 and 1.

1. Cumulative error at the first look. For the O'Brien-Fleming type function, $\Phi^{-1}(1 - 0.0125) = 2.2414$, divided by $\sqrt{0.25} = 0.5$ gives 4.4828, and the cumulative spend $\alpha_{OF}(0.25) = 2 - 2\Phi(4.4828)$ is about 0.0000074. For the Pocock type function, $\ln(1 + (e - 1) \times 0.25) = \ln(1.42957) = 0.35737$, so $\alpha_{P}(0.25) = 0.025 \times 0.35737 = 0.008934$. With only one analysis so far, the first boundary is the normal quantile for that error, which is 4.333 for the O'Brien-Fleming type and 2.368 for the Pocock type.

2. Later boundaries. Boundaries at looks 2 to 4 are solved numerically from the joint normal distribution above; the Pocock type values match those printed in the gsDesign technical manual.

3. Boundaries as hazard ratios. With equal allocation the variance of the estimated log hazard ratio is approximately four divided by the number of deaths, so crossing boundary $c_k$ at $d_k$ deaths requires an observed hazard ratio no larger than the following.

$$\widehat{HR}^{*}_{k} = \exp\left(-\frac{2 c_k}{\sqrt{d_k}}\right)$$

where $\widehat{HR}^{*}_{k}$ is the largest observed hazard ratio that crosses the efficacy boundary at analysis $k$, $c_k$ is the critical value and $d_k$ is the number of deaths. At look 2 under the O'Brien-Fleming type function the threshold is as follows.

$$\widehat{HR}^{*}_{2} = \exp\left(-\frac{2 \times 2.963}{\sqrt{200}}\right) = \exp(-0.4190) = 0.658$$

where 2.963 is the look 2 critical value and 200 is the number of deaths at that look. This conversion assumes proportional hazards; with a delayed effect the hazard ratio thresholds, and the chance of stopping at early looks, can mislead, although type I error control is unaffected.

4. Interpretation. Under the O'Brien-Fleming type plan the trial stops at 100 deaths only if the observed hazard ratio is 0.42 or lower, and the final critical value of 2.014 is close to the 1.96 of a fixed design. Under the Pocock type plan early stopping is far easier, but the final hurdle of 2.350 is much higher. If the true hazard ratio were 0.80, the O'Brien-Fleming type plan stops by look 2 in about 8% of trials and the Pocock type plan in about 25%, and every one of those early stops reports a hazard ratio of about 0.715 or lower, well below the true 0.80. A survival model fed that estimate would overstate the gain in QALYs, and its survival extrapolation would rest on half or fewer of the planned deaths.

Effect estimates from trials that stop early

Alpha spending controls the probability of a false positive. It does not make the estimated effect at the stopping point unbiased. The FDA guidance states that conventional estimates tend to be biased towards greater effects than the true value when a group sequential design is used, that confidence intervals do not have their nominal coverage, and that methods adjusting for the stopping rule should be planned in advance and used for reporting.

Empirical studies point the same way. Bassler and colleagues compared 91 trials stopped early for benefit with 424 matching trials that were not, and found a pooled ratio of relative risks of 0.71 (95% confidence interval 0.65 to 0.77). Relative risks in the truncated trials were on average about 29% lower, a larger apparent benefit, than in comparable trials not stopped early. The difference did not depend on whether a statistical stopping rule was used, and large overestimates occurred in truncated trials with fewer than 500 events. An earlier review by Montori and colleagues found that 129 of 143 trials stopped early for benefit did not report an analysis adjusted for interim monitoring and truncation.

Adjusted methods exist within the spending framework. DeMets and Lan describe confidence intervals built on an ordering of the possible stopping outcomes, which give coverage closer to the nominal level than naive intervals, and repeated confidence intervals that use the boundary critical values. An appraisal can ask whether such estimates were reported, how many events had occurred at stopping, and whether the model uses the adjusted or the naive estimate. Economic analyses of adaptive trials often leave this unclear, and a review by Flight and colleagues of 37 trials with an adaptive design and a health economic analysis found that in 17 it was unclear whether the economic analysis accounted for the design where this was thought necessary.

Immature survival data and the economic model

Early stopping changes what the economic model can draw on as well as the size of the effect. The Decision Support Unit's technical support document on survival extrapolation (TSD 14) notes that censored trial data force extrapolation, that the parametric models can give very different results when much extrapolation is needed, and that care is needed in the common case where lifetime data are immature. A trial stopped at an interim look leaves more patients censored, so more of the mean survival in a partitioned survival model comes from the extrapolated tail, and the choice of curve adds structural uncertainty.

Inputs other than the primary outcome are affected too. The FDA guidance notes that a trial stopped early for efficacy has a smaller sample for safety and important secondary endpoints, and ICH E9 asks that the effects of early stopping on other important variables be considered; for an economic model these can include resource use and health-related quality of life. In a hypothetical case study based on the CACTUS pilot trial, Flight and colleagues found that model parameters calculated from the primary outcome changed after bias adjustment even when the primary and health economic outcomes were uncorrelated; the overall impact of adjustment was small, although the optimal design changed when the correlation was 0.4.

Using economics when choosing a stopping boundary

Choosing a stopping boundary also changes what the trial is worth to a health system. In the same hypothetical case study, Flight and colleagues calculated the expected value of sample information for a fixed design and for Pocock and O'Brien-Fleming stopping rules with two or five analyses, and the O'Brien-Fleming rule with two analyses gave the highest expected net benefit of sampling. Value-based sequential designs, one branch of adaptive trial design, go further and set stopping to maximise expected net benefit rather than to spend an error budget.

Alpha spending beside related monitoring ideas

The original Pocock and O'Brien-Fleming boundaries assumed a set number of equally spaced looks, which the spending approach relaxes. Two neighbouring ideas differ from it as follows.

Common misreadings in appraisal documents

One misreading treats the cumulative error spent at a look as the p-value needed at that look. In the example, the Pocock type plan has spent 0.0155 by look 2, but the nominal one-sided p-value needed there is about 0.009, because earlier looks have used part of the budget; likewise the final O'Brien-Fleming type critical value of 2.014 corresponds to a nominal one-sided p-value of about 0.022, so a final result just under 0.025 can fail.

Another reads a crossed boundary as an automatic stop, whereas DeMets and Lan describe boundaries as guidelines that a monitoring committee weighs alongside safety, other outcomes and external evidence. A third overlooks binding futility rules, about which the FDA guidance warns that a trial continuing past a binding futility boundary is likely to be treated as having failed to provide evidence of efficacy.

Sources

• Bassler D, Briel M, Montori VM, et al. Stopping randomized trials early for benefit and estimation of treatment effects: systematic review and meta-regression analysis. JAMA. 2010;303(12):1180-1187.
• DeMets DL, Lan KKG. Interim analysis: the alpha spending function approach. Statistics in Medicine. 1994;13:1341-1352.
• Flight L, Arshad F, Barnsley R, Patel K, Julious S, Brennan A, Todd S. A review of clinical trials with an adaptive design and health economic analysis. Value in Health. 2019;22(4):391-398.
• Flight L, Julious S, Brennan A, Todd S. Expected value of sample information to guide the design of group sequential clinical trials. Medical Decision Making. 2022;42(4):461-473.
• Flight L, Brennan A, Wilson I, Chick SE. A tutorial on value-based adaptive designs: could a value-based sequential 2-arm design have created more health economic value for the Big CACTUS trial? Value in Health. 2024;27(10):1328-1337.
• International Council for Harmonisation. E9: Statistical principles for clinical trials. Step 4 version, 5 February 1998. Section 4.5.
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Decision Support Unit, University of Sheffield; 2011, updated March 2013.
• Anderson KM. gsDesign Technical Manual, chapters 6 and 8, and gsDesign package reference for sfLDOF and sfLDPocock. https://keaven.github.io/gsd-tech-manual/ (accessed October 2026).
• Montori VM, Devereaux PJ, Adhikari NK, et al. Randomized trials stopped early for benefit: a systematic review. JAMA. 2005;294(17):2203-2209.
• US Food and Drug Administration. Adaptive designs for clinical trials of drugs and biologics: guidance for industry. Silver Spring, MD: FDA; November 2019. Section V.A.