Sequential Analysis

Sequential Analysis in Clinical Trials and Safety Monitoring

Sequential analysis is a way of running a study in which the number of observations is not fixed at the outset. After each patient, or each group of patients, the evidence so far is compared with limits set before the study began, and the study either stops with a conclusion or carries on. In health care it decides when a randomised controlled trial may stop early and how licensed medicines and vaccines are watched for safety signals, and in value of information work it asks whether one more patient is worth recruiting. Abraham Wald's sequential probability ratio test is the founding method, and the group sequential designs of confirmatory trials are its practical descendants. Health economists inherit the consequences, because a trial that stops early is smaller and shorter and tends to overstate the effect that drives a cost-effectiveness model.

Why the sample size is left open

Wald, in his 1947 book, described the characteristic feature of sequential analysis as a number of observations not determined in advance, with the decision to stop depending at each stage on the results already seen. Tests can be built this way that need, on average, substantially fewer observations than equally reliable fixed-sample tests, and he devised the sequential probability ratio test in 1943.

The FDA guidance on adaptive designs (2019) states that sequential analyses can provide ethical and efficiency advantages by reducing the expected sample size and duration of trials. ICH E9 explains why group sequential designs are the most commonly applied, since it is more practicable to assess grouped outcomes at periodic intervals than continuously as each result arrives.

Wald's sequential probability ratio test

The test compares two simple hypotheses about a parameter, such as a response rate of 0.20 under the null hypothesis against 0.40 under the alternative. After each observation it updates the likelihood ratio of the data under the two hypotheses.

$$\Lambda_n = \prod_{i=1}^{n} \frac{f(x_i \mid \theta_1)}{f(x_i \mid \theta_0)}, \qquad A \approx \frac{1-\beta}{\alpha}, \qquad B \approx \frac{\beta}{1-\alpha}$$

where $\Lambda_n$ is the likelihood ratio after $n$ observations, $x_i$ is the $i$th observation, $f(x \mid \theta)$ is its probability or density when the parameter equals $\theta$, $\theta_0$ and $\theta_1$ are the parameter values under the null and alternative hypotheses, $\alpha$ is the planned type I error rate, $\beta$ is the planned type II error rate, and $A$ and $B$ are the upper and lower boundaries. Sampling continues while $B < \Lambda_n < A$, the alternative is accepted once $\Lambda_n \ge A$, and the null is accepted once $\Lambda_n \le B$.

Wald proposed these boundaries as approximations because exact values are laborious to find, and showed that the achieved errors $\alpha'$ and $\beta'$ satisfy $\alpha' \le \alpha/(1-\beta)$, $\beta' \le \beta/(1-\alpha)$ and $\alpha' + \beta' \le \alpha + \beta$, so neither can rise appreciably and the only possible cost is some extra observations. For a simple hypothesis against a single alternative he showed that no other test gives a larger saving in the average number of observations, and he reported that the test frequently saves about 50 per cent of the observations of the most efficient fixed-sample test.

On the log scale the test is a running total that moves with each observation until it leaves the band between $\ln B$ and $\ln A$. Wald's approximation for the average sample number follows from that picture.

$$E_\theta(n) \approx \frac{L(\theta)\ln B + \left[1-L(\theta)\right]\ln A}{E_\theta(z)}$$

where $E_\theta(n)$ is the expected number of observations when the true parameter is $\theta$, $L(\theta)$ is the probability of accepting the null hypothesis at $\theta$, and $E_\theta(z)$ is the expected log likelihood ratio contributed by one observation. It ignores the overshoot of the last step, so it slightly understates the true average.

Worked example with a binary response in a single-arm trial

All figures in this example are illustrative. A developer runs a single-arm phase II study of a new treatment. A response rate of 0.20 would not justify a larger trial, while 0.40 would. The design sets $\alpha = 0.05$ and $\beta = 0.10$, and each patient costs GBP 4,000 to recruit and follow up.

1. Boundaries. A = 0.90 / 0.05 = 18 and B = 0.10 / 0.95 = 0.1053, so $\ln A = 2.8904$ and $\ln B = -2.2513$.

2. Contribution of each patient. A response adds $\ln(0.40/0.20) = \ln 2 = 0.6931$ to the running total and a non-response adds $\ln(0.60/0.80) = \ln 0.75 = -0.2877$.

3. Boundaries in responses. With d responses among n patients the total is 0.6931 d minus 0.2877 (n minus d). Dividing through by 0.6931 + 0.2877 = 0.9808, the study stops for promise when d reaches 2.947 + 0.2933 n and for lack of promise when d falls to 0.2933 n minus 2.295.

4. One study path. After 14 patients with 7 responses, 7 × 0.6931 = 4.852 and 7 × 0.2877 = 2.014, leaving a total of 2.838, just below 2.890, so recruitment continues. Patient 15 responds, and 8 × 0.6931 = 5.545 less the same 2.014 leaves 3.531, which crosses the upper boundary. The study stops after 15 patients at a cost of 15 × 4,000 = GBP 60,000.

5. Average sample number. Under the alternative one patient contributes on average 0.40 × 0.6931 = 0.2772 less 0.60 × 0.2877 = 0.1726, a net 0.1046. The numerator is 0.90 × 2.8904 = 2.6014 less 0.10 × 2.2513 = 0.2251, a net 2.3763, so Wald's approximation gives 2.3763 / 0.1046 = 22.7 patients. Under the null the average contribution is 0.20 × 0.6931 = 0.1386 less 0.80 × 0.2877 = 0.2302, a net of minus 0.0916, and the numerator is 0.05 × 2.8904 = 0.1445 less 0.95 × 2.2513 = 2.1387, a net of minus 1.9942, giving 21.8.

6. Comparison with a fixed design. The smallest fixed exact binomial design with the same error limits enrols 47 patients and declares promise at 15 or more responses, with an actual type I error of 0.037 and statistical power of 0.901. It costs 47 × 4,000 = GBP 188,000.

7. Exact operating characteristics. Counting every possible path, with recruitment capped at 200, the sequential test has an actual type I error of 0.039 and type II error of 0.088, inside Wald's bounds of 0.05 / 0.90 = 0.056 and 0.10 / 0.95 = 0.105. The exact averages are 23.7 patients under the null and 25.5 under the alternative, a little above the approximations. Against 47 patients these are savings of about 50 and 46 per cent, and the expected cost under the alternative is 25.5 × 4,000 = GBP 102,000.

8. Interpretation. The saving is largest when the truth sits at one of the two hypotheses. At a true rate of 0.30 the average rises to 37.8 patients and about 0.2 per cent of studies are still running at 200, which is why practical designs set a maximum. The path above reports 8 / 15 = 0.53, and across all studies that stop for promise when the true rate is 0.40 the average naive estimate is 0.49, so a model fed that estimate would overstate the benefit.

Repeated looks and group sequential designs

An ordinary significance test at every look is not a valid sequential procedure. With two-sided tests at 0.05 and equally spaced looks at accumulating normal data, simulation gives an overall false positive rate of about 0.083 for two looks, 0.14 for five and 0.19 for ten. ICH E9 states that the procedures selected should always ensure that the overall probability of type I error is controlled.

A group sequential design applies boundaries at a few planned interim analyses rather than after every patient. The FDA guidance notes that O'Brien-Fleming boundaries need very persuasive early results while Pocock boundaries stop early more often, and that alpha spending allows flexibility in the number and timing of looks. It reports that one interim analysis at half the information with an O'Brien-Fleming efficacy boundary cuts the expected sample size by roughly 15 per cent when the alternative is true, at 90 per cent power. Stopping when success has become unlikely is covered under futility analysis.

Sequential monitoring of medicine and vaccine safety

After licensing the data arrive as a stream of exposures and adverse events rather than as a trial. Silva, Li and Kulldorff (2019) state that sequential analysis is now commonly used for post-market drug and vaccine safety surveillance, typically with a Poisson process for rare adverse events. Under the null hypothesis the expected count is usually the number of people exposed multiplied by the baseline risk in a post-exposure window, and under the alternative it is raised by a relative risk above one.

Kulldorff and colleagues proposed the maximized sequential probability ratio test for this setting in 2011, and Silva and colleagues describe it as widely used for detecting increased risks with licensed vaccines. A conditional version compares accruing data with historical data when the expected count is itself uncertain. A confirmed signal changes the adverse event inputs of any economic model of the product.

Sequential decisions valued in health economic terms

Classical boundaries are set by error rates, which say nothing about what more evidence is worth to a health system. Chick, Forster and Pertile (2017) model a fully sequential experiment with a delayed primary endpoint, seeking the policy that maximises the expected benefits of the adoption decision minus sampling costs, with outcomes valued by standard cost utility analysis. The solution shows when no experiment, a fixed sample or a sequential experiment is best, and gives optimal stopping boundaries.

Sampling then continues while the expected gain in decision quality from more patients exceeds their cost, the logic of the expected value of sample information and the expected net benefit of sampling revisited after each patient, with the value of continuing including the option to sample again. Bayesian interim rules are covered under Bayesian adaptive design, and fixed designs chosen on value grounds under optimal sample size.

Early stopping and the evidence used in appraisal

A health technology assessment has to read a trial that stopped at a boundary on its own terms. The FDA guidance states that conventional estimates tend to be biased towards greater effects under a group sequential design, that confidence intervals lack their nominal coverage, and that estimates allowing for the stopping rule should be planned in advance and used for reporting. A trial stopped early for efficacy also has a smaller sample for safety and secondary endpoints, and a shorter trial leaves more of the lifetime benefit in a model to extrapolation.

Sequential analysis beside related ideas

Several neighbouring terms name parts or extensions of sequential analysis. They differ in when the data are examined and in what sets the stopping point.

Sequential Monte Carlo shares the name but is a family of simulation algorithms for updating estimates and does not decide when a study stops.

Limitations and common misreadings

Most errors come from treating a sequential result as if it came from a fixed-sample study. Each of the following can distort the evidence passed to an economic model.

1. Reading the stopping estimate as the effect. Crossing a boundary shows that the evidence met a planned standard, and the naive estimate at that point tends to be too large.
2. Looking without a plan. ICH E9 states that an interim analysis not planned appropriately may flaw the results of a trial and that such analyses should be avoided.
3. Expecting every study to stop early. When the truth lies between the two hypotheses the average sample is much larger, and an open-ended test can run for a long time unless a maximum is set.
4. Ignoring delayed outcomes. Patients treated but not yet assessed when recruitment stops still add data.
5. Equating error control with a good decision. A design can hold its type I error and still recruit patients whose information is worth less than it costs.

Sources

• Chick S, Forster M, Pertile P. A Bayesian decision-theoretic model of sequential experimentation with delayed response. Journal of the Royal Statistical Society Series B. 2017;79(5):1439-1462. https://doi.org/10.1111/rssb.12225
• Food and Drug Administration. Adaptive Designs for Clinical Trials of Drugs and Biologics. Guidance for industry, November 2019. Section V.A. https://www.fda.gov/media/78495/download
• International Council for Harmonisation. ICH E9 Statistical Principles for Clinical Trials. Step 4, 5 February 1998. Sections 3.4 and 4.5. https://database.ich.org/sites/default/files/E9_Guideline.pdf
• Silva IR, Li L, Kulldorff M. Exact conditional maximized sequential probability ratio test adjusted for covariates. Sequential Analysis. 2019;38(1):115-133. https://doi.org/10.1080/07474946.2019.1574446
• Wald A. Sequential Analysis. New York: John Wiley and Sons; 1947. Introduction and chapter 3. https://archive.org/details/in.ernet.dli.2015.510091