Cumulative Meta-Analysis

Cumulative Meta-Analysis and the Timing of Evidence in HTA

A cumulative meta-analysis shows at what point the trials on a treatment had already settled its effect, which one final pooled estimate cannot show. A team led by Thomas Chalmers and Frederick Mosteller introduced the term in 1992 and used it to show advice and new trials lagging behind available evidence. For health economists and HTA bodies the method bears on research waste, on delays in adopting effective care and on whether a further trial is justified, a question that value of information analysis answers in decision terms. Its main statistical hazard is that reading every update as a significance test produces too many false-positive findings.

How the repeated pooling works

Viechtbauer describes the procedure as obtaining the average effect sequentially as studies are added, typically in chronological order. With $k$ studies the same model is fitted $k$ times, first to the earliest study, then to the first two, and so on, and any other ordering can be specified. Each step is an ordinary meta-analysis, and a cumulative forest plot shows one row per step.

For a fixed-effect model with the inverse-variance weights described in the Cochrane Handbook, the estimate after the first $j$ studies is as follows.

$$\hat{\theta}_{(j)} = \frac{\sum_{i=1}^{j} w_i y_i}{\sum_{i=1}^{j} w_i}, \qquad \mathrm{SE}\big(\hat{\theta}_{(j)}\big) = \frac{1}{\sqrt{\sum_{i=1}^{j} w_i}}, \qquad w_i = \frac{1}{v_i}$$

where $\hat{\theta}_{(j)}$ is the pooled effect after $j$ studies (on the log scale for a ratio measure), $y_i$ is the estimate from study $i$, $v_i$ is its variance and $w_i$ its weight. Under a random-effects model the between-study variance is re-estimated at each step, and the Cochrane Handbook warns that adding more heterogeneous studies can widen the confidence interval, whereas a fixed-effect interval can only narrow.

What the first cumulative meta-analyses showed

Lau and colleagues studied 15 treatments for myocardial infarction. For intravenous streptokinase, 33 trials ran between 1959 and 1988, yet a consistent, statistically significant fall in total mortality (odds ratio 0.74, 95% confidence interval 0.59 to 0.92) was present by 1973, after eight trials with 2,432 patients. The 25 later trials, with 34,542 more patients, barely moved the odds ratio and mainly narrowed the interval. Antman and colleagues found that review articles and textbooks were slow to recommend effective treatments and kept recommending some with no effect on mortality or with potential for harm.

The method has also exposed harms. Jüni and colleagues found that by the end of 2000 the trials of rofecoxib gave a relative risk of myocardial infarction of 2.30 (95% CI 1.22 to 4.33), and concluded that the drug, which had just been withdrawn when they reported in 2004, should have been withdrawn several years earlier. Fergusson and colleagues found 64 trials of aprotinin in cardiac surgery from 1987 to 2002, with the cumulative odds ratio for perioperative transfusion stable at 0.25 by the twelfth study in June 1992, while the trials cited a median of 20% of earlier trials.

Research waste and the case for another trial

Clarke, Brice and Chalmers searched for cumulative meta-analyses of clinical interventions published from 1992 to 2012 and found 50 reports containing more than 1,500 of them. Early positive results sometimes became null, early null or negative results were sometimes overturned, stable results would have been visible had a meta-analysis been done before a new trial, and later trials had sometimes been far too small to settle what remained uncertain. They argue that reviewing existing evidence first would have brought earlier uptake of effective care, fewer participants given less effective treatment and less waste from unjustified research.

Some cumulative meta-analyses have informed the case for a new trial. Sutton and colleagues proposed powering a trial on the updated meta-analysis that will include it, and found that under considerable heterogeneity even a very large study may change the meta-analysis little.

Repeated testing and trial sequential analysis

The Cochrane Handbook explains the problem. If there is no true effect, each update tested at the 5% level has a 5% chance of a false-positive result, so the chance that at least one update shows P below 0.05 is higher than 5%, and type I error is not controlled. Type II error is not controlled either.

Wetterslev and colleagues adapted the group sequential methods used for interim analysis of a single trial. Their trial sequential analysis sets a required information size, at least the sample size of one adequately powered trial, and judges each update against monitoring boundaries built with a Lan and DeMets alpha spending function. In six neonatal meta-analyses, five showed early P values below 0.05 that were potentially spurious. For a binary outcome the required information size is as follows.

$$\mathrm{RIS} = \frac{1}{1 - D^2} \cdot 4 \cdot \big(z_{1-\alpha/2} + z_{1-\beta}\big)^2 \cdot \frac{P(1-P)}{\mu^2}, \qquad P = \frac{P_C + P_E}{2}, \qquad \mu = P_C - P_E$$

where $P_C$ and $P_E$ are the anticipated event proportions in the control and experimental groups, $\mu$ is the anticipated effect, $\alpha$ and $\beta$ are the accepted type I and type II error risks, $z$ denotes standard normal quantiles and $D^2$ is the diversity measure of heterogeneity. Futility boundaries show when the assumed effect has become unlikely. Wetterslev and colleagues acknowledge that the information size is a moving target, since heterogeneity and control-group risk change as trials are added, and argue that a moving target is better than none.

The Cochrane Handbook discourages formal sequential methods for the main analyses or conclusions of updated reviews, because the meta-analyst does not control which trials appear, a review serves many decision makers, and the methods have important limitations under heterogeneity. They may be prespecified secondary analyses, or the primary approach in a prospectively planned series of trials.

Worked example with five trials and a required information size

All numbers in this example are illustrative. Five placebo-controlled trials of a drug report deaths, and a fixed-effect inverse-variance meta-analysis of the log odds ratio is repeated after each.

Step 1 (pooling). Trial B has log odds ratio $\ln[(40 \times 245)/(260 \times 55)] = -0.378$ and variance 1/40 + 1/260 + 1/55 + 1/245 = 0.0511, a weight of 19.57. With trial A (weight 9.54, log odds ratio $-0.606$), the pooled value is $(9.54 \times -0.606 + 19.57 \times -0.378)/29.11 = -0.453$, with standard error $\sqrt{1/29.11} = 0.185$, giving 0.636 (0.442 to 0.914).

Step 2 (required information size). Planning on a control risk of 20% and a 20% relative risk reduction gives $P_E = 0.16$, $\mu = 0.04$, $P = 0.18$ and $P(1-P) = 0.1476$. With two-sided $\alpha = 0.05$ and 90% power, $(1.960 + 1.282)^2 = 10.51$, so with no heterogeneity the size is 4 × 10.51 × 0.1476 / 0.0016 = 3,878 participants. With $D^2 = 0.25$ it would be 3,878 / 0.75 = 5,171.

Step 3 (absolute effect). At a 20% control risk the odds are 0.25. The 2010 estimate gives treated odds of 0.25 × 0.636 = 0.159, a risk of 13.72% and 62.8 deaths avoided per 1,000, against 0.25 × 0.710 = 0.1775, 15.07% and 49.3 per 1,000 in 2019.

By 2010 the naive interval excluded 1 with under a quarter of the required information, and the estimate later drifted towards the null. If net cost per patient were fixed and QALYs gained were proportional to deaths avoided, an ICER built on the 2010 estimate would be too low by a factor of 62.8 / 49.3 = 1.27. A sequential analysis would have treated 2010 as too early, while the stability from 2012 bears on whether trial E was needed.

Decision use in HTA and value of information

Statistical sufficiency and decision sufficiency differ. Claxton argued that rules of inference are irrelevant to the decisions economic evaluations inform, that the choice should rest on expected net benefit whatever the significance level, and that uncertainty in net benefit matters only for deciding whether more information is needed. A cumulative meta-analysis that crosses a significance boundary therefore neither licenses adoption on its own nor shows that more research is worthless. Value of information analysis weighs the expected population health gain from reduced decision error against the cost of the research, as Minelli and Baio set out for research funders.

The two methods answer different questions. A cumulative meta-analysis shows how the evidence behind a model parameter evolved and when it stabilised, while expected value of sample information, proposed for setting the optimum sample size of a trial, values a proposed study against the decision at hand.

Boundaries with nearby methods

A standard systematic review reports one meta-analysis at one date. A living review, which the Cochrane Handbook describes as continually updated as new evidence becomes available, produces a cumulative sequence in real time, and Simmonds and colleagues note that type I error rises rapidly with the number of updates, so more caution may be needed when it feeds a decision. Trial sequential analysis adds a required information size and boundaries to a cumulative meta-analysis. A prospective meta-analysis identifies or plans its trials before their results are known, the setting in which the Handbook accepts sequential stopping rules. Whether year or design explains a drift in effects is a question for meta-regression.

Limitations and common misreadings

Clarke and colleagues cite evidence from Cochrane reviews that early trials tend to overestimate treatment effects. They also cite Herbison and colleagues, who found that a median of four studies brought the estimate within 10% of its final value, though it is hard to tell which meta-analyses will change further.

1. The first update with P below 0.05 marks when the effect was proven. Repeated testing inflates false positives, and the first significant update may be a chance high.
2. A stable point estimate means research should stop. Stability in one effect does not settle uncertainty in other model inputs, which value of information analysis weighs.
3. A retrospective analysis shows what was knowable at the time. It pools the trials a later search found, which investigators then may not have had, and shares any publication bias in that set.

Sources

• Lau J, Antman EM, Jimenez-Silva J, Kupelnick B, Mosteller F, Chalmers TC. Cumulative meta-analysis of therapeutic trials for myocardial infarction. New England Journal of Medicine. 1992;327(4):248-254. https://doi.org/10.1056/NEJM199207233270406
• Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. JAMA. 1992;268(2):240-248. https://doi.org/10.1001/jama.1992.03490020088036
• Jüni P, Nartey L, Reichenbach S, Sterchi R, Dieppe PA, Egger M. Risk of cardiovascular events and rofecoxib: cumulative meta-analysis. Lancet. 2004;364(9450):2021-2029. https://doi.org/10.1016/S0140-6736(04)17514-4
• Fergusson D, Glass KC, Hutton B, Shapiro S. Randomized controlled trials of aprotinin in cardiac surgery: could clinical equipoise have stopped the bleeding? Clinical Trials. 2005;2(3):218-229. https://doi.org/10.1191/1740774505cn085oa
• Clarke M, Brice A, Chalmers I. Accumulating research: a systematic account of how cumulative meta-analyses would have provided knowledge, improved health, reduced harm and saved resources. PLoS One. 2014;9(7):e102670. https://doi.org/10.1371/journal.pone.0102670
• Sutton AJ, Cooper NJ, Jones DR, Lambert PC, Thompson JR, Abrams KR. Evidence-based sample size calculations based upon updated meta-analysis. Statistics in Medicine. 2007;26(12):2479-2500. https://doi.org/10.1002/sim.2704
• Wetterslev J, Thorlund K, Brok J, Gluud C. Trial sequential analysis may establish when firm evidence is reached in cumulative meta-analysis. Journal of Clinical Epidemiology. 2008;61(1):64-75. https://doi.org/10.1016/j.jclinepi.2007.03.013
• Wetterslev J, Jakobsen JC, Gluud C. Trial Sequential Analysis in systematic reviews with meta-analysis. BMC Medical Research Methodology. 2017;17:39. https://doi.org/10.1186/s12874-017-0315-7
• Thomas J, Askie LM, Berlin JA, Elliott JH, Ghersi D, Simmonds M, Takwoingi Y, Tierney JF, Higgins JPT. Chapter 22: Prospective approaches to accumulating evidence (last updated October 2019). In: Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-22
• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses (last updated November 2024). In: Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
• Simmonds M, Salanti G, McKenzie J, Elliott J; Living Systematic Review Network. Living systematic reviews: 3. Statistical methods for updating meta-analyses. Journal of Clinical Epidemiology. 2017;91:38-46. https://doi.org/10.1016/j.jclinepi.2017.08.008
• Viechtbauer W. Conducting meta-analyses in R with the metafor package. Journal of Statistical Software. 2010;36(3):1-48. https://doi.org/10.18637/jss.v036.i03
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/S0167-6296(98)00039-3
• Ades AE, Lu G, Claxton K. Expected value of sample information calculations in medical decision modeling. Medical Decision Making. 2004;24(2):207-227. https://doi.org/10.1177/0272989X04263162
• Minelli C, Baio G. Value of information: a tool to improve research prioritization and reduce waste. PLoS Medicine. 2015;12(9):e1001882. https://doi.org/10.1371/journal.pmed.1001882