Aggregate Data Meta-Analysis

Aggregate Data Meta-Analysis of Trial Summary Results for Model Inputs

Many health technology assessment models take their treatment effect from trials whose participant records the modelling team never sees. Each trial's reported estimate and uncertainty are pooled by meta-analysis, and the result enters the model as a relative effect. The pooling formulas are standard, but between-study heterogeneity has to be measured and carried into the model, and the approach changes when participant-level records or a network of comparisons are available. An illustrative worked example pools three hazard ratios and converts the result into an annual transition probability for a Markov model.

What study-level summaries carry

An aggregate data synthesis works only with what each study reports. For a generic inverse-variance analysis, the Cochrane Handbook states that the basic data needed are each study's effect estimate and its standard error. Ratio measures such as the hazard ratio or odds ratio are pooled on the natural log scale and transformed back for presentation.

The NICE Decision Support Unit's technical support document 2 (TSD 2) gives one framework for synthesising aggregate data from randomised trials for probabilistic decision making. No summary carries the variation inside a trial, so a study-level table holds a mean age, not each participant's age.

Inverse-variance pooling with common-effect and random-effects estimates

The pooled estimate is a weighted average in which each study's weight is the inverse of the variance of its effect estimate. More precise studies therefore count for more. Under a common-effect (fixed-effect) model, which the Cochrane Handbook describes as assuming that all estimates target the same underlying effect, the calculation is as follows.

$$\hat{\theta}_{F} = \frac{\sum_{i=1}^{k} w_i y_i}{\sum_{i=1}^{k} w_i}, \qquad w_i = \frac{1}{v_i}, \qquad SE(\hat{\theta}_{F}) = \sqrt{\frac{1}{\sum_{i=1}^{k} w_i}}$$

where $\hat{\theta}_{F}$ is the pooled common-effect estimate, $y_i$ is the effect estimate from study $i$ (for example a log hazard ratio), $v_i$ is its variance, $w_i$ is its weight, $k$ is the number of studies and $SE(\hat{\theta}_{F})$ is the standard error of the pooled estimate.

A random-effects model assumes the studies estimate different but related effects, with between-study variance $\tau^2$, and weights the studies as follows.

$$w_i^{*} = \frac{1}{v_i + \tau^2}, \qquad \hat{\theta}_{R} = \frac{\sum_{i=1}^{k} w_i^{*} y_i}{\sum_{i=1}^{k} w_i^{*}}$$

where $w_i^{*}$ is the random-effects weight of study $i$, $\tau^2$ is the between-study variance and $\hat{\theta}_{R}$ is the pooled mean effect. The simplest estimator of $\tau^2$ is the moment-based method of DerSimonian and Laird, shown below.

$$\hat{\tau}^2_{DL} = \max\left\{0, \frac{Q - (k-1)}{\sum w_i - \sum w_i^2 / \sum w_i}\right\}$$

where $\hat{\tau}^2_{DL}$ is the DerSimonian and Laird estimate, $Q$ is Cochran's statistic defined below, $k - 1$ is its degrees of freedom and $w_i$ are the common-effect weights. The Cochrane Handbook records that other estimators, such as Paule and Mandel or restricted maximum likelihood (REML), performed better in simulation studies. The NICE manual (PMG36) prefers the DSU evidence synthesis series methods for network meta-analysis, and TSD 2 takes a Bayesian approach; the manual adds that in networks with few studies, informative prior distributions for the between-study heterogeneity may be preferable.

Measuring heterogeneity before it reaches the model

Heterogeneity is variation between study results beyond what chance would produce. In a decision model it governs how far the treatment effect could differ in the population being appraised. Cochran's Q sums each study's weighted squared deviation from the common-effect estimate, and I-squared rescales it as shown below.

$$Q = \sum_{i=1}^{k} w_i \left(y_i - \hat{\theta}_{F}\right)^2, \qquad I^2 = \max\left\{0, \frac{Q - (k-1)}{Q}\right\} \times 100\%$$

where $Q$ is Cochran's statistic, $I^2$ is the percentage of total variation across studies attributed to heterogeneity rather than chance, and the other terms are as above. Higgins and colleagues define I-squared this way, with negative values set to zero.

The Cochrane Handbook warns that the chi-squared test based on Q has low power with few or small studies, that I-squared and $\tau^2$ are very uncertain with few studies, and that simple thresholds should not be used. Because $\tau^2$ is hard to read, it suggests prediction intervals, which combine $\tau^2$ with the squared standard error of the mean and use a t-distribution with $k - 1$ degrees of freedom, but can be very problematic when studies are few. The NICE manual asks that pooling be accompanied by an assessment of heterogeneity, notes that a random-effects model takes it into account only to some extent, and asks for its causes to be explored, including through subgroup analysis and meta-regression.

Worked example with three trials pooled into a Markov transition

The figures below are illustrative. Three randomised trials compare a new treatment with standard care on time to disease progression, and each log hazard ratio and standard error has been extracted from the published confidence interval.

1. Common-effect estimate. The weighted sum is divided by the total weight.

$$\hat{\theta}_{F} = \frac{25 \times (-0.60) + 100 \times (-0.05) + 16 \times (-0.40)}{25 + 100 + 16} = \frac{-26.4}{141} = -0.1872$$

where $\hat{\theta}_{F}$ is the pooled log hazard ratio. Its standard error is $\sqrt{1/141} = 0.0842$, giving a hazard ratio of 0.829 with a 95% confidence interval of 0.703 to 0.978.

2. Heterogeneity. The deviations from the pooled value are -0.4128, 0.1372 and -0.2128, giving weighted squared terms of 4.2594, 1.8833 and 0.7243, from which Q and I² follow.

$$Q = 4.2594 + 1.8833 + 0.7243 = 6.8670, \qquad I^2 = \frac{6.8670 - 2}{6.8670} \times 100\% = 70.9\%$$

where $Q$ has 2 degrees of freedom and $I^2$ is the share of variation attributed to heterogeneity.

3. Between-study variance. The squared weights sum to 10,881, so the denominator is 141 minus 10,881/141, or 63.8298, which gives the estimate below.

$$\hat{\tau}^2_{DL} = \frac{6.8670 - 2}{63.8298} = 0.07625$$

where $\hat{\tau}^2_{DL}$ is the DerSimonian and Laird estimate, so $\hat{\tau} = 0.2761$.

4. Random-effects estimate. Adding 0.07625 to each variance gives weights of 8.602, 11.594 and 7.207, which sum to 27.404, and the pooled estimate is as follows.

$$\hat{\theta}_{R} = \frac{8.602 \times (-0.60) + 11.594 \times (-0.05) + 7.207 \times (-0.40)}{27.404} = \frac{-8.6239}{27.404} = -0.3147$$

where $\hat{\theta}_{R}$ is the pooled mean log hazard ratio. Its standard error is $\sqrt{1/27.404} = 0.1910$, giving a hazard ratio of 0.730 with a 95% confidence interval of 0.502 to 1.062. Cochrane's rough illustration, the mean plus or minus 1.96 times 0.2761, gives hazard ratios of 0.425 to 1.254; the full prediction interval, with standard deviation $\sqrt{0.07625 + 0.1910^2} = 0.336$ and a t value of 4.303 on 2 degrees of freedom, gives about 0.17 to 3.10. Three trials are too few for either range to be reliable.

5. Conversion to a model input. An external source is assumed to give an annual progression probability of 0.20 on standard care. Assuming a constant hazard within each annual cycle and proportional hazards, the hazard ratio multiplies the rate, so the probability is converted to a rate, adjusted and converted back as follows.

$$r_0 = -\ln(1 - 0.20) = 0.2231, \qquad p_1 = 1 - e^{-r_0 \times HR}$$

where $r_0$ is the annual progression rate on standard care, $HR$ is the pooled hazard ratio and $p_1$ is the annual progression probability on the new treatment. The common-effect hazard ratio of 0.829 gives a rate of 0.1850 and $p_1 = 0.169$; the random-effects hazard ratio of 0.730 gives a rate of 0.1629 and $p_1 = 0.150$.

Interpretation. Trial 2 carries 70.9% of the common-effect weight (100 of 141) but 42.3% under random effects (11.594 of 27.404), so the two smaller trials, which report larger effects, count for much more, and the random-effects mean is larger, less certain and has an interval crossing 1. Larger effects in smaller trials form the small-study pattern that, as the final section explains, may reflect bias, so the random-effects mean is not automatically the better input and the choice belongs in a scenario analysis.

Carrying the pooled effect into a probabilistic model

In cost-effectiveness analysis, the pooled relative effect is usually applied to a baseline from another source, such as a registry. TSD 2 treats trial baselines as nuisance parameters and builds absolute rates from a baseline plus the relative effect on the linear scale. If event risk differs substantially between the trials' control groups, the NICE manual asks for an assessment of whether the relative effect is constant over baseline risk, which it calls especially important when the model's baseline rate is very different from the trial rates.

For probabilistic sensitivity analysis, the pooled log hazard ratio can be sampled from a normal distribution with its mean and standard error, which TSD 2 frames as propagating parameter uncertainty through the decision model. TSD 3 argues that, with heterogeneity, the predictive distribution of the effect in a new trial may often be more relevant to decision making than that of the mean, approximating its variance as the between-trial variance plus the variance of the mean; in step 4 that gives a standard deviation of 0.336, against 0.191 for the mean.

Aggregate data compared with individual patient data

Individual patient data meta-analysis collects, checks and reanalyses each participant's original record. The Cochrane Handbook chapter on individual participant data lists advantages such as longer follow-up, checks of assumptions such as proportional hazards, and more power to study effect modification, but notes that such reviews of therapeutic interventions typically take at least two years and that aggregate data synthesis may be an appropriate first step.

The sharpest difference concerns effect modifiers. The Cochrane Handbook gives the example of age, where a strong relationship between age and treatment effect within each trial can be invisible across trials with similar mean ages, a problem known as aggregation bias or the ecological fallacy, and TSD 3 adds that regression on individual data is usually much more powerful and can avoid it. Both sources treat comparisons of effect modifiers across studies as observational, even when every trial was randomised.

From pairwise pooling to network meta-analysis

A pairwise meta-analysis pools trials of the same two treatments. Appraisals often involve comparators that no single trial includes, and network meta-analysis extends aggregate data pooling to a connected set of comparisons, combining direct and indirect evidence. TSD 2 handles both within one framework.

The NICE manual asks for a network meta-analysis, if appropriate, where technologies have not been compared within one randomised trial, and says population adjustment, which can be considered when effect modifiers may be imbalanced, needs individual patient data from at least one trial. TSD 18 notes that standard indirect comparisons use aggregate data and assume no difference between trials in the distribution of effect modifiers, and describes matching-adjusted indirect comparison and simulated treatment comparison as applicable where individual data are available in one population and aggregate data in another.

Where aggregate data synthesis misleads

The value that reaches the model depends on choices easily made silently, such as the effect measure, the model, the heterogeneity estimator and whether to sample the mean or the predictive distribution. A common error is to choose the model from the heterogeneity test, which has little power with few trials. Another is to read I-squared as the amount of heterogeneity, when the Cochrane Handbook notes that the importance of a value depends on the size and direction of effects and the strength of evidence for heterogeneity. It also notes that random-effects pooling gives relatively more weight to smaller studies, so if they report larger effects the mean moves towards them, which may reflect small-study bias rather than true variation. Finally, a pooled effect is only as complete as the reporting behind it, and unreported outcomes and short follow-up pass straight into the model.

Sources

• DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7(3):177-188.
• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.3, 10.10 and 10.11.
• Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 2: A generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. Decision Support Unit; 2011, last updated September 2016.
• Dias S, Sutton AJ, Welton NJ, Ades AE. NICE DSU Technical Support Document 3: Heterogeneity: subgroups, meta-regression, bias and bias-adjustment. Decision Support Unit; 2011, last updated April 2012.
• Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-560.
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 3.4.10, 3.4.11, 3.4.14 and 3.4.16.
• Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU Technical Support Document 18: Methods for population-adjusted indirect comparisons in submissions to NICE. Decision Support Unit; 2016.
• Tierney JF, Stewart LA, Clarke M. Chapter 26: Individual participant data. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024.
• Veroniki AA, Jackson D, Viechtbauer W, et al. Methods to estimate the between-study variance and its uncertainty in meta-analysis. Research Synthesis Methods. 2016;7(1):55-79.