Pairwise Meta-Analysis

Pairwise Meta-Analysis for Pooling Head-to-Head Trials of Two Treatments

A pairwise meta-analysis of head-to-head trials takes every eligible randomised trial, found by a systematic review, that compared the same two treatments, for example drug A against drug B, and combines their results into one relative effect with a confidence interval. Each trial contributes its own effect estimate, and more precise trials carry more weight. The NICE technology appraisal manual (PMG36) states a preference for randomised trials that compare the intervention directly with a relevant comparator, so a pooled direct comparison is a natural starting point for the clinical evidence in a health technology assessment, and the manual gives particular attention to relative effects that will be used in an economic model. Pairwise pooling uses only direct evidence, which sets it apart from an indirect comparison or a network meta-analysis.

Two stages, a summary per trial and then a weighted average

The Cochrane Handbook (chapter 10) describes meta-analysis as typically a two-stage process. First, a summary statistic is calculated for each study in the same way, such as a risk ratio for a dichotomous outcome or a difference in means for a continuous one. Second, the summary effect is a weighted average of those study estimates, and its standard error gives the confidence interval.

The choice of effect measure comes first. For dichotomous outcomes the Handbook names the risk ratio and odds ratio as relative measures and the risk difference and number needed to treat as absolute ones. It reports that relative measures are, on average, more consistent across studies than absolute measures, and generally recommends pooling risk ratios or odds ratios, then re-expressing the result in absolute terms. The number needed to treat has no simple variance estimator and is computed from the pooled result afterwards. Ratio measures, including the hazard ratio for time-to-event outcomes, are pooled on the natural log scale.

Fixed-effect and random-effects pooling

The two standard models differ in what they assume the trials are estimating. Both can be fitted with the inverse-variance method, which gives each study a weight equal to one over the variance of its effect estimate; for dichotomous outcomes with few events the Handbook generally prefers the Mantel-Haenszel method for a fixed-effect analysis. Under a fixed-effect model the pooled estimate and its standard error are given by the expressions below.

$$\hat{\mu}_{FE} = \sum_{i=1}^{k} \frac{w_i}{W} y_i, \qquad W = \sum_{i=1}^{k} \frac{1}{v_i}, \qquad SE(\hat{\mu}_{FE}) = \frac{1}{\sqrt{W}}$$

where $y_i$ is the effect estimate from head-to-head trial $i$ (for example a log risk ratio), $v_i$ is its variance, $w_i = 1/v_i$ is its inverse-variance weight, $W$ is the total weight, $k$ is the number of trials and $\hat{\mu}_{FE}$ is the pooled fixed-effect estimate. The Handbook states that this model assumes all trials estimate the same underlying effect, also called the common-effect assumption.

A random-effects model assumes the trials estimate different but related effects drawn from a distribution, conventionally normal, whose variance is the between-study variance $\tau^2$. In the RevMan algorithms set out by Deeks and Higgins, the simplest estimator, from DerSimonian and Laird, and the resulting weights are as follows.

$$\hat{\tau}^2_{DL} = \max\left\{0, \frac{Q - df}{C}\right\}, \qquad C = W - \frac{\sum_{i=1}^{k} w_i^2}{W}, \qquad w_i^{*} = \frac{1}{v_i + \hat{\tau}^2_{DL}}$$

where $\hat{\tau}^2_{DL}$ is the estimated between-study variance, $Q$ is Cochran's heterogeneity statistic defined below, $df = k - 1$ is its degrees of freedom, $C$ is a scaling term built from the fixed-effect weights and $w_i^{*}$ are the random-effects weights, which replace $w_i$ in the weighted average. The Handbook records that simulation studies favour other estimators, such as Paule and Mandel or restricted maximum likelihood (REML), now the RevMan default, although no single approach is universally preferable and estimators seldom change the summary effect much. The Hartung-Knapp-Sidik-Jonkman adjustment widens the confidence interval to reflect uncertainty in $\tau^2$, but can be overly wide with few studies and overly narrow when $\hat{\tau}^2$ is zero.

The NICE Decision Support Unit's technical support document 2 (TSD 2) sets the same models in a Bayesian generalised linear modelling framework. In the random-effects model, trial-specific effects are treated as exchangeable, the fixed-effect model is the special case with zero between-trial variance, and the two models are compared using residual deviance and the deviance information criterion.

Measuring heterogeneity with Q, I-squared and tau-squared

Every trial in a pairwise analysis compares the same two treatments, so head-to-head results that disagree by more than chance point to differences in patients, interventions, outcome definitions or trial conduct. That excess variation is heterogeneity, and it limits how far one pooled head-to-head effect can stand for the effect in the population being appraised. Cochran's Q and its rescaled form, as Higgins and colleagues defined it, are shown below.

$$Q = \sum_{i=1}^{k} \frac{\left(y_i - \hat{\mu}_{FE}\right)^2}{v_i}, \qquad I^2 = 100\% \times \max\left\{0, 1 - \frac{df}{Q}\right\}$$

where $Q$ is compared with a chi-squared distribution on $df = k - 1$ degrees of freedom and $I^2$ is the percentage of total variation across studies due to heterogeneity rather than chance.

With few head-to-head trials the Handbook warns that the test has low power and that I-squared and tau-squared are very uncertain, so simple thresholds mislead and the choice between fixed and random effects should never rest on a heterogeneity test.

Worked example of four head-to-head trials of drug A and drug B

The figures below are illustrative. Four randomised trials compare drug A with drug B, and the outcome is hospital admission within 12 months. Each log risk ratio has variance $\frac{1}{a} + \frac{1}{c} - \frac{1}{n_1} - \frac{1}{n_2}$ (the square of the standard error given in the RevMan algorithms), where $a$ and $c$ are the admissions and $n_1$ and $n_2$ the patients on drug A and drug B; for trial 1 that is 1/30 plus 1/45 minus 1/200 minus 1/200, or 0.04556.

1. Fixed-effect estimate. The weights sum to 121.29 and the weighted log risk ratios to -25.1146.

$$\hat{\mu}_{FE} = \frac{-25.1146}{121.29} = -0.2071$$

where $\hat{\mu}_{FE}$ is the pooled log risk ratio. Its standard error is $\sqrt{1/121.29} = 0.0908$, so the risk ratio is 0.813 with a 95% confidence interval of 0.680 to 0.971.

2. Heterogeneity. The deviations from -0.2071 are -0.1984, 0.0529, -0.5269 and 0.1425, giving weighted squared terms of 0.8640, 0.1039, 2.6867 and 1.0676.

$$Q = 0.8640 + 0.1039 + 2.6867 + 1.0676 = 4.7222, \qquad I^2 = \frac{4.7222 - 3}{4.7222} \times 100\% = 36.5\%$$

where $Q$ has 3 degrees of freedom and $I^2$ is the share of variation attributed to heterogeneity.

3. Between-study variance. The squared weights sum to 4,714.1, so the denominator is 121.29 minus 4,714.1/121.29, or 82.42.

$$\hat{\tau}^2_{DL} = \frac{4.7222 - 3}{82.42} = 0.0209$$

where $\hat{\tau}^2_{DL}$ is the DerSimonian and Laird estimate, so $\hat{\tau} = 0.1446$ on the log scale.

4. Random-effects estimate. Adding 0.0209 to each variance gives weights of 15.048, 20.899, 8.050 and 25.047, summing to 69.044, and the weighted log risk ratios sum to -16.8479.

$$\hat{\mu}_{RE} = \frac{-16.8479}{69.044} = -0.2440$$

where $\hat{\mu}_{RE}$ is the pooled mean log risk ratio. Its standard error is $\sqrt{1/69.044} = 0.1203$, so the risk ratio is 0.784 with a 95% confidence interval of 0.619 to 0.992.

5. Model input. An illustrative registry gives a 12-month admission risk of 0.15 on drug B in the population being appraised. Applying the pooled risk ratio gives 0.15 x 0.813 = 0.122 on drug A under fixed effects, about 28 fewer admissions per 1,000 patients, and 0.15 x 0.784 = 0.118 under random effects, about 32 fewer per 1,000.

Interpretation. Trial 4 carries 43.3% of the fixed-effect weight (52.555 of 121.29) but 36.3% under random effects (25.047 of 69.044), while trial 3 rises from 8.0% to 11.7%. The smaller trials report the larger benefits. This is the small-study pattern the Handbook warns about. Random-effects weighting moves the estimate towards the smaller trials, and if the pattern reflects bias both models are problematic, so the analyst would report both, flag the possible small-study effect and test the choice in a scenario analysis. With four trials, $\hat{\tau}^2$ is poorly estimated, and the Handbook encourages prediction intervals only with a reasonable number of studies, such as five or more, so none is computed here.

How the pooled effect enters an appraisal and a model

PMG36 has a subsection headed pairwise meta-analysis. It says pooling is appropriate when there are enough relevant and valid data with comparable outcome measures, asks for each study's characteristics to be reported with a forest plot, and asks for pooling to be accompanied by an assessment of heterogeneity, which a random-effects model takes into account only to some extent. Known clinical heterogeneity may be explored with subgroup analysis and meta-regression.

In cost-effectiveness analysis the relative effect is usually applied to a baseline risk from another source, as in step 5. The Handbook notes that converting a relative effect to an absolute one always requires an assumed comparator risk, and PMG36 asks for a check that the relative effect is constant over baseline risk when control-group risks differ substantially between trials, especially when the model's baseline differs greatly from the trials. TSD 2 states that for syntheses of two-arm trials frequentist methods can be used, provided parameter uncertainty is propagated through the decision model, which in a probabilistic sensitivity analysis means sampling the pooled log risk ratio rather than fixing it. Where heterogeneity is present, TSD 3 argues that the predictive distribution of the effect in a new trial may often be more relevant to decision making than the distribution of the mean.

Pairwise, indirect and network meta-analysis

A pairwise meta-analysis uses direct evidence on one pair of treatments only. When the treatments of interest have never been compared in the same trial, PMG36 asks for the series of pairwise head-to-head trials to be presented together with a network meta-analysis if appropriate. It defines an adjusted indirect comparison as a synthesis of trials linked through a common comparator, and a mixed treatment comparison as one that also includes head-to-head trials of the technologies of interest.

TSD 2 describes pairwise meta-analysis and indirect comparisons as special cases of network meta-analysis, so the same model applies to all three. PMG36 asks for results from pairwise meta-analyses of the direct comparisons to be presented alongside the full network results, and for heterogeneity between pairwise comparisons and inconsistency between direct and indirect evidence to be reported. Pairwise pooling therefore remains a check on the network even when a network meta-analysis supplies the model inputs. Pooling of summary results, as opposed to participant records, is covered under aggregate data meta-analysis.

Limitations and common misreadings

The confidence interval from a random-effects analysis describes uncertainty about the mean effect, not the spread of effects; the Handbook calls quoting it in isolation inappropriate. Under heterogeneity a fixed-effect estimate may describe an effect that exists in no population, with an interval that is too narrow. Random-effects weighting gives smaller trials relatively more weight, so if smaller trials are systematically different, for example through publication bias, the random-effects result exaggerates that bias.

Rare events need different methods. The Handbook advises against inverse-variance methods, including DerSimonian and Laird, when events are rare, and reports the Peto odds ratio as the least biased and most powerful method at event rates below 1% under stated conditions. Finally, the Handbook's mandatory standard is to meta-analyse only when participants, interventions, comparisons and outcomes are similar enough for the answer to be clinically meaningful; a precise pooled number from dissimilar trials is not evidence of a single treatment effect.

Sources

• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.2 to 10.4 and 10.10.
• Deeks JJ, Higgins JPT. Statistical algorithms in Review Manager. Cochrane Statistical Methods Group; May 2022.
• Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 2: A generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. Decision Support Unit; 2011, last updated September 2016.
• Dias S, Sutton AJ, Welton NJ, Ades AE. NICE DSU Technical Support Document 3: Heterogeneity: subgroups, meta-regression, bias and bias-adjustment. Decision Support Unit; 2011, last updated April 2012.
• Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-560.
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 3.4.1 and 3.4.8 to 3.4.20.