Indirect Comparison

Indirect Comparison by the Bucher Method

In health technology assessment, an indirect comparison is the calculation that estimates how treatment B performs against treatment C when no randomised trial has put the two side by side, using the trials that compare each of them with a third treatment A. The two relative effects against A are subtracted on a scale where effects add, such as the log hazard ratio, and their variances are summed to give the uncertainty of the result. The NICE manual (PMG36, section 3.4.11) sets expectations for exactly this case, where technologies have not been evaluated within a single trial. The method published by Bucher and colleagues in 1997 is the simplest anchored method in the family of indirect treatment comparison methods, and network meta-analysis extends the same arithmetic to many treatments. The shared arm itself is the subject of the common comparator page, and the result usually becomes an input to a health economic model.

Why relative effects are subtracted and arms are not pooled

The tempting shortcut is to take the B arms from one set of trials and the C arms from another and compare them as if they came from one trial. Bucher and colleagues argued that pooling the active arms in this way offers no advantage over observational data and is prone to bias, whereas comparing the differences between each treatment and placebo preserves the randomisation of the original groups. Simulation studies by Glenny and colleagues found that the naive method is liable to bias and also gives over-precise answers.

PMG36 (section 3.4.20) requires trial randomisation to be preserved in adjusted indirect comparisons and does not accept comparisons of single treatment arms from different randomised trials.

The Bucher estimator and its variance

The Cochrane Handbook describes the calculation for a trio of treatments. Each direct comparison is first summarised, by a single trial or a pairwise meta-analysis, and the indirect estimate is the difference between the two summaries.

$$\hat{d}_{BC} = \hat{d}_{BA} - \hat{d}_{CA}$$

where $\hat{d}_{BA}$ is the estimated effect of B relative to the common comparator A, $\hat{d}_{CA}$ is the estimated effect of C relative to A, and $\hat{d}_{BC}$ is the indirect estimate of B relative to C, all on an additive scale such as a log ratio or a mean difference.

The two direct estimates come from different trials, so they are independent and their variances add.

$$\text{Var}(\hat{d}_{BC}) = \text{Var}(\hat{d}_{BA}) + \text{Var}(\hat{d}_{CA})$$

where each variance on the right is the squared standard error of a direct estimate. A 95% confidence interval then follows in the usual way.

$$\hat{d}_{BC} \pm 1.96 \times \sqrt{\text{Var}(\hat{d}_{BC})}$$

where 1.96 is the 97.5th percentile of the standard normal distribution. For a ratio measure the estimate and both limits are exponentiated at the end, so the indirect ratio equals one direct ratio divided by the other.

Published trials often report a ratio with its confidence interval and no standard error. The Cochrane Handbook states that ratio measures are analysed on the natural log scale and that a 95% interval is 3.92 standard errors wide, which gives the following.

$$SE = \frac{\ln U - \ln L}{3.92}$$

where $SE$ is the standard error of the log ratio, $U$ is the upper 95% limit and $L$ is the lower 95% limit of the reported ratio.

Choosing the scale for the subtraction

The subtraction is only meaningful on a scale on which the relative effect is expected to be the same across the trials being linked. The EU HTA Coordination Group methodological guideline notes that the Bucher method is compatible with odds ratios, risk ratios, risk differences, standardised mean differences and hazard ratios. NICE DSU Technical Support Document (TSD) 18, Recommendation 5, states that indirect comparisons must be carried out on the usual linear predictor scale for that outcome, which it describes as typically the logit scale for proportions and the log scale for rate outcomes.

The choice matters because effect modification is defined on a scale. TSD 18 refers to effect modifiers as variables that modify treatment effects on the scale of the comparison, so a variable that leaves the log odds ratio unchanged can still change the risk difference. PMG36 (section 3.4.10) adds that where control-group risks differ substantially, the analyst should assess whether the relative effect is constant over baseline risks, especially when the model's baseline event rate differs from the rates in the trials.

Precision lost by going round the comparator

Because the variances add, the indirect estimate is always less precise than either of the direct estimates it is built from. Glenny and colleagues found that four times as many similarly sized trials are needed for the indirect approach to have the same power as directly randomised comparisons.

The factor of four follows from the variance formula. If one direct trial of B against C has variance $v$, trials of the same size of B against A and of C against A have about the same variance each, so the indirect route gives $v + v$ from two trials. Halving that back to $v$ needs two trials on each side, four in all.

PMG36 (section 3.4.11) states that the committee will consider the additional uncertainty associated with the lack of direct evidence when relative effectiveness comes from indirect sources only. A wide interval that includes no difference does not show that the two treatments are equivalent.

Worked example with hazard ratios for progression-free survival

All numbers here are illustrative. A new cancer drug B is to be appraised against C, the treatment the NHS currently uses. Neither has been trialled against the other, but both have been compared with an older chemotherapy A, which serves as the common comparator. Hazard ratios are for progression or death, and below 1 favours the first-named treatment.

The standard errors come from the interval formula. For B against A, $(\ln 0.75 - \ln 0.48)/3.92 = (-0.2877 + 0.7340)/3.92 = 0.1138$. For C against A, $(\ln 0.91 - \ln 0.62)/3.92 = (-0.0943 + 0.4780)/3.92 = 0.0979$.

The indirect log hazard ratio of B against C is the difference of the two log hazard ratios.

$$\ln HR_{BC} = -0.5108 - (-0.2877) = -0.2231$$

where $\ln HR_{BC}$ is the log hazard ratio of B against C estimated through A. Exponentiating gives a hazard ratio of 0.80, which equals 0.60 divided by 0.75.

The variance is $(0.1138)^2 + (0.0979)^2 = 0.01295 + 0.00958 = 0.02253$, a standard error of 0.1501. The 95% limits on the log scale are $-0.2231 \pm 1.96 \times 0.1501$, that is -0.5174 and 0.0711, so the hazard ratio is 0.80 with a 95% interval of 0.60 to 1.07. The z statistic is $-0.2231 / 0.1501 = -1.49$, a two-sided p value of about 0.14. Both direct trials showed a clear benefit against A, yet the indirect estimate for B against C does not exclude a hazard ratio of 1.

The estimate then enters the cost-effectiveness analysis. Suppose progression-free survival on C in the decision population follows an exponential curve with a median of 8 months, so the monthly hazard is $\ln 2 / 8 = 0.0866$. The hazard on B is 0.80 times 0.0866, which is 0.0693, giving a median of $\ln 2 / 0.0693 = 10.0$ months. Mean progression-free time under an exponential curve is the inverse of the hazard, 11.54 months on C and 14.43 months on B, a gain of 2.89 months.

The interval matters as much as the point estimate. At the lower limit of 0.596 the hazard on B is 0.0516 and the mean is 19.36 months, a gain of 7.82 months, while at the upper limit of 1.074 the hazard is 0.0930 and the mean is 10.75 months, a loss of 0.79 months. In probabilistic sensitivity analysis the log hazard ratio would be sampled from a normal distribution with mean -0.2231 and standard deviation 0.1501, the standard error of the estimate, so the hazard ratio follows a log-normal distribution. That spread is parameter uncertainty.

Shortcuts with confidence limits that give the wrong interval

Analysts working from published intervals sometimes divide the limits directly. The point estimate survives, since 0.60 divided by 0.75 is still 0.80, but the interval does not. Dividing lower by lower and upper by upper, 0.48 by 0.62 and 0.75 by 0.91, gives 0.774 to 0.824, far too narrow. Dividing each lower limit by the other upper limit, 0.48 by 0.91 and 0.75 by 0.62, gives 0.527 to 1.210, too wide, and close to the 0.528 to 1.212 obtained by adding the two standard errors in place of the variances. Only the log-scale calculation gives 0.596 to 1.074.

Doing the calculation in a spreadsheet

The method needs only logs and a square root, so it is often built in Excel next to the model. The layout below puts the hazard ratio, lower limit and upper limit for B against A in cells B2, C2 and D2, and the same for C against A in row 3, with the first two formulas copied down to row 3.

Conditions under which the Bucher result can be trusted

The arithmetic is exact, but the answer is only as good as the comparability of the two sets of trials. The Cochrane Handbook describes indirect comparisons as observational evidence across randomised trials that may suffer the biases of observational studies, such as confounding, and calls the requirement that the sets of trials are similar in all important factors other than the treatments compared transitivity. If the B against A trials enrolled mainly earlier-stage disease and the C against A trials mainly later-stage disease, and stage modifies the effect, the indirect estimate is biased however precise it looks.

The EU HTA Coordination Group methodological guideline adds conditions specific to the Bucher method. Consistency cannot be assessed, because there is no direct evidence on the comparison of interest, so a thorough assessment of similarity is even more important. The method assumes independent pairwise comparisons, which fails for multi-arm trials. Where random-effects models have been used for one or more direct comparisons, the guideline says use of the Bucher method is problematic and should be avoided, and it prefers network meta-analysis methods that incorporate random effects.

PMG36 (section 3.4.14) warns that indirect comparisons in networks of few trials are highly vulnerable to systematic bias, and allows population adjustment when effect modifiers may be imbalanced and individual patient data exist for at least one trial. That is the setting for anchored matching-adjusted indirect comparison and simulated treatment comparison, which keep the common comparator but reweight or model one trial towards the other trial's population.

When direct evidence also exists

The EU HTA Coordination Group practical guideline describes the Bucher method for testing consistency in a single loop. The direct and indirect estimates are subtracted and the null hypothesis of no inconsistency is referred to the normal distribution, and the test needs three independent sources of data.

Agreement cannot be assumed. Glenny and colleagues found considerable discrepancies between direct and indirect estimates in their case studies, with no predictable direction. PMG36 (section 3.4.11) says NICE prefers the network meta-analysis methods set out in the DSU evidence synthesis series, which can combine direct and indirect evidence in one model.

Sources

• Bucher HC, Guyatt GH, Griffith LE, Walter SD. The results of direct and indirect treatment comparisons in meta-analysis of randomized controlled trials. Journal of Clinical Epidemiology. 1997;50(6):683-691.
• Glenny AM, Altman DG, Song F, Sakarovitch C, Deeks JJ, D'Amico R, Bradburn M, Eastwood AJ. Indirect comparisons of competing interventions. Health Technology Assessment. 2005;9(26):1-134.
• Chaimani A, Caldwell DM, Li T, Higgins JPT, Salanti G. Chapter 11: Undertaking network meta-analyses (last updated October 2019). In: Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024.
• Higgins JPT, Li T, Deeks JJ, editors. Chapter 6: Choosing effect measures and computing estimates of effect (last updated August 2023). In: Higgins JPT, Thomas J, Chandler J, et al., editors. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024.
• Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU Technical Support Document 18: Methods for population-adjusted indirect comparisons in submissions to NICE. Decision Support Unit, ScHARR, University of Sheffield; December 2016.
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; published 31 January 2022, last updated 13 October 2026. Sections 3.4.10 to 3.4.14 and 3.4.20.
• Health Technology Assessment Coordination Group. Methodological Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons. Adopted 8 March 2024 under Regulation (EU) 2021/2282.
• Health Technology Assessment Coordination Group. Practical Guideline for Quantitative Evidence Synthesis: Direct and Indirect Comparisons. Adopted 8 March 2024 under Regulation (EU) 2021/2282.