Between-Study Heterogeneity

Between-Study Heterogeneity in Treatment Effects and Why It Matters for Decisions

Between-study heterogeneity is the part of the spread in trial results that chance cannot explain, because the trials in a meta-analysis are estimating effects that genuinely differ. In health economics it decides whether one pooled relative effect can stand for the population in an appraisal, and how wide the uncertainty around that effect should be in a cost-effectiveness model.

Chance versus real differences between studies

Every trial estimate carries sampling error, so results differ even when the underlying effect is identical. Heterogeneity is the excess spread beyond that error, and it changes what a pooled result means. The usual random-effects formulation separates the two layers as follows.

$$y_i = \theta_i + \varepsilon_i, \qquad \varepsilon_i \sim N(0, v_i), \qquad \theta_i \sim N(\mu, \tau^2)$$

where $y_i$ is the observed effect in study $i$ (for example a log odds ratio), $\theta_i$ is the true effect in that study, $\varepsilon_i$ is its sampling error with within-study variance $v_i$, $\mu$ is the mean of the distribution of true effects and $\tau^2$ is the between-study variance. Heterogeneity means $\tau^2 > 0$. The normal distribution for $\theta_i$ is a convention, and the Cochrane Handbook notes that its validity is difficult to establish.

The Handbook separates three ideas. Clinical diversity is variation in participants, interventions and outcomes; methodological diversity is variation in design, measurement and risk of bias; statistical heterogeneity is variation in the effects themselves, a consequence of either or both. The distinction matters for decisions. Clinical diversity creates heterogeneity when the factors that vary modify the treatment effect, which is a real difference between populations. Heterogeneity arising solely from methodological diversity indicates that studies suffer from different degrees of bias, and does not necessarily mean that the true effect varies.

Reading Q, I-squared and tau-squared together

The three standard statistics answer different questions; their formulae and the DerSimonian and Laird calculation are set out on the pairwise meta-analysis page. Cochran's Q tests whether the spread is compatible with chance alone. The Handbook warns that it has low power when studies are few or small, so a non-significant result must not be taken as evidence of no heterogeneity, which is why a P value of 0.10 is sometimes used; with many studies it can detect heterogeneity too small to matter clinically.

In the words of Higgins and colleagues' 2003 paper, I-squared describes the percentage of total variation across studies that is due to heterogeneity rather than chance. They tentatively attached the adjectives low, moderate and high to values of 25%, 50% and 75%, while warning that a naive categorisation would not suit all circumstances. The current Handbook gives overlapping bands (0% to 40% might not be important, 75% to 100% is considerable) and its mandatory standard says simple thresholds should be avoided because there is much uncertainty in I-squared and tau-squared when studies are few. RevMan now computes the statistic from the variance estimate, as shown below.

$$I^2 = \frac{\tau^2}{\tau^2 + \tilde{v}} \times 100\%$$

where $\tau^2$ is the estimated between-study variance and $\tilde{v}$ is a typical within-study variance. The formula shows the main weakness. Rücker and colleagues showed that as studies become larger, estimates of $\tau^2$ remain unchanged on average while I-squared rises towards 100%. I-squared is a relative measure, since it says how much of the observed spread is real, not how large the real spread is.

Tau-squared is the absolute measure, and its square root $\tau$ is in the units of the effect measure, so on the log scale it can be compared with the size of the mean effect. Higgins and colleagues noted that its values depend on the effect scale, so, unlike I-squared, it is not comparable across outcomes on different scales.

Prediction intervals and the spread a decision maker faces

A random-effects model estimates the mean of the distribution of effects. The Handbook stresses that the confidence interval for that mean describes uncertainty in the location of the mean, not the degree of heterogeneity; with many studies the interval can be very tight despite a large amount of heterogeneity. A prediction interval expresses the spread directly. The Handbook gives a simple 95% interval, shown below.

$$\hat{\mu} \pm t_{k-1} \sqrt{\hat{\tau}^2 + SE(\hat{\mu})^2}$$

where $\hat{\mu}$ is the random-effects summary mean, $t_{k-1}$ is the 97.5th percentile of a t-distribution with $k-1$ degrees of freedom, $k$ is the number of studies, $\hat{\tau}^2$ is the estimated heterogeneity and $SE(\hat{\mu})$ is the standard error of the summary mean. Higgins, Thompson and Spiegelhalter's original 2009 proposal used $k-2$ degrees of freedom. The Handbook reads the interval as a summary of the spread of underlying effects, encourages it when there are five or more studies and no clear funnel plot asymmetry, and warns that with few studies it can be spuriously wide or narrow.

Explaining heterogeneity with subgroups and meta-regression

When heterogeneity is found, the Handbook lists several options. Analysts can recheck the data, since errors such as entering standard errors as standard deviations create spurious heterogeneity; reconsider the effect measure, because when comparator group risks vary, homogeneous risk ratios imply heterogeneous risk differences; explore causes; use a random-effects model for what cannot be explained; or decline to pool. A random-effects model does not make heterogeneity cease to be an issue.

Subgroup analysis compares groups of studies, and meta-regression extends it to continuous and multiple study characteristics, weighting studies by precision and allowing residual heterogeneity. The Handbook sets the following limits.

• Observational evidence. Patients are randomised within trials, not between them, so study-level comparisons can be confounded by other study characteristics.
• Enough studies. Meta-regression should generally not be considered with fewer than ten studies, and typical advice is at least ten studies for each characteristic modelled.
• Pre-specification. Explanations devised after heterogeneity is seen can at best generate hypotheses.
• Aggregation bias. A strong within-trial relationship with a patient characteristic such as age can be invisible when trials have similar mean ages.
• Underlying risk. Regressing effects on comparator group risk produces a false correlation through regression to the mean unless corrected.

The NICE Decision Support Unit's technical support document 3 (TSD 3) adds that a single analysis with an interaction term is superior to separate analyses for each subgroup, and that meta-regression with individual patient data estimates effect modifiers far more precisely because covariate values are more widely spread.

Worked example with nine trials, one pooled odds ratio and a wide range of local effects

All numbers are illustrative. Nine equally sized trials of a new drug against standard care report log odds ratios for hospital admission within 12 months. Each has within-study variance 0.0176, and a random-effects analysis gives $\hat{\mu} = -0.30$ and $\hat{\tau}^2 = 0.04$, so $\hat{\tau} = 0.20$.

Step 1 (precision of the mean). Each random-effects weight is $w_i = 1/(0.0176 + 0.04) = 17.361$; nine trials give a total of 156.25, so $SE(\hat{\mu}) = \sqrt{1/156.25} = 0.08$. The pooled odds ratio is $e^{-0.30} = 0.741$, with a Wald-type 95% confidence interval of $-0.30 \pm 1.96 \times 0.08$, or $-0.457$ to $-0.143$, which is 0.633 to 0.867 on the odds ratio scale.

Step 2 (inconsistency). $I^2 = 0.04/(0.04 + 0.0176) = 69.4\%$.

Step 3 (prediction interval). The predictive standard deviation adds $\hat{\tau}^2$ to the squared standard error, $\sqrt{0.04 + 0.08^2} = \sqrt{0.0464} = 0.2154$, and $t_8 = 2.306$, so the half-width is 0.4967 and the prediction interval is as follows.

$$-0.30 \pm 0.4967 = (-0.797,\ 0.197)$$

where the limits are log odds ratios, giving odds ratios of 0.451 to 1.217. With $k-2$ degrees of freedom ($t_7 = 2.365$) the interval is 0.445 to 1.233.

Step 4 (chance of harm in a new setting). Under a normal approximation, the probability that the mean effect is harmful (log odds ratio above zero) is that of $z = 0.30/0.08 = 3.75$, about 0.009%. For the true effect in a new similar setting it is $z = 0.30/0.2154 = 1.39$, about 8%.

Step 5 (larger trials). If each trial were four times larger (within-study variance 0.0044) with the same $\tau^2$, I-squared would rise to $I^2 = 0.04/0.0444 = 90.1\%$, yet the prediction interval would barely move, to 0.454 to 1.208.

Step 6 (model input). With a baseline admission risk of 0.20 (odds 0.25) and 4,000 GBP per admission, the pooled odds ratio gives odds of 0.185 and a risk of 0.156, which means 0.044 fewer admissions and about 175 GBP saved per patient. At the prediction limits the risk is 0.101 (about 395 GBP saved) or 0.233 (about 133 GBP extra).

Interpretation. The mean effect is clearly beneficial, but the effect in a new setting could range from a large saving to a net cost. I-squared rose from 69% to 90% without any change in the clinically relevant spread, which is why the judgement about spread rests on $\tau$ and the prediction interval, not I-squared.

Carrying heterogeneity into HTA and cost-effectiveness models

The NICE technology appraisal and highly specialised technologies manual (PMG36, last updated 13 October 2026) asks, in section 3.4.10, for statistical pooling to be accompanied by an assessment of heterogeneity, which a random-effects model can take into account only to some extent, with the degree and reasons explored as fully as possible. The same section asks for a check that relative effectiveness is constant over baseline risks when control-group risks differ substantially, especially when the model's baseline differs from the trials. Section 3.4.16 asks for external information to help estimate between-study heterogeneity and, in networks with few studies, suggests informative prior distributions, an approach developed in Bayesian meta-analysis.

TSD 3 argues that the predictive distribution of the effect in a new trial may often be more relevant to decision making than the distribution of the mean effect. In a probabilistic sensitivity analysis this means drawing the effect from $N(\hat{\mu}, \hat{\tau}^2 + SE(\hat{\mu})^2)$ rather than $N(\hat{\mu}, SE(\hat{\mu})^2)$, which in the example raises the share of harmful draws from almost none to about 8% and widens the uncertainty in net benefit. TSD 3 also cautions that the predictive variance should reflect only true variation between populations, that observed heterogeneity is likely to overestimate it, and that when the target population resembles some trials more than others, adjustment for external validity should be considered.

Boundaries with nearby concepts

Between-study heterogeneity sits one level above patients. Between-subject variation is the spread among individuals within a study, and treatment effect heterogeneity is variation in the effect across patients. In a network meta-analysis, heterogeneity within each comparison is distinct from inconsistency between direct and indirect evidence, and PMG36 asks for both to be reported. The usual confidence interval from a fixed-effect model assumes heterogeneity is absent. The practical review workflow is described under heterogeneity assessment.

Common misreadings

A high I-squared is often read as a large spread of effects, but it can arise from very precise trials with clinically trivial differences. A non-significant Q test with few trials does not show homogeneity. The Handbook warns that, under heterogeneity, a fixed-effect estimate may describe an effect that exists in no population, with an interval that is too narrow. Random-effects weighting gives smaller studies relatively more weight, so where small studies differ systematically, for example through publication bias, it exacerbates that bias. The choice between the two models should never rest on a heterogeneity test. A prediction interval describes the effect in a new setting, not the spread of responses among individual patients.

Sources

• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses (last updated November 2024). In: Higgins JPT, Thomas J, Chandler J, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.10 and 10.11.
• Dias S, Sutton AJ, Welton NJ, Ades AE. NICE DSU Technical Support Document 3: Heterogeneity: subgroups, meta-regression, bias and bias-adjustment. Decision Support Unit; September 2011, last updated April 2012.
• Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557-560. doi:10.1136/bmj.327.7414.557
• Higgins JPT, Thompson SG, Spiegelhalter DJ. A re-evaluation of random-effects meta-analysis. Journal of the Royal Statistical Society Series A. 2009;172(1):137-159. doi:10.1111/j.1467-985X.2008.00552.x
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 3.4.8 to 3.4.16.
• Rücker G, Schwarzer G, Carpenter JR, Schumacher M. Undue reliance on I2 in assessing heterogeneity may mislead. BMC Medical Research Methodology. 2008;8:79. doi:10.1186/1471-2288-8-79