Heterogeneity Assessment

Heterogeneity Assessment: Concept Architecture

Ask whether one effect can stand for every setting

Heterogeneity assessment examines how and why effects or outcomes differ across studies, populations, interventions, or settings. In evidence synthesis it combines a substantive review of comparability with statistical examination of variation beyond sampling error. This page follows that judgment from the study designs through summary measures, investigation, and the choice of an effect for a health economic model.

Check clinical and methodological differences first

Compare eligibility, baseline risk, disease severity, dose, delivery, comparator, outcome definition, follow-up, and jurisdiction before pooling results. A difference in the “same” treatment's effect can reflect genuine effect modification, inconsistent outcome measurement, study bias, or chance. Statistical agreement does not establish that clinically incompatible studies answer the same decision question.

Clinical heterogeneity concerns participants, interventions, comparators, and outcomes; methodological heterogeneity concerns design, conduct, and analysis. Statistical heterogeneity describes variation in estimated effects beyond what their within-study sampling errors would explain under a common-effect model. The Cochrane Handbook treats these as connected judgments, not as interchangeable labels.

Read a forest plot and quantify inconsistency

Place each study's effect estimate and uncertainty on a common effect scale and inspect direction, magnitude, overlap, and any outlying results. Cochran's $Q$ statistic compares observed dispersion with the precision-weighted mean; a common descriptive expression for inconsistency is:

$$I^2=\max\!\left(0,\frac{Q-(K-1)}{Q}\right)\times100\%, \qquad Q>0,$$

where $K$ is the number of studies and $K-1$ is the heterogeneity test's degrees of freedom. If five illustrative studies yield $Q=10$, the descriptive value is $(10-4)/10=0.60$, or $I^2=60\%$. That calculation alone does not say which studies differ, why they differ, or whether the difference matters clinically.

$I^2$ is influenced by study precision and has substantial uncertainty with few studies. It is not the percentage of patients whose treatment effects differ or a stand-alone threshold for accepting a pooled estimate. The Cochrane guidance recommends interpreting it alongside effects, confidence intervals, the number of studies, and the substantive context.

Distinguish variance from uncertainty in the average

In a random-effects meta-analysis, $\tau^2$ represents the estimated between-study variance of true effects on the analysis scale. Its units are the squared effect units: for log risk ratios, it is on the squared log-ratio scale. A large standard error for a pooled mean is uncertainty about the average; a large $\tau$ concerns variation of true effects across settings. One cannot compute $\tau^2$ from the illustrative $I^2$ value alone without the study estimates, within-study variances, and an estimation method.

A prediction interval aims to show a plausible range for the true effect in a new comparable setting under the random-effects model, incorporating between-study variation and uncertainty in its estimation. It can include no benefit or harm even when the confidence interval for the pooled mean excludes the null. With few studies, both the variance estimate and prediction interval can be unstable, so disclose methods and assumptions rather than treating a plotted interval as a guarantee.

Investigate differences with a prior rationale

Prespecify a small set of plausible effect modifiers, such as baseline severity, care setting, or treatment dose. Subgroup analyses and meta-regression can explore whether effects vary with them, but between-study associations can reflect confounding and ecological bias. Use an interaction comparison to assess subgroup differences; “significant in one subgroup but not the other” is not itself evidence of an interaction.

Examine influential studies, risk of bias, incompatible outcome definitions, and the consequences of alternative synthesis methods. Avoid selecting a subgroup or excluding a study solely because it gives a preferred result. Cochrane's reporting guidance calls for the proposed modifiers, timing of their selection, and comparison method to be explicit; decision-focused evidence-synthesis research describes implications for cost-effectiveness.

Choose an input for the target decision

An economic model needs an effect applicable to its specific population, comparator, care pathway, and setting. Depending on those relationships, a pooled mean, a context-adjusted estimate, a study-specific estimate, or a predictive distribution might be justified; they answer different questions. Carry uncertainty about that choice into scenario and probabilistic analyses without double-counting between-study variation as though it were also independent sampling error.

Heterogeneity in baseline risk, costs, practice patterns, and health-state utilities can matter even when a relative treatment effect is similar. Transporting an entire cost-effectiveness result across countries is especially sensitive to prices and service delivery. ISPOR's transferability task-force report discusses which economic inputs can vary across jurisdictions, while research on random-effects interpretation shows why the chosen treatment-effect summary changes model results.

Report the judgment, including unresolved gaps

State what was compared, whether pooling was justified, the effect scale, forest-plot patterns, $Q$, $I^2$, $\tau^2$ when estimated, and uncertainty in those measures. Explain prespecified investigations and their limitations, then identify the effect estimate and uncertainty used for the target decision. A low heterogeneity statistic cannot repair bias or prove that every patient benefits equally, while a high statistic is a prompt to understand variation rather than an automatic ban on synthesis.