Heterogeneity Assessment

A statistical evaluation of how much effect estimates vary across studies in a meta-analysis, commonly quantified using the I-squared statistic.

Source: Higgins et al. 2003

Heterogeneity assessment looks for differences in the effects reported by the studies being combined in a meta-analysis, beyond what chance alone would produce. When results vary widely, it signals that the studies differ in ways that matter, such as their patients, doses, or outcome definitions, so averaging them into one figure may obscure more than it reveals. Detecting this variation guides whether a single pooled estimate is sensible or whether the sources of difference must be explored. It gauges whether studies belong together. Higgins and colleagues (2019) describe this.

Source: Higgins et al. 2019

Heterogeneity is measured using statistics that quantify variation in effect estimates across studies beyond chance, notably I-squared, which expresses the percentage of total variation attributable to differences between studies rather than sampling error, and related measures such as the Cochran Q test and tau-squared, which estimates the between-study variance. Visual inspection of forest plots also helps. Higher I-squared indicates greater heterogeneity. So heterogeneity is measured through these statistical indicators, which together convey how much the study results differ, guiding whether and how the studies should be combined in meta-analysis.

Source: Higgins et al. 2003

The I-squared statistic indicates the proportion of the total variation in effect estimates across studies that is due to genuine differences between the studies rather than to chance, expressed as a percentage. A low I-squared suggests the studies are fairly consistent, with variation largely due to sampling error, while a high I-squared suggests substantial heterogeneity, with the studies estimating different effects. Rough thresholds are sometimes used to describe low, moderate, and high heterogeneity, though interpretation depends on context. So I-squared quantifies how much of the variation across studies reflects real inconsistency, informing the interpretation of a meta-analysis.

Source: Higgins et al. 2003

Heterogeneity matters in meta-analysis because substantial variation in effects across studies affects whether it is appropriate to combine them into a single pooled estimate and how that estimate should be interpreted: high heterogeneity suggests the studies may be estimating different effects, so a single pooled figure may be misleading, and it lowers confidence in the result. It also prompts investigation of the reasons for the variation, which can be informative. So assessing heterogeneity is important for deciding how to synthesise studies, how to interpret the pooled estimate, and whether to explore the sources of inconsistency.

Source: Higgins et al. 2003

Heterogeneity is handled in meta-analysis by choosing an appropriate model, such as a random-effects model that allows for between-study variation, rather than a fixed-effect model assuming a single common effect; by investigating the sources of heterogeneity through subgroup analyses or meta-regression; and, where heterogeneity is very high, by questioning whether pooling is appropriate. Reporting the degree of heterogeneity conveys the consistency of the evidence. So heterogeneity is addressed by modelling it, exploring its causes, and interpreting the pooled estimate accordingly, ensuring the synthesis reflects the variation among the studies rather than obscuring it.

Source: Guyatt et al. 2008