Meta-Analysis

What meta-analysis estimates

It can improve precision, describe variation among studies and provide an overall estimate for evidence-based decisions. This page explains effect measures, weighting, fixed-effect and random-effects models, heterogeneity, prediction intervals, bias assessment and the transfer of results into health-economic models.

Meta-analysis is not simply an arithmetic average of study results. It requires a defensible review question, compatible estimands, correct within-study uncertainty and a model that matches the intended interpretation.

The review question determines what can be combined

Studies should be combined only after defining the population, interventions, comparators, outcomes, time points and eligible designs. Statistical similarity cannot compensate for clinically different questions. The protocol should state which effect measure and follow-up time represent the estimand before results are known.

Important alignment checks include:

• The populations have sufficiently comparable disease stage, baseline risk and treatment history.
• The interventions and comparators have compatible doses, schedules and co-interventions.
• The outcome definitions measure the same construct on a compatible scale.
• The follow-up times represent comparable clinical periods.
• The study designs estimate effects with a compatible causal interpretation.
• The direction of benefit and harm is coded consistently.

Broad eligibility can be appropriate when heterogeneity is expected and clinically meaningful, but the model and interpretation must acknowledge that scope.

Each study contributes an estimate and uncertainty

A conventional aggregate-data meta-analysis uses one effect estimate (y_i) and sampling variance (v_i) from each independent study. More precise studies generally receive greater weight because their estimates have smaller sampling variance. The effect measure and variance calculation must be appropriate to the outcome and design.

Common effect measures include:

Ratio measures are usually analysed on the logarithmic scale because their sampling distributions are more symmetric and the null becomes zero. The pooled result and interval are then exponentiated for reporting.

Inverse-variance weighting gives precise studies more influence

Inverse-variance weighting assigns study (i) a weight proportional to the inverse of its estimated variance. The pooled estimate is a weighted mean, and the weights sum to one after normalisation. A large study does not automatically receive the greatest weight if its effect estimate is imprecise because of rare events, clustering or other design features.

Under a fixed-effect model:

$$w_i=\frac{1}{v_i}$$

The pooled effect is:

$$\hat\mu= \frac{\sum_{i=1}^{k}w_i y_i} {\sum_{i=1}^{k}w_i}$$

Its standard error is:

$$SE(\hat\mu)= \sqrt{\frac{1}{\sum_{i=1}^{k}w_i}}$$

A normal-approximation confidence interval is:

$$\hat\mu\pm z_{1-\alpha/2}SE(\hat\mu)$$

Alternative small-sample or robust methods may be preferable when the number of studies is limited or assumptions are uncertain.

A worked inverse-variance example

Suppose three studies estimate mean differences of 2.0, 3.0 and 5.0 with variances 1.0, 4.0 and 1.0. The first and third studies are more precise and therefore receive larger weights. The calculation shows why the pooled result is not the simple mean of 3.33.

The inverse-variance weights are:

$$w_1=1, \qquad w_2=0.25, \qquad w_3=1$$

The pooled estimate is:

$$\hat\mu= \frac{(1\times2.0)+(0.25\times3.0)+(1\times5.0)} {1+0.25+1} =3.44$$

The standard error is:

$$SE(\hat\mu)= \sqrt{\frac{1}{2.25}} =0.667$$

The approximate 95% confidence interval is:

$$3.44\pm1.96\times0.667 =[2.14,\;4.75]$$

This calculation assumes one common underlying effect and correct independent variances. A random-effects analysis would add between-study variance to the weights and generally distribute weight more evenly.

A fixed-effect model estimates one common effect

A fixed-effect meta-analysis assumes that every study estimates the same underlying effect and that observed differences arise only from within-study sampling error. Its pooled estimate describes that common effect for the included studies. The model is appropriate only when a single common effect is scientifically credible for the estimand.

The sampling model can be written as:

$$y_i\sim\operatorname{Normal}(\mu,v_i)$$

where (\mu) is the common effect. The label fixed effect does not mean study populations or methods are identical; it means that any real effect differences are excluded from the model.

A random-effects model estimates a distribution of effects

A random-effects meta-analysis allows the true effect to vary across studies. It estimates a mean of the distribution of study-specific effects and a between-study variance. Its interpretation depends on the studies representing a meaningful distribution of settings rather than an arbitrary collection.

A hierarchical formulation is:

$$y_i\sim\operatorname{Normal}(\theta_i,v_i)$$ $$\theta_i\sim\operatorname{Normal}(\mu,\tau^2)$$

where (\tau^2) is the between-study variance. Random-effects inverse-variance weights are:

$$w_i^*=\frac{1}{v_i+\tau^2}$$

When (\tau^2>0), large studies remain influential but smaller studies receive relatively more weight than under the fixed-effect model. A random-effects model does not make heterogeneous studies comparable; it only represents variation in the effect estimates statistically.

Between-study variance needs careful estimation

The heterogeneity variance (\tau^2) is difficult to estimate precisely when few studies are available. Different estimators can produce materially different values, confidence intervals and study weights. The estimation method should be prespecified or justified rather than selected for a preferred pooled result.

Methods include restricted maximum likelihood, maximum likelihood, Paule-Mandel and method-of-moments estimators such as DerSimonian-Laird. Interval estimation for the pooled effect may use a normal approximation, a Hartung-Knapp-type method or a Bayesian posterior framework.

The analyst should report:

• The estimator used for (\tau^2).
• The method used for the pooled-effect interval.
• Any constraints or prior distribution on heterogeneity.
• Sensitivity to alternative defensible methods when results are unstable.

Heterogeneity statistics describe different features

Heterogeneity should be evaluated clinically, methodologically and statistically. Statistical measures quantify aspects of observed variation but cannot explain its cause. Their uncertainty is often large when the number of studies is small.

Cochran's heterogeneity statistic is:

$$Q=\sum_{i=1}^{k}w_i(y_i-\hat\mu_F)^2$$

where (\hat\mu_F) is the fixed-effect estimate. The (I^2) statistic is commonly calculated as:

$$I^2=\max\left(0,\frac{Q-(k-1)}{Q}\right)\times100\%$$

A high (I^2) does not automatically prohibit pooling, and a low (I^2) does not establish homogeneity. The decision depends on the question, effect scale and consequences of averaging.

A prediction interval describes a future setting

A confidence interval around the random-effects mean quantifies uncertainty in the mean effect. A prediction interval estimates the range in which the true effect of a new sufficiently similar study or setting may lie, incorporating between-study heterogeneity. It is often more relevant when an economic model is intended for a particular new jurisdiction or population.

An approximate prediction interval is:

$$\hat\mu\pm t^* \sqrt{\widehat{\tau}^2+SE(\hat\mu)^2}$$

The exact critical value and method vary. Prediction intervals are imprecise with few studies and inherit the assumed distribution of effects, so they should not be presented as firm bounds.

Binary outcomes require an appropriate effect and zero-event method

Binary outcomes can be pooled as risk differences, risk ratios or odds ratios. These measures have different statistical and decision interpretations, especially when baseline risk varies. The chosen effect should match the causal assumption and the needs of downstream absolute-risk estimation.

Studies with zero events in one or both arms require particular care. Adding a continuity correction can materially affect results when events are rare or groups are imbalanced. Depending on the estimand and data, alternatives include likelihood-based models, beta-binomial approaches, exact methods or risk-difference analyses.

A study with zero events in both arms provides no information about a relative odds or risk ratio under common methods but may contain information about absolute risk. Excluding it silently can change the estimand and should be documented.

Continuous outcomes can use different scales

When studies measure the same outcome with the same instrument, a mean difference preserves the original unit and is usually easy to interpret. When different instruments measure a common construct, a standardised mean difference expresses effects in standard-deviation units. This requires the standard deviations to be comparable measures of variability rather than consequences of differing reliability or population heterogeneity.

A common standardised effect is:

$$SMD_i=J_i \frac{\bar X_{1i}-\bar X_{0i}} {S_{pooled,i}}$$

where (J_i) is a small-sample bias correction. A standardised effect may need a defensible transformation before it can enter an economic model requiring a natural unit, response probability or utility change.

Change scores and final values can sometimes estimate a common effect but require correct variances and compatible interpretation. Missing within-person correlations should be explored through sensitivity analysis rather than assigned without comment.

Time-to-event outcomes need consistent hazard-ratio interpretation

Time-to-event meta-analysis commonly pools log hazard ratios and their standard errors. This approach assumes that the hazard ratio provides a meaningful summary over the relevant follow-up and that effect direction is consistent. Non-proportional hazards can make a single hazard ratio difficult to interpret or unsuitable for extrapolating survival in an economic model.

If (HR_i) is the study hazard ratio:

$$y_i=\log(HR_i)$$

The pooled log hazard ratio is exponentiated after analysis:

$$HR_{pooled}=\exp(\hat\mu)$$

When proportional hazards are questionable, alternative estimands such as restricted mean survival difference or time-varying effects may be more appropriate.

Correlated estimates cannot be treated as independent

Some studies contribute more than one relevant estimate, such as multiple treatment arms, time points, outcomes or subgroups. These estimates share participants and are correlated. Treating them as separate independent studies double counts information and can make intervals too narrow.

Approaches include:

• Selecting one prespecified estimate per study.
• Combining eligible arms without creating unit-of-analysis error.
• Using a multivariate meta-analysis with an appropriate covariance matrix.
• Using robust variance estimation when its assumptions and study count are adequate.
• Fitting a hierarchical model that represents estimates nested within studies.

The analysis should preserve the randomised comparison and avoid counting one control group several times without adjustment.

Meta-regression examines study-level effect variation

Meta-regression relates study effect estimates to study-level characteristics. It can explore whether effect size varies with baseline risk, dose, follow-up or another prespecified modifier. It is observational across studies and can be confounded even when every included study is randomised.

A random-effects meta-regression is:

$$y_i=\beta_0+\beta_1x_i+u_i+\varepsilon_i$$

where:

$$u_i\sim\operatorname{Normal}(0,\tau^2)$$

and:

$$\varepsilon_i\sim\operatorname{Normal}(0,v_i)$$

Study-level associations do not establish patient-level treatment-effect modification. Meta-regression generally requires more studies than a simple pooled analysis and should avoid data-driven searches across many covariates.

Subgroup analysis can generate misleading differences

Separate significance within one subgroup and non-significance within another does not establish that subgroup effects differ. The relevant test is an interaction or between-subgroup comparison. Subgroup definitions and hypotheses should be prespecified and supported by a plausible effect-modification mechanism.

Subgroup credibility is stronger when:

• The subgroup variable is measured before treatment.
• The direction of modification is prespecified.
• The comparison is within rather than only between studies.
• The interaction is consistent across studies.
• The number of hypotheses is limited.
• The effect is clinically plausible and supported by related evidence.

Subgroup analyses with few studies produce unstable heterogeneity estimates and should not be treated as definitive inputs without sensitivity analysis.

Forest plots show estimates, uncertainty and weight

A forest plot displays study effect estimates and intervals on a common scale, often with marker size reflecting study weight. It also shows the pooled estimate and may show a prediction interval. The plot supports interpretation but does not diagnose all bias or heterogeneity.

A well-labelled forest plot should identify:

• The effect measure and outcome direction.
• The null value.
• Study estimates and confidence intervals.
• Study weights and pooling model.
• The pooled estimate and interval.
• Heterogeneity estimates.
• The analysis population, follow-up time and subgroup when relevant.

Truncated axes, inconsistent scales or unclear event direction can visually exaggerate or reverse effects.

Publication bias and small-study effects need more than a funnel plot

Evidence may be missing because studies, outcomes or analyses were not reported. Funnel-plot asymmetry can reflect reporting bias, but it can also arise from heterogeneity, chance, design differences or an association between study size and population. Visual or statistical asymmetry tests do not prove publication bias.

Assessment may include:

• Searches of trial registries, protocols and regulatory records.
• Comparison of prespecified and reported outcomes.
• Funnel plots when enough studies exist for meaningful interpretation.
• Statistical tests for small-study effects with recognised limitations.
• Selection models or sensitivity analyses under explicit assumptions.
• Consideration of how missing evidence could change the decision.

Methods such as trim-and-fill rely on strong assumptions and should not be presented as correcting publication bias automatically.

Risk of bias affects more than study inclusion

Meta-analysis combines estimates, but it does not neutralise bias within the studies. A precise pooled result can be precisely wrong when the contributing evidence is systematically biased. Risk-of-bias judgments should inform sensitivity analysis, evidence certainty and interpretation.

Potential actions include:

• Excluding studies with a prespecified critical problem in sensitivity analysis.
• Separating estimates by risk-of-bias domain when scientifically justified.
• Using bias-adjustment models under explicit elicited assumptions.
• Downgrading certainty when bias threatens the pooled estimate.
• Avoiding numerical quality scores that conceal which biases matter.

The primary analysis should not be chosen after comparing which risk-of-bias rule produces the preferred result.

Individual participant data can answer different questions

An individual participant data meta-analysis uses participant-level records from the included studies. It can harmonise outcomes, model participant-level interactions, apply common analysis methods and improve handling of time-to-event or repeated outcomes. It remains vulnerable to unavailable studies or incomplete participant data.

One-stage approaches model participants nested within studies in a single hierarchical model. Two-stage approaches estimate study-specific effects from harmonised participant data and then pool those estimates. Both should preserve within-study randomisation and distinguish participant-level from study-level effect modification.

Economic models need absolute outcomes and joint uncertainty

Meta-analysis commonly provides a relative treatment effect, while an economic model requires absolute event probabilities, survival, resource use or utility. The relative estimate must be combined with a relevant baseline risk or natural-history model on the correct effect scale. Applying an odds ratio as a risk ratio or ignoring non-proportional hazards can distort cost-effectiveness results.

For a pooled log odds ratio (\hat\mu) and baseline probability (p_0):

$$OR=\exp(\hat\mu)$$ $$p_1= \frac{OR\,p_0} {1-p_0+OR\,p_0}$$

Probabilistic sensitivity analysis should sample the pooled effect using its uncertainty distribution and preserve covariance with other jointly estimated parameters. The analyst should decide whether the decision requires uncertainty in the mean effect, predictive heterogeneity for a new setting or both.

Common analysis and interpretation errors

Meta-analysis can produce a pooled estimate even when the question, data or model are unsuitable. The most important errors involve incompatible estimands, incorrect independence assumptions and overinterpretation of statistical summaries. The following failures can materially affect a health-economic decision.

• Pooling clinically different outcomes or time points produces an estimate without a coherent meaning.
• Using an arithmetic mean of study estimates ignores precision and study-specific variance.
• Choosing fixed or random effects solely from a heterogeneity test reduces a scientific decision to a low-power statistical rule.
• Treating a random-effects model as permission to pool anything confuses statistical variance with clinical comparability.
• Interpreting (I^2) as the percentage of studies that disagree misstates the statistic.
• Reporting the confidence interval for the mean as the range of effects in settings ignores prediction uncertainty.
• Treating several estimates from one study as independent double counts participants.
• Applying continuity corrections without sensitivity analysis in rare-event data can drive the pooled effect.
• Interpreting funnel-plot asymmetry as proof of publication bias ignores alternative causes.
• Transferring a relative effect to an unrelated baseline risk or scale can produce incorrect absolute economic-model inputs.

A practical validation sequence

Validation should begin with the estimand and study data before testing the pooled calculation. Every study estimate, variance and weight should be traceable, and the economic-model transformation should be independently reproduced. The following sequence provides an auditable minimum.

1. Define the population, intervention, comparator, outcome, time point and effect measure before extracting results.
2. Verify study eligibility and reconcile duplicate publications or overlapping populations.
3. Recalculate representative study effect estimates and variances from source data when possible.
4. Confirm effect direction and analysis scale across every study.
5. Identify multi-arm, repeated or otherwise correlated estimates and model their covariance or select one prespecified estimate.
6. Reproduce the inverse-variance weights and pooled estimate independently.
7. Estimate and report heterogeneity with its uncertainty and examine clinical and methodological causes.
8. Compare defensible fixed-effect, random-effects and interval methods when conclusions are method sensitive.
9. Report a prediction interval when variation across future settings is relevant and estimable.
10. Test risk-of-bias, rare-event, missing-data and reporting-bias assumptions through planned sensitivity analyses.
11. Validate the transformation from relative to absolute effect before transferring results into an economic model.
12. Propagate the appropriate effect and heterogeneity uncertainty without double counting or assuming independence incorrectly.

Meta-analysis is most useful when it produces a pooled estimate with a clear estimand and transparent limits. Precision gains are credible only when study differences, dependence, bias and downstream interpretation are handled explicitly.