Mean Difference

Mean Difference in Trials, Meta-Analysis and Cost-Effectiveness Analysis

A mean difference keeps a treatment effect in the natural units of the outcome, such as millimetres of mercury, days in hospital, pounds sterling or QALYs, which is why it is the default effect measure when every study used the same scale. In evidence synthesis it is the quantity pooled across trials of a continuous outcome, and in economic evaluation the difference in mean costs and the difference in mean QALYs are the two halves of the incremental cost-effectiveness ratio. Because it is an arithmetic mean, it answers the question a budget holder asks, namely how much a whole treated population will gain or cost on average, and it inherits the sensitivity of the mean to skewed data.

Difference in means within one trial

The Cochrane Handbook describes the mean difference as the absolute difference between the mean values in two groups of a randomised controlled trial, estimating how much the experimental intervention changes the outcome on average compared with the comparator. It notes that "difference in means" is the more correct name. For one study the estimate and its standard error are computed from the group summaries.

$$MD_i = \bar{x}_{1i} - \bar{x}_{2i}, \qquad SE(MD_i) = \sqrt{\frac{s_{1i}^2}{n_{1i}} + \frac{s_{2i}^2}{n_{2i}}}$$

where $MD_i$ is the mean difference in study $i$, $\bar{x}_{1i}$ and $\bar{x}_{2i}$ are the mean outcomes in the experimental and comparator groups, $s_{1i}$ and $s_{2i}$ are their standard deviations and $n_{1i}$ and $n_{2i}$ are the group sizes. This is the form used by RevMan, the Cochrane review software. The sign follows the order of subtraction, so a review has to state whether a negative value means benefit or harm on its outcome.

Cochrane reviews used to call these analyses weighted mean difference (WMD) analyses. The Handbook now calls that name potentially confusing, because no weighting is involved in summarising a single study and every meta-analysis weights its studies, so "WMD" in older reviews simply means a pooled mean difference.

When a trial reports a 95% confidence interval for the mean difference instead of group standard deviations, the Handbook recovers the standard error by dividing the width of the interval by 3.92 in large trials, with a larger divisor from the t distribution when groups are small.

Pooling mean differences by inverse variance

A meta-analysis of mean differences gives each study a weight equal to the inverse of the variance of its estimate, so larger and more precise studies count for more. The Cochrane Handbook explains that this choice of weights minimises the imprecision of the pooled estimate.

$$\widehat{MD} = \frac{\sum_i w_i MD_i}{\sum_i w_i}, \qquad w_i = \frac{1}{SE(MD_i)^2}, \qquad SE(\widehat{MD}) = \sqrt{\frac{1}{\sum_i w_i}}$$

where $\widehat{MD}$ is the pooled mean difference and $w_i$ is the weight of study $i$. Under a fixed-effect model the studies are taken to estimate one common effect. A random-effects model adds the between-study variance, tau-squared, to each study's variance before weighting; in the DerSimonian and Laird version tau-squared is estimated from the heterogeneity statistic $Q$ and set to zero when $Q$ is no larger than its degrees of freedom.

The standard deviations play a narrower role here than in a standardised mean difference analysis. In a mean difference analysis they only set the weights, so a study with a small standard deviation gains weight. The Handbook regards this as appropriate when the spread differs because outcome measurement is more or less reliable, and probably not appropriate when it reflects genuinely more varied patients.

Change scores, final values and adjusted estimates

Trials report continuous outcomes as values after treatment, as changes from baseline or as estimates adjusted for baseline. The Handbook states that mixing post-intervention values and change scores is not a problem in a meta-analysis of mean differences, because in a randomised trial both can usually be assumed to address the same underlying effect, though it may be clearer to show them as subgroups. It prefers estimates from analysis of covariance with the baseline value as a covariate, as these give the most precise and least biased estimates, and they enter the analysis through the generic inverse-variance method.

Worked example with three blood pressure trials

All numbers in this example are illustrative. Three placebo-controlled trials of an antihypertensive report systolic blood pressure at 12 weeks in mmHg, so the outcome is on the same scale in every study and a mean difference can be pooled directly.

1. Compute each trial's standard error. For trial A the variance of the difference is $\frac{196}{50} + \frac{225}{50} = 8.42$, so the standard error is the square root of 8.42, which is 2.902, and the weight is $\frac{1}{8.42} = 0.1188$. Trials B and C follow the same steps, with variances of 4.582 and 2.16.

2. Pool by inverse variance. The weights sum to 0.7999 and the weighted differences sum to -3.0835.

$$\widehat{MD} = \frac{-3.0835}{0.7999} = -3.855, \qquad SE(\widehat{MD}) = \sqrt{\frac{1}{0.7999}} = 1.118$$

where the pooled value is in mmHg. The 95% interval is -3.855 plus or minus 1.96 times 1.118, or -6.05 to -1.66 mmHg.

3. Check heterogeneity. The heterogeneity statistic is $Q = 0.98$ on 2 degrees of freedom. Because $Q$ is below its degrees of freedom, the DerSimonian and Laird tau-squared is zero, I-squared is 0% and the random-effects result equals the fixed-effect result here.

4. Interpret. The treatment lowers systolic pressure by about 3.9 mmHg on average, and trial C, with the largest sample, carries most of the weight. A decision model needs the pooled difference with its standard error of 1.118, because the NICE manual (PMG36, section 4.7.10) asks for distributions that describe the precision of mean parameter values in probabilistic sensitivity analysis.

Mean costs, mean QALYs and the ICER

In a trial-based economic evaluation, the incremental cost is a difference in mean costs and the incremental effectiveness is usually a difference in mean QALYs. The NICE manual (PMG36, section 4.2.16) states that reported ICERs must be the ratio of expected additional total cost to expected additional QALYs, and an expected value is a population mean.

$$ICER = \frac{\bar{C}_1 - \bar{C}_0}{\bar{E}_1 - \bar{E}_0}$$

where $\bar{C}_1$ and $\bar{C}_0$ are mean costs per patient with the new treatment and the comparator, and $\bar{E}_1$ and $\bar{E}_0$ are mean QALYs per patient. The same two mean differences also give the incremental net benefit once the QALY difference is valued at a threshold.

Thompson and Barber argue that the arithmetic mean is the informative measure for cost data despite their usual skewness, because the total budget needed equals the mean cost multiplied by the number of patients treated, and no other summary gives that total. Their review of 45 trials with economic evaluations published in 1995 found that only about half of the papers reporting statistical comparisons used methods that addressed differences in arithmetic means. In their menorrhagia example the mean cost was GBP 790 with endometrial resection and GBP 1,110 with hysterectomy, a difference of GBP 320, while the median cost in the resection group was only GBP 523. The 95% interval for the GBP 320 difference was GBP 204 to GBP 437 by a t test method and GBP 192 to GBP 426 by bootstrapping, so with 78 and 70 patients the simple method held up despite the skew.

Illustrative cost and QALY comparison

All numbers are illustrative. A new service has mean cost GBP 4,250 per patient (standard deviation GBP 5,100) against GBP 3,400 (standard deviation GBP 3,900) for usual care, and mean QALYs of 1.412 against 1.367 over the trial.

$$ICER = \frac{4250 - 3400}{1.412 - 1.367} = \frac{850}{0.045} = 18889$$

where the result is in GBP per QALY gained. If the median costs were GBP 3,100 and GBP 3,050, a difference of GBP 50, a ratio built on medians would be $\frac{50}{0.045} = 1111$ GBP per QALY and would understate the extra spending seventeenfold. The Cochrane Handbook gives a rough skew check for outcomes with a lowest possible value, the mean minus that value divided by the standard deviation, where a ratio below 1 is strong evidence of skew. For the new service it is $\frac{4250}{5100} = 0.83$, so the cost data are clearly skewed, yet the mean difference remains the quantity the decision needs and its interval is better checked by bootstrapping than replaced by a comparison of medians.

Baseline imbalance matters for the QALY half. Manca, Hawkins and Sculpher note that mean baseline utility is invariably imbalanced between trial arms and that failing to control for it can give a misleading ICER. They demonstrate that multiple regression controlling for baseline utility produces appropriate estimates of the difference in mean QALYs, the economic counterpart of the Cochrane preference for baseline-adjusted estimates.

Reading a mean difference against the minimal important difference

A mean difference in scale points needs a yardstick before patients or decision makers can judge it. The usual yardstick for a patient-reported outcome is the minimal clinically important difference. Chapter 15 of the Cochrane Handbook describes dividing each study's mean difference by the instrument's minimal important difference instead of by its standard deviation, so that results pool in minimal important difference units without the problem of standard deviations that vary across studies. It warns that a mean difference below the minimal important difference may be read as trivial when a substantial proportion of patients may have achieved an important benefit, because a mean says nothing about how the gain is spread across individuals.

Mean difference compared with nearby measures

The mean difference is a difference measure, and the Cochrane Handbook also calls such measures absolute, in contrast to ratio measures such as the risk ratio and odds ratio. When studies measure the same construct on different instruments, a standardised mean difference divides each study's mean difference by a standard deviation first, and Hedges' g and Cohen's d are versions of it, while effect size is the wider family. The ratio of means is a third option for continuous data but suits only outcomes that are strictly greater than zero.

Limitations and common misreadings

Most errors with mean differences come from what the mean hides. A mean difference carries its units but says nothing about how large a change on that scale matters to patients, and nothing about how the effect is spread across them.

1. Reading a pooled mean difference across different instruments. It is only valid when every study used the same scale; otherwise a standardised measure or a rescaling is needed.
2. Replacing mean costs with medians or geometric means. These answer a different question and cannot be multiplied up to a budget.
3. Ignoring skew in small trials. Analyses based on means assume approximately normal outcomes or very large trials, so small skewed trials need a bootstrap check of the interval.
4. Ignoring baseline imbalance. Unadjusted differences in mean QALYs can mislead the ICER when baseline utility differs between arms.
5. Treating a mean below the minimal important difference as no benefit for anyone. A modest average can conceal a substantial share of patients with an important gain.

Sources

• Higgins JPT, Li T, Deeks JJ (editors). Chapter 6: Choosing effect measures and computing estimates of effect. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 6.1.2, 6.3.1, 6.5.1 and 6.5.2. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-06
• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.3 and 10.5. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
• Schünemann HJ, Vist GE, Higgins JPT, Santesso N, Deeks JJ, Glasziou P, Akl EA, Guyatt GH. Chapter 15: Interpreting results and drawing conclusions. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Section 15.5.3.5. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-15
• Deeks JJ, Higgins JPT, on behalf of the Cochrane Statistical Methods Group. Statistical algorithms in Review Manager. Cochrane; May 2022. https://documentation.cochrane.org/revman-kb/files/210600101/210600103/1/1654774257333/Statistical_methods_in_revman.pdf
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320:1197-1200. https://doi.org/10.1136/bmj.320.7243.1197
• Barber JA, Thompson SG. Analysis and interpretation of cost data in randomised controlled trials: review of published studies. BMJ. 1998;317:1195-1200. https://doi.org/10.1136/bmj.317.7167.1195
• Manca A, Hawkins N, Sculpher MJ. Estimating mean QALYs in trial-based cost-effectiveness analysis: the importance of controlling for baseline utility. Health Economics. 2005;14(5):487-496. https://doi.org/10.1002/hec.944
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 2022, last updated 13 October 2026. Sections 4.2.16 and 4.7.10. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2