Standardised Mean Difference

Standardised Mean Difference in Evidence Synthesis and Real-World Evidence

A standardised mean difference (SMD) puts each trial's effect on a continuous outcome into standard deviation units, so that trials which measured pain, fatigue or quality of life on different instruments can be pooled in one meta-analysis. In health technology assessment it appears most often when patient-reported outcome scales differ between trials, and the pooled figure must be turned back into scale points, responders or utilities before a cost-effectiveness model can use it. The same calculation applied to baseline covariates is the usual check on whether propensity score matching has balanced treated and untreated groups in real-world evidence.

From a mean difference to standard deviation units

The Cochrane Handbook describes the mean difference as the absolute difference between the mean values in two groups, usable in meta-analysis when every study measured the outcome on the same scale. Cochrane reviews once called these weighted mean difference analyses, a name the Handbook now calls potentially confusing because the weighting belongs to the pooling step, not to any single study. When the studies assess the same outcome with different instruments, each study's difference is divided by a standard deviation first.

$$SMD = \frac{\bar{x}_1 - \bar{x}_2}{s}$$

where $\bar{x}_1$ and $\bar{x}_2$ are the mean outcomes in the experimental and comparator groups and $s$ is the chosen standard deviation. The method does not correct for scales that run in opposite directions, so the means from one set of studies must be multiplied by minus one before standardising.

Which standard deviation goes in the denominator

The members of the SMD family differ only in the standardiser, and the choice changes the number. An SMD cannot be checked or reused unless its standardiser is named.

The Cochrane Handbook states that its SMD is Hedges' adjusted g, which assumes the two groups have similar spread. An external standard deviation is typically treated as known without error, so the mean difference and its standard error are both divided by it.

Pooling trials that used different instruments

RevMan, the Cochrane review software, implements Hedges' adjusted g. It corrects each study's SMD for small-sample bias and gives it a standard error, as follows.

$$g_i = \frac{m_{1i} - m_{2i}}{s_i}\left(1 - \frac{3}{4N_i - 9}\right), \qquad SE(g_i) = \sqrt{\frac{N_i}{n_{1i} n_{2i}} + \frac{g_i^2}{2(N_i - 3.94)}}$$

where $g_i$ is the SMD for study $i$, $m_{1i}$ and $m_{2i}$ are the group means, $s_i$ is the pooled standard deviation, $n_{1i}$ and $n_{2i}$ are the group sizes and $N_i = n_{1i} + n_{2i}$. In a fixed-effect model each study is weighted by the reciprocal of its variance, $w_i = 1/SE(g_i)^2$; a random-effects model adds the between-study variance to each study's variance before weighting.

In a mean difference analysis the standard deviations only set the study weights, but in an SMD analysis they also rescale each study's effect, so a study with a small standard deviation produces a larger SMD. The Handbook says this is appropriate only if between-study variation in standard deviations reflects differences in measurement scales, not in the reliability of the instruments or the variability of the populations. Its example is explanatory trials pooled with pragmatic trials, whose wider range of patients may give larger standard deviations and smaller SMDs for the same true effect.

The same logic rules out mixing outcome types. Post-intervention values and change-from-baseline scores can be combined in a mean difference analysis, but the Handbook says they should not in principle be combined as SMDs, because one standard deviation describes people at a single time and the other describes how they changed. The ratio of means pools different scales without standard deviation units, but only for strictly positive outcomes.

Worked example with three fatigue trials

All numbers are illustrative. Three trials of a fatigue programme report post-treatment scores on different instruments, all oriented so that higher is better. Trial A used a 0 to 100 scale (40 per arm, means 62 and 55, standard deviations 18 and 20), trial B a 1 to 7 scale (60 per arm, means 4.6 and 4.2, standard deviations 1.1 and 1.2) and trial C a 0 to 52 scale (100 per arm, means 36 and 33, standard deviations 10 and 11).

1. Standardise each trial. For trial A the pooled standard deviation is the square root of $(39 \times 324 + 39 \times 400)/78 = 362$, which is 19.03, so the uncorrected SMD is $d_A = 7/19.03 = 0.368$ and the corrected value is $g_A = 0.368 \times (1 - 3/311) = 0.364$.

2. Pool by inverse variance. Under a fixed-effect model the pooled SMD is the weighted average of the three estimates.

$$\hat{g} = \frac{\sum_i w_i g_i}{\sum_i w_i} = \frac{31.434}{98.692} = 0.3185, \qquad SE(\hat{g}) = \sqrt{\frac{1}{98.692}} = 0.1007$$

where $\hat{g}$ is the pooled SMD and $w_i$ are the weights in the table. The 95% interval is 0.3185 plus or minus 1.96 times 0.1007, or 0.12 to 0.52 standard deviations.

3. Re-express on the familiar instrument. Multiplying by trial A's pooled standard deviation of 19.03 gives a gain of 6.1 points on the 0 to 100 scale, with an interval of 2.3 to 9.8 points. If the instrument's minimal important difference were 10 points, even the upper limit would fall short of it, which the bare figure of 0.32 does not show. The Handbook warns that a mean difference below the minimal important difference can still include a substantial proportion of patients with an important benefit.

4. Test the standardiser. Had trial C been a pragmatic trial with a pooled standard deviation of 14 instead of 10.51, the same 3-point difference would give an uncorrected SMD of $d_C = 3/14 = 0.214$ in place of 0.285. The pooled SMD would fall from 0.32 to 0.28 although the effect in scale points was unchanged.

Turning an SMD into inputs for a decision model

A cost-effectiveness model needs effects in units that carry costs and preference weights, such as event probabilities and utility values. A standard deviation unit carries neither, so an SMD has to be re-expressed before it can enter a cost-effectiveness analysis. Chapter 15 of the Cochrane Handbook gives three ways to re-express an SMD, as generic labels, in units of a familiar instrument or as an odds ratio, and two alternative measures, the ratio of means and minimal important difference units. The Cohen's d page works through the instrument and odds ratio routes as far as a QALY. In the Handbook's dexamethasone example, an SMD of 0.79 became 8.1 points less post-operative pain on a 0 to 100 scale, against a minimal important difference of about 10. A pooled SMD on a disease-specific scale reaches a QALY only through an external standard deviation and a mapping function, each adding uncertainty.

Standardised differences as a balance diagnostic

In observational studies the SMD is computed on baseline covariates. Austin defines the standardised difference for a continuous covariate and for a binary covariate as follows.

$$d_{cont} = \frac{\bar{x}_T - \bar{x}_C}{\sqrt{(s_T^2 + s_C^2)/2}}, \qquad d_{bin} = \frac{\hat{p}_T - \hat{p}_C}{\sqrt{\left[\hat{p}_T(1 - \hat{p}_T) + \hat{p}_C(1 - \hat{p}_C)\right]/2}}$$

where $\bar{x}_T$ and $\bar{x}_C$ are the covariate means in treated and untreated patients, $s_T^2$ and $s_C^2$ their variances, and $\hat{p}_T$ and $\hat{p}_C$ the prevalences of a binary covariate. Austin notes that, unlike a significance test, the standardised difference is not influenced by sample size.

In an illustrative registry study, mean age is 68 years in treated and 64 in untreated patients (standard deviations 9 and 11), giving $d = 4/\sqrt{101} = 0.40$. After matching, means of 66.2 and 65.6 with standard deviations of 9.5 and 10 give $d = 0.6/9.75 = 0.06$. Diabetes prevalence of 30% against 22% gives $d = 0.08/\sqrt{0.1908} = 0.18$ before matching, and 27% against 25% gives 0.05 after.

There is no agreed cut-off. Austin reports that a value below 0.1 has been taken to indicate a negligible difference, and the NICE real-world evidence framework (ECD9) says absolute values below 0.1 are generally considered good balance, though small differences may still matter for a variable with a strong effect on the outcome. Austin adds that balance is a large-sample property, so moderate imbalance can occur by chance in small matched samples, and that checks should go beyond means to variances and graphical comparisons.

Standardised mean difference compared with nearby measures

Effect size is the wider family, a term the Handbook avoids in reviews because it is also used loosely for any intervention effect. The mean difference keeps natural units and is the better choice whenever all studies used the same instrument. The standardised response mean divides a group's mean change by the standard deviation of that change to gauge an instrument's responsiveness, not a difference between groups.

Limitations and common misreadings

Most errors come from forgetting that an SMD mixes the size of an effect with the spread of the population. As a balance check it describes recorded covariates only.

1. Treating 0.2, 0.5 and 0.8 as decision thresholds. Generic labels say nothing about whether an effect justifies its cost.
2. Reading a larger SMD as a larger benefit. A narrow explanatory population inflates the SMD relative to a broad pragmatic one with the same difference in scale points.
3. Putting an SMD straight into a model. The external standard deviation and mapping choice used to re-express it should be stated and varied in sensitivity analysis.
4. Treating a balance SMD below 0.1 as proof of no confounding. Conditioning on the propensity score need not balance unmeasured covariates, so confounding by unrecorded variables remains possible.

Sources

• Higgins JPT, Li T, Deeks JJ (editors). Chapter 6: Choosing effect measures and computing estimates of effect. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Section 6.5.1. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-06
• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.3, 10.5 and 10.6. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
• Schünemann HJ, Vist GE, Higgins JPT, Santesso N, Deeks JJ, Glasziou P, Akl EA, Guyatt GH. Chapter 15: Interpreting results and drawing conclusions. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Section 15.5. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-15
• Deeks JJ, Higgins JPT, on behalf of the Cochrane Statistical Methods Group. Statistical algorithms in Review Manager. Cochrane; May 2022. https://documentation.cochrane.org/revman-kb/files/210600101/210600103/1/1654774257333/Statistical_methods_in_revman.pdf
• Austin PC. Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. Statistics in Medicine. 2009;28:3083-3107. https://doi.org/10.1002/sim.3697
• Austin PC. An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research. 2011;46(3):399-424. https://doi.org/10.1080/00273171.2011.568786
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). 2022. Methods for real-world studies of comparative effects. https://www.nice.org.uk/corporate/ecd9/chapter/methods-for-real-world-studies-of-comparative-effects
• Revicki DA, Cella D, Hays RD, Sloan JA, Lenderking WR, Aaronson NK. Responsiveness and minimal important differences for patient reported outcomes. Health and Quality of Life Outcomes. 2006;4:70. https://doi.org/10.1186/1477-7525-4-70
• Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: a practical primer for t-tests and ANOVAs. Frontiers in Psychology. 2013;4:863. https://doi.org/10.3389/fpsyg.2013.00863