Simpson Paradox

Simpson Paradox in Treatment Comparisons and Pooled Evidence

Simpson's paradox, also called the reversal paradox or the amalgamation paradox, describes data in which one treatment does better than another inside every subgroup of patients and yet worse, or no differently, when all patients are counted together. In health economics it appears when patients are spread unevenly across subgroups that also predict outcome or cost, for example when the hardest cases are steered to one treatment, or when trials with unequal arms are added up as if they were a single trial. Incremental costs and effects in a cost-effectiveness analysis are differences between arms, so a reversal of this kind can make an option look dominated when, compared like with like, it buys extra benefit at a price. The arithmetic is simple, but choosing between the pooled and the subgroup comparison needs knowledge of how the data arose.

Why pooled totals can point the other way

A crude success rate for a treatment is an average of its subgroup rates, weighted by how that treatment's own patients are distributed across the subgroups. Two treatments compared on crude rates are therefore compared on different weights. When one arm holds far more of the high-risk subgroup, the gap in weights can outweigh every subgroup difference and flip the sign of the comparison.

$$\bar{p}_1 - \bar{p}_0 = \sum_{s} \pi_{s \mid 1}\,(p_{s1} - p_{s0}) + \sum_{s} (\pi_{s \mid 1} - \pi_{s \mid 0})\, p_{s0}$$

where $\bar{p}_a = \sum_s \pi_{s \mid a}\, p_{sa}$ is the crude rate in arm $a$ (1 for the index treatment, 0 for the comparator), $p_{sa}$ is the rate in subgroup $s$ under arm $a$, and $\pi_{s \mid a}$ is the share of arm $a$'s patients who fall in subgroup $s$. The first sum is the within-subgroup effect, weighted by the index arm's case mix. The second sum is a composition term, zero when both arms share the same mix.

A reversal needs both a difference in mix between arms and a difference in baseline rates between subgroups. If either is absent the composition term vanishes. The same logic applies to mean costs.

Kidney stone treatment as the standard clinical case

Charig and colleagues compared treatments for renal calculi to establish which was the most cost effective and successful. Success, meaning stones eliminated or reduced to less than 2 mm after three months, was reported for 273 (78%) patients after open surgery and 289 (83%) after percutaneous nephrolithotomy. Bonovas and Piovani tabulate both arms by stone size.

Open surgery has the higher success rate for small stones and for large stones, but the lower rate overall. The reason is allocation. Most patients with small stones (270 of 357, 75.6%) had percutaneous nephrolithotomy, while most with large or multiple stones (263 of 343, 76.7%) had open surgery, and large stones are harder to clear whichever method is used.

With open surgery as arm 1, the within-subgroup term is 0.2486 × 0.0644 + 0.7514 × 0.0425 = 0.0480, a gain of about 4.8 percentage points. The composition term is 0.5229 × (0.6875 − 0.8667) = −0.0937, because open surgery holds 52.3 percentage points more of its patients in the large-stone group, where success under the comparator is 17.9 points lower. The two terms add to −0.0457, the crude gap of 78.0% against 82.6%.

Which comparison to believe depends on causal structure

Hernán, Clayton and Keiding return to Simpson's own 1951 example, a hypothetical set of 52 individuals and three binary variables, in which two variables had an odds ratio of 1 overall but a common odds ratio different from 1 within each level of the third. Simpson attached two stories to identical numbers. In one, a pack of playing cards had been dirtied by a baby who preferred court cards and red cards, and the sensible measure was the marginal one. In the other, the variables were a medical treatment, death and sex, and the sensible measure was the conditional one.

Their explanation is causal. Dirtiness is a common effect, or collider, of card type and colour, and conditioning on a collider creates an association between its causes, which epidemiologists call selection bias. Sex is a common cause of treatment and death, a confounder, and conditioning on it removes confounding. The data cannot tell the two structures apart, so subject-matter knowledge has to decide. For 2 × 2 tables they reduce the guidance to conditioning only on confounders, with the caveat that the rule does not generalise to complex longitudinal settings with time-varying treatments and confounders.

They note that Simpson's article contained no reversal of direction, though many later authors reserve the name for that case, and they separate out a second source of apparent paradox, non-collapsibility. The risk ratio and risk difference are collapsible, meaning the marginal measure can be written as a weighted average of the conditional measures, but the odds ratio is not. In a randomised controlled trial with no confounding and no effect modification, conditional odds ratios can still differ from the marginal odds ratio, and the authors call that difference a mathematical oddity that reflects no bias. Both odds ratios are then valid, and they simply differ.

Adding up trials as if they were one

Evidence synthesis has its own version. Each trial is randomised internally, but arms are sometimes unequal, most often because similar active arms are combined against one control. Chapter 15 of the Cochrane Handbook advises against computing risk differences and numbers needed to treat from aggregated totals across trials, which ignores the randomisation within studies and may seriously mislead if any study used unbalanced randomisation.

Cates took an example from a Cochrane review of high intensity nursing support for smoking cessation. Adding the raw totals gave cessation in 546 of 3,820 patients (14.3%) with nursing support against 356 of 2,301 (15.5%) without, which implies harm with a number needed to harm of 83. The weighted meta-analysis of the individual trials gave a pooled risk difference of 0.037 in favour of nursing support, a number needed to treat of 27. One trial had roughly three times as many patients in its combined nursing arms as in its control group.

Altman and Deeks describe the effect as a bias more than a true paradox. Two nicotine gum trials, each with a risk difference of 0.06, gave a pooled risk difference of 0.060 by the Mantel-Haenszel method but 0.033 by the treat-as-one-trial method, so the number needed to treat rose to 31, against 17 by the standard method. Across all 51 trials in that review, where most trials were balanced, the two methods gave 0.061 and 0.081, and the confidence intervals barely overlapped. They state that the risk grows with the discrepancy in group sizes, the variation in control group event rates and the heterogeneity in treatment effects, and that standard stratified methods are not affected. Both papers favour pooling a relative effect by trial and applying it to the baseline risk of the patients concerned, as a decision model does.

Worked example of a reversed cost-effectiveness comparison

The success counts are the Charig data above. The costs per patient by stone size are illustrative and not taken from the study. Assume open surgery costs GBP 3,200 for a small stone and GBP 3,800 for a large one, and percutaneous nephrolithotomy GBP 1,900 and GBP 2,500. The outcome is a stone-free patient, so the result is a cost per additional success, not a cost per QALY.

1. Crude comparison. Open surgery treats 87 of its 350 patients for small stones (24.86%) and 263 for large ones (75.14%), so its mean cost is 0.2486 × 3200 + 0.7514 × 3800 = 3650.9. Percutaneous nephrolithotomy treats 270 small (77.14%) and 80 large (22.86%), so its mean cost is 0.7714 × 1900 + 0.2286 × 2500 = 2037.1. Open surgery costs GBP 1,614 more and succeeds 4.57 percentage points less often, so on crude data it appears dominated, the case described under dominance.

2. Choose a standard case mix. Taking the 700 patients in the two arms together, 357 had small stones and 343 large, giving weights of 0.51 and 0.49.

$$p^{\text{std}}_a = \sum_s w_s\, p_{sa}$$

where $p^{\text{std}}_a$ is the standardised success rate under arm $a$, $w_s$ is the share of the standard population in subgroup $s$ and $p_{sa}$ is the observed success rate in subgroup $s$ under arm $a$. Mean costs are standardised with the same weights.

3. Standardised effects. For open surgery 0.51 × 0.9310 + 0.49 × 0.7300 = 0.8325, and for percutaneous nephrolithotomy 0.51 × 0.8667 + 0.49 × 0.6875 = 0.7789. Open surgery now succeeds 5.37 percentage points more often, about 53.7 extra stone-free patients per 1,000 treated.

4. Standardised costs. For open surgery 0.51 × 3200 + 0.49 × 3800 = 3494, and for percutaneous nephrolithotomy 0.51 × 1900 + 0.49 × 2500 = 2194, an increment of GBP 1,300 per patient.

5. Ratio. Dividing GBP 1,300 by 0.0537 gives about GBP 24,200 per additional stone-free patient.

6. Interpretation. On like-for-like terms open surgery is more effective and more costly, and the question becomes whether an extra stone-free patient is worth about GBP 24,200. The reading holds only if no factor other than stone size both guided the procedure and affects outcome or cost. Hernán and Robins, in Causal Inference, What If, note that a standardised risk is the risk had everyone received that treatment only under conditional exchangeability, so an unmeasured factor such as surgeon preference would leave bias. The result also depends on the standard, and weights from the population the decision is for, as in an ICER for one health system, are more relevant than the study's own mix.

Provider comparisons and grouping variables

The subgroup can also be a place. In a multicentre study of hospital infections by Reintjes and colleagues, re-tabulated by Bonovas and Piovani, antibiotic prophylaxis was associated with fewer urinary tract infections across all eight Dutch hospitals combined (3.3% against 4.6%, risk ratio 0.71). Within low-incidence hospitals the rates were 1.8% against 0.7% (risk ratio 2.59), and within high-incidence hospitals 13.3% against 6.5% (risk ratio 2.03). Prophylaxis was given to 60.7% of patients in low-incidence hospitals but 9.8% in high-incidence ones, so hospital acted as a confounder linked to both exposure and outcome.

Provider comparisons of cost and outcome carry this risk whenever case mix differs, the problem that case-mix adjustment and wider risk adjustment address. When the data are group averages, such as regional spending against regional outcomes, the problem is confounding by aggregation.

Related ideas and where they differ

Confounding by stone size, by hospital or by trial drives the examples above, but a collider structure or the non-collapsibility of the odds ratio can produce the same gap between marginal and conditional measures. Hernán, Clayton and Keiding argue that equating the paradox with confounding misses Simpson's main point, that statistical reasoning alone cannot choose between the two measures.

Effect modification means that the effect itself differs between subgroups on a stated measure, a real feature to report through subgroup analysis. In the kidney stone data the effect points the same way in both groups, and only the allocation changes the crude result. Confounding by indication, where the condition that prompts a treatment also predicts the outcome, is a common clinical source of that uneven allocation. Choosing what to adjust for belongs to causal inference.

Common misreadings

Most errors with Simpson's paradox come from choosing between the pooled and the stratified table on statistical grounds alone. Each one treats a statistical pattern as if it settled a causal question.

1. Always trusting the stratified result. Conditioning on a collider creates bias, so for 2 × 2 tables Hernán, Clayton and Keiding advise conditioning only on confounders.
2. Reading a crude and adjusted odds ratio gap in a randomised trial as bias. Non-collapsibility produces such gaps without confounding.
3. Assuming randomised evidence is immune. Adding arms across trials compares treated patients in one trial with controls in others, which Cates notes loses the benefit of randomisation.
4. Treating one stratifier as enough. A standardised comparison removes only the confounding by the variables used.

Sources

• Altman DG, Deeks JJ. Meta-analysis, Simpson's paradox, and the number needed to treat. BMC Medical Research Methodology. 2002;2:3. https://doi.org/10.1186/1471-2288-2-3
• Bonovas S, Piovani D. Simpson's paradox in clinical research: a cautionary tale. Journal of Clinical Medicine. 2023;12(4):1633. https://doi.org/10.3390/jcm12041633
• Cates CJ. Simpson's paradox and calculation of number needed to treat from meta-analysis. BMC Medical Research Methodology. 2002;2:1. https://doi.org/10.1186/1471-2288-2-1
• Charig CR, Webb DR, Payne SR, Wickham JE. Comparison of treatment of renal calculi by open surgery, percutaneous nephrolithotomy, and extracorporeal shockwave lithotripsy. British Medical Journal (Clinical Research Edition). 1986;292(6524):879-882. https://doi.org/10.1136/bmj.292.6524.879
• Hernán MA, Clayton D, Keiding N. The Simpson's paradox unraveled. International Journal of Epidemiology. 2011;40(3):780-785. https://doi.org/10.1093/ije/dyr041
• Hernán MA, Robins JM. Causal Inference: What If. Boca Raton: Chapman & Hall/CRC; 2020 (online edition of 19 August 2026), chapter 2. https://miguelhernan.org/whatifbook
• Reintjes R, de Boer A, van Pelt W, Mintjes-de Groot J. Simpson's paradox: an example from hospital epidemiology. Epidemiology. 2000;11(1):81-83. https://pubmed.ncbi.nlm.nih.gov/10615849/
• Schünemann HJ, Vist GE, Higgins JPT, et al. Chapter 15: Interpreting results and drawing conclusions (last updated August 2023). In: Higgins JPT, Thomas J, Chandler J, et al., eds. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. Section 15.4.4. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-15