Difference-in-Differences

Difference-in-Differences in Health Policy Evaluation

Difference-in-differences (DiD) is one of the main ways health economists estimate what a policy, payment reform or service change did when the change was not randomised. Fixed differences between groups and shocks common to both drop out, and the estimate rests mainly on one untestable assumption about what would have happened without the policy. An illustrative worked example, with a violated-trends variant, shows how the estimate feeds an economic evaluation.

The two-group, two-period estimator

The simplest design observes one group exposed to a policy and one unexposed group, each once before and once after the policy starts. The estimate is the change in the exposed group minus the change in the comparison group. In terms of group means it is written as follows.

$$\hat{\tau}_{DiD} = (\bar{Y}_{T,1} - \bar{Y}_{T,0}) - (\bar{Y}_{C,1} - \bar{Y}_{C,0})$$

where $\hat{\tau}_{DiD}$ is the DiD estimate, $\bar{Y}_{T,1}$ and $\bar{Y}_{T,0}$ are the treated group's mean outcomes after and before the policy, and $\bar{Y}_{C,1}$ and $\bar{Y}_{C,0}$ are the comparison group's. The first bracket removes what is fixed about the treated group, such as case mix; subtracting the second removes changes common to both, such as a national trend in admissions. A classic early application outside health compared fast-food employment in New Jersey, where the minimum wage rose in April 1992, with eastern Pennsylvania, where it did not.

The same estimate comes from a linear regression.

$$Y_{it} = \alpha + \gamma D_i + \lambda P_t + \tau (D_i \times P_t) + \varepsilon_{it}$$

where $Y_{it}$ is the outcome for unit $i$ in period $t$, $D_i$ equals 1 for the treated group, $P_t$ equals 1 after the policy, $\alpha$ is the comparison-group mean before the policy, $\gamma$ is the fixed gap between groups, $\lambda$ is the common change over time, $\tau$ is the DiD estimate and $\varepsilon_{it}$ is the error term.

Under parallel trends and no anticipation, the quantity identified is the average effect on the units that received the policy, the average treatment effect on the treated. It differs from the population average treatment effect when adopters differ in how much they would benefit.

The parallel trends assumption and how it is assessed

The estimate has a causal reading only if, without the policy, the treated group's outcome would have changed by the same amount as the comparison group's. This is the parallel trends assumption. In the potential-outcomes notation of causal inference it reads as follows.

$$E[Y_{i1}(0) - Y_{i0}(0) \mid D_i = 1] = E[Y_{i1}(0) - Y_{i0}(0) \mid D_i = 0]$$

where $Y_{it}(0)$ is the outcome unit $i$ would have had in period $t$ without the policy, periods 0 and 1 are before and after, and $D_i$ marks the treated group. The assumption concerns changes, not levels, so equal starting levels are neither required nor sufficient. No anticipation fails when hospitals or patients change behaviour once a reform is announced. The framework also assumes no interference (the stable unit treatment value assumption), meaning that one unit's outcome does not depend on another unit's treatment, which rules out spillovers.

The post-policy counterfactual is never observed, so analysts check instead whether the groups moved together over several pre-policy years. That check proves less than it appears to. Pre-trend tests often have low power, and proceeding only when a pre-test is passed can itself distort estimates and confidence intervals. Parallel trends is also sensitive to scale, since trends parallel in admission rates need not be parallel in log rates.

Design choices strengthen the comparison. An evaluation of state Medicaid expansions in the United States used neighbouring non-expanding states as comparators. The estimators of Callaway and Sant'Anna allow parallel trends to hold conditional on observed covariates, and adding conditions the policy did not target gives a triple-difference design. Rambachan and Roth bound how far post-policy violations could differ from pre-policy differences in trends, and report which conclusions survive.

Worked example with an illustrative hospital admissions policy

The figures below are illustrative. Region A introduces a care-coordination payment for general practices in 2024; region B does not. The outcome is emergency admissions per 1,000 residents aged 75 and over per year.

1. Two-by-two estimate. Taking 2023 as before and 2025 as after gives the following.

$$\hat{\tau} = (47.0 - 52.0) - (46.0 - 48.0) = -5.0 + 2.0 = -3.0$$

where $\hat{\tau}$ is the DiD estimate in admissions per 1,000 residents per year. Region A's counterfactual under parallel trends is 52.0 plus region B's change of minus 2.0, which is 50.0, and the observed 47.0 is 3.0 below it.

2. Why simpler comparisons mislead. A before-and-after comparison in region A would claim the whole fall of 5.0, while comparing the regions in 2025 alone would suggest harm, because region A starts higher.

3. From effect to economic consequence. Suppose, illustratively, region A has 200,000 residents aged 75 and over, an emergency admission costs £3,000 and the payment costs £1.2 million a year. The estimate implies 600 fewer admissions a year (200,000 ÷ 1,000 × 3.0), gross savings of £1.8 million and a net saving of £0.6 million; a cost-effectiveness analysis would also value the health change. With one treated and one comparison region, the uncertainty around the 3.0 is hard to quantify, for the reasons given in the section on standard errors below.

4. A violated-trends variant. The 2021 column changes the picture. Before any policy, region A fell by 6.0 and region B by 2.0.

$$\hat{\delta}_{pre} = (52.0 - 58.0) - (48.0 - 50.0) = -6.0 + 2.0 = -4.0$$

where $\hat{\delta}_{pre}$ is a placebo DiD over the two pre-policy years, which should be near zero if trends were parallel. If that differential decline had continued for two more years, it would explain a gap of 4.0, more than the whole estimated effect.

$$\hat{\tau}_{adj} = -3.0 - (-4.0) = 1.0$$

where $\hat{\tau}_{adj}$ is the estimate after removing a linearly extrapolated differential trend. An increase of 1.0 per 1,000 means about 200 more admissions a year (200,000 ÷ 1,000 × 1.0), costing £0.6 million (200 × £3,000). Added to the £1.2 million payment, the £0.6 million net saving becomes a net cost of about £1.8 million. The extrapolation is itself an assumption, which is why bounded sensitivity analysis is preferable to one trend-corrected number, and the 2023 to 2025 comparison alone could not have revealed the problem.

Event studies across several periods

With many periods, the single interaction is usually replaced by terms for each period relative to adoption. This event-study form shows how the effect builds or fades, and the pre-policy coefficients used to judge parallel trends.

$$Y_{it} = \alpha_i + \lambda_t + \sum_{k \neq -1} \beta_k \, \mathbf{1}[t - G_i = k] + \varepsilon_{it}$$

where $\alpha_i$ is a unit fixed effect, $\lambda_t$ is a period fixed effect, $G_i$ is the period in which unit $i$ adopts the policy, $\mathbf{1}[t - G_i = k]$ equals 1 when period $t$ is $k$ periods from adoption, $\beta_k$ is the effect at that distance relative to the omitted period just before adoption ($k = -1$), and $\varepsilon_{it}$ is the error term. Pre-policy coefficients near zero are consistent with parallel trends but do not establish it.

Staggered adoption and two-way fixed effects

Health policies are often adopted at different times, as with Medicaid expansion by individual states in different years. Each adoption date then creates its own before-and-after comparison. The usual response was a regression with unit and period fixed effects and one treatment indicator.

$$Y_{it} = \alpha_i + \lambda_t + \beta D_{it} + \varepsilon_{it}$$

where $D_{it}$ equals 1 once unit $i$ has adopted the policy by period $t$, $\beta$ is the two-way fixed effects coefficient, and the other terms are as above. With two groups and two periods $\beta$ equals the DiD estimate; with staggered timing it does not have that simple meaning.

Goodman-Bacon showed that $\beta$ is a weighted average of all possible two-group, two-period comparisons, weighted by group size and the variance of the treatment indicator. Groups treated mid-panel count most as treated groups, while units treated near the start or end of the panel can get more weight as controls. Some comparisons use already-treated units as controls; when effects change over time, changes in their effects are subtracted, typically biasing the estimate away from the sign of the true effect, so a causal reading needs parallel trends and effects constant over time. De Chaisemartin and D'Haultfoeuille showed that the coefficient is a weighted sum of group and period average effects with weights that may be negative, so it can be negative when every underlying effect is positive. In the event-study form, Sun and Abraham showed that each lead or lag coefficient can be contaminated by effects from other periods, and that apparent pre-trends can arise purely from differences in effects across adoption groups.

Alternative estimators avoid comparisons between already-treated units. Callaway and Sant'Anna estimate an effect for each adoption cohort and period.

$$ATT(g,t) = E[Y_t - Y_{g-1} \mid G = g] - E[Y_t - Y_{g-1} \mid G = g']$$

where $ATT(g,t)$ is the average effect in period $t$ for units first treated in period $g$, $Y_t - Y_{g-1}$ is the change from the last pre-adoption period to period $t$, $G$ is the adoption period, and $g'$ is a cohort not yet treated by period $t$ (or never treated). These effects are combined with weights the analyst states, for example by time since adoption, which shows any heterogeneous treatment effect rather than hiding it.

Standard errors when outcomes are correlated over time

DiD studies typically use many years of outcomes that are correlated from year to year within an area or hospital. Bertrand, Duflo and Mullainathan showed how badly conventional standard errors then fail. Assigning placebo laws at random to states in wage data, they found a spurious "effect" significant at the 5% level for up to 45% of the placebo interventions. Corrections allowing arbitrary correlation within each state over time worked for moderate numbers of states, and collapsing the data into one pre-period and one post-period, with the effective sample size taken into account, worked even with small numbers of states.

The usual practice is a cluster-robust standard error, built on the sandwich estimator, at the level at which the policy was assigned, such as the region or trust. This needs enough clusters, and Roth and colleagues note that a cluster wild bootstrap can perform poorly with few treated clusters, and describe alternatives for many control units and few treated ones. A one-region policy should report this limitation rather than hide it behind a conventional confidence interval.

From a DiD estimate to an economic evaluation

DiD often supplies the effect size an economic evaluation of a service or policy change needs, where trials are rare. The NICE real-world evidence framework (ECD9, 2022) lists difference-in-difference estimation among common quasi-experimental methods, frequently used in public health. How the estimate is carried forward can change the result substantially.

The Advancing Quality pay-for-performance scheme, introduced in 2008 in all NHS hospitals in north-west England, shows the chain. A DiD regression of 30-day in-hospital mortality for pneumonia, heart failure and acute myocardial infarction, against other English hospitals and against conditions outside the scheme, found an absolute reduction of 1.3 percentage points, equivalent to 890 fewer deaths over 18 months. A cost-effectiveness analysis of the scheme's first 18 months found about 5,200 QALY gains and £4.4 million of savings from shorter stays against total programme costs of about £13 million, and judged the scheme cost-effective.

Two later findings show where the translation is fragile. By the end of a 42-month follow-up the reduction for incentivised conditions was no longer significant, after mortality fell faster in comparison hospitals, while mortality for conditions outside the scheme fell more in participating hospitals, suggesting a possible positive spillover. For patients admitted with pneumonia, converting the mortality effect into life years with parametric survival analysis models rather than general population life expectancy raised the estimated mean life-year gain from 0.154 to 0.380 years. An evaluation built on DiD therefore needs explicit choices on effect duration, spillovers and how the estimate's standard error enters probabilistic analysis.

Where DiD sits among quasi-experimental designs

DiD is one of several quasi-experimental designs, and the structure of the data usually decides between them. Interrupted time series needs many time points rather than a comparison group, and assumes the prior level and trend would have continued. Synthetic control suits one or a few treated aggregate units, weighting untreated units to match the treated unit before the policy, as in the evaluation of California's Proposition 99 tobacco control programme. Propensity score matching can be combined with DiD to choose comparison units.

Common misreadings of DiD results

Two further errors recur in health policy evaluations, and both concern who the estimate applies to.

• Spillovers ignored. If the comparison group also benefits, for example through shared clinicians, the estimate is biased towards zero; if patients are displaced into the comparison group, it is biased away from zero.
• The adopters' effect read as the effect of national roll-out. Early adopters may have been the most motivated or best resourced, so later adopters may gain less.

Sources

• Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. Journal of the American Statistical Association. 2010;105(490):493-505.
• Angrist JD, Pischke JS. Mostly Harmless Econometrics: An Empiricist's Companion. Princeton: Princeton University Press; 2009. Chapter 5.
• Bertrand M, Duflo E, Mullainathan S. How much should we trust differences-in-differences estimates? Quarterly Journal of Economics. 2004;119(1):249-275.
• Callaway B, Sant'Anna PHC. Difference-in-differences with multiple time periods. Journal of Econometrics. 2021;225(2):200-230.
• Card D, Krueger AB. Minimum wages and employment: a case study of the fast-food industry in New Jersey and Pennsylvania. American Economic Review. 1994;84(4):772-793.
• de Chaisemartin C, D'Haultfoeuille X. Two-way fixed effects estimators with heterogeneous treatment effects. American Economic Review. 2020;110(9):2964-2996.
• Goodman-Bacon A. Difference-in-differences with variation in treatment timing. Journal of Econometrics. 2021;225(2):254-277.
• Kristensen SR, Meacock R, Turner AJ, Boaden R, McDonald R, Roland M, Sutton M. Long-term effect of hospital pay for performance on mortality in England. New England Journal of Medicine. 2014;371(6):540-548.
• McDonald R, Boaden R, Roland M, Kristensen SR, Meacock R, Lau YS, Mason T, Turner AJ, Sutton M. A qualitative and quantitative evaluation of the Advancing Quality pay-for-performance programme in the NHS North West. Health Services and Delivery Research. 2015;3(23). doi:10.3310/hsdr03230.
• Meacock R, Kristensen SR, Sutton M. The cost-effectiveness of using financial incentives to improve provider quality: a framework and application. Health Economics. 2014;23(1):1-13.
• Meacock R, Sutton M, Kristensen SR, Harrison M. Using survival analysis to improve estimates of life year gains in policy evaluations. Medical Decision Making. 2017;37(4):415-426.
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). London: NICE; 2022. Section: Methods for real-world studies of comparative effects.
• Rambachan A, Roth J. A more credible approach to parallel trends. Review of Economic Studies. 2023;90(5):2555-2591.
• Roth J. Pretest with caution: event-study estimates after testing for parallel trends. American Economic Review: Insights. 2022;4(3):305-322.
• Roth J, Sant'Anna PHC, Bilinski A, Poe J. What's trending in difference-in-differences? A synthesis of the recent econometrics literature. Journal of Econometrics. 2023;235(2):2218-2244.
• Sommers BD, Baicker K, Epstein AM. Mortality and access to care among adults after state Medicaid expansions. New England Journal of Medicine. 2012;367(11):1025-1034.
• Sun L, Abraham S. Estimating dynamic treatment effects in event studies with heterogeneous treatment effects. Journal of Econometrics. 2021;225(2):175-199.
• Sutton M, Nikolova S, Boaden R, Lester H, McDonald R, Roland M. Reduced mortality with hospital pay for performance in England. New England Journal of Medicine. 2012;367(19):1821-1828.
• Wing C, Simon K, Bello-Gomez RA. Designing difference in difference studies: best practices for public health policy research. Annual Review of Public Health. 2018;39:453-469.