Synthetic Control

Synthetic Control in Health Policy Evaluation

The synthetic control method estimates what a policy did to one region, country or group of hospitals by asking what the same place would have looked like without the policy, and answering with a weighted average of places that never adopted it. The weights are chosen so that the average tracks the treated unit closely for years before the change, and the gap that opens afterwards is read as the effect. Health economists use the method when a tobacco tax, payment scheme or service reform lands on a single aggregate unit and no single comparison area resembles it. The method belongs with difference-in-differences and interrupted time series among the quasi-experimental designs, and suits settings where parallel trends look doubtful.

Why a blend of untreated units replaces a single comparison area

Comparative case studies judge a policy by setting the treated area beside similar areas. Abadie, in his 2021 review, notes that the choice of comparison units is informal, and that when the units are a few regions or countries no single one may resemble the treated unit. The synthetic control method rests on the idea that a combination of unaffected units often gives a better comparison than any one of them, and it turns the choice into a data-driven procedure.

Abadie and Gardeazabal built a synthetic Basque Country from other Spanish regions and estimated that, after terrorism broke out in the late 1960s, per capita GDP fell about 10 percentage points relative to the synthetic region. Abadie, Diamond and Hainmueller then studied Proposition 99, which raised California's cigarette excise tax by 25 cents per pack in 1988 and earmarked the revenue for health and anti-smoking education. They removed from the donor pool, the set of candidate comparison states, any state that adopted a large tobacco control programme or raised cigarette taxes by 50 cents or more between 1989 and 2000, leaving 38 states. Synthetic California combined Colorado, Connecticut, Montana, Nevada and Utah, and by 2000 annual per capita cigarette sales in California were estimated to be about 26 packs lower than they would have been without Proposition 99.

The estimator and how the weights are chosen

The notation follows Abadie's review. With unit 1 treated, units 2 to $J+1$ in the donor pool and the policy starting after period $T_0$, the estimated effect in each later period is the observed outcome minus the weighted donor outcome.

$$\hat{\tau}_{1t} = Y_{1t} - \sum_{j=2}^{J+1} w_j^{*} Y_{jt}, \quad t > T_0$$

where $\hat{\tau}_{1t}$ is the estimated effect on the treated unit in period $t$, $Y_{1t}$ is its observed outcome, $Y_{jt}$ is the outcome of donor unit $j$, and $w_j^{*}$ is that donor's weight from the minimisation below. The effect can vary over time, which suits health policies whose effects build up or fade.

The weights make the synthetic unit resemble the treated unit before the policy on a set of predictors, usually including pre-policy values of the outcome.

$$W^{*} = \arg\min_{W} (X_1 - X_0 W)^{\top} V (X_1 - X_0 W) \quad \text{subject to} \quad w_j \ge 0, \; \sum_{j=2}^{J+1} w_j = 1$$

where $W$ is the vector of donor weights, $X_1$ is the vector of $k$ pre-policy predictors for the treated unit, $X_0$ is the $k \times J$ matrix of the same predictors for the donors, and $V$ is a diagonal matrix giving each predictor its importance. In the original papers $V$ is chosen to minimise the pre-policy mean squared prediction error, and Abadie also describes choosing it by validation within the pre-policy years.

Non-negative weights that sum to one make the synthetic unit an average of real donors, so there is no extrapolation beyond the data. A regression counterfactual is also a weighted sum, but its weights can be negative, and in Abadie's German reunification example four countries received negative regression weights. When the treated unit lies outside the convex hull of the donors, as is typical, the solution is sparse, with no more nonzero weights than predictors, so the counterfactual can be inspected donor by donor. Post-policy outcomes are not needed to compute the weights, so they can be published before results arrive, much like a pre-analysis plan in a randomised controlled trial.

Assumptions that decide whether the gap is an effect

The method can be motivated by a linear factor model in which unobserved characteristics affect outcomes with weights that change over time. A difference-in-differences or fixed effects model is the special case in which those effects are constant, which is why Kreif and colleagues describe the synthetic control method as relaxing the parallel trends assumption. The price is a different set of conditions.

• Long and close pre-policy fit. The bias bound in the 2010 paper depends on the ratio of the scale of transitory shocks to the number of pre-policy periods, and helps only when the fit is good. With few pre-policy periods, a large donor pool can match noise, a form of overfitting.
• Similar donors. Abadie calls the inclusion of unsuitable donors a recipe for bias, since averaging very different units can hide large discrepancies.
• No similar policy or large shock among donors. Units that adopted a comparable intervention should be removed.
• No anticipation. If behaviour changes before the formal start, Abadie advises backdating the intervention in the data.
• No spillovers. If donors are affected in the same direction as the treated unit, the estimate is attenuated and becomes a lower bound on the size of the effect.

Bouttell and colleagues, reviewing health uses, add that there is no consensus on what counts as a good fit or how to judge similarity.

Placebo tests in place of standard errors

With one treated unit there is no sampling distribution in the usual sense, so Abadie, Diamond and Hainmueller proposed permutation inference. The method is rerun with each donor treated as if it had received the policy, and the treated unit's gap is judged against the placebo gaps. Because some donors fit poorly before the policy, the comparison uses the ratio of post-policy to pre-policy fit.

$$r_j = \frac{R_j(T_0 + 1, T)}{R_j(1, T_0)}, \quad R_j(t_1, t_2) = \left( \frac{1}{t_2 - t_1 + 1} \sum_{t=t_1}^{t_2} (Y_{jt} - \hat{Y}^{N}_{jt})^2 \right)^{1/2}$$

where $R_j(t_1, t_2)$ is the root mean square error of unit $j$'s synthetic control over periods $t_1$ to $t_2$, $\hat{Y}^{N}_{jt}$ is the outcome its synthetic control predicts, $T$ is the last period and $r_j$ is the ratio. The p-value is the share of all $J+1$ units whose ratio is at least as large as the treated unit's, a form of permutation test. For Proposition 99, California's post-policy mean squared prediction error was about 130 times its pre-policy value and the p-value was 1/39 = 0.026. An in-time placebo, which backdates the policy, and a leave-one-out analysis, which drops each contributing donor in turn, are the usual further checks.

Worked example with an illustrative regional alcohol policy

All figures in this example are illustrative. Region A, with 2,000,000 residents, introduces an alcohol policy at the start of year 5. The outcome is alcohol-related hospital admissions per 100,000 residents in A and four donor regions that made no such change.

1. Choose the weights. Fitting the four pre-policy years gives weights close to 0.5 for B, 0.3 for C and 0.2 for D, used rounded. Region E, far above A, gets almost no weight and is dropped.

2. Build the synthetic region. In year 1, 0.5 × 560 + 0.3 × 480 + 0.2 × 470 = 518.

3. Check the pre-policy fit. The gaps in years 1 to 4 are 1.0, 1.5, minus 0.6 and 0.6. Their squares sum to 3.97, and the square root of 3.97 / 4 gives a fit error of about 1.0 admission per 100,000. A plain donor average would give 540 in year 1, a gap of 21, and a fit error of 17.7.

4. Estimate the effect. In year 5 the gap is 525 minus 566.5, or minus 41.5 admissions per 100,000. In year 6 it is 515 minus 576.6, or minus 61.6.

5. Test against placebos. The post-policy fit error is the square root of (1,722.25 + 3,794.56) / 2, about 52.5, so the ratio for Region A is about 52.7. Suppose the four placebo runs give ratios of 3.1, 2.4, 4.8 and 1.9. Region A ranks first of five, so the p-value is 1/5 = 0.2, the smallest value four donors permit, however large the gap. Abadie, Diamond and Hainmueller note that, after discarding poorly fitting placebos, 19 control states gave a corresponding probability of 5 per cent.

6. Translate into resource use. In a region of 2,000,000, each 1 per 100,000 is 20 admissions, so the gaps imply 41.5 × 20 = 830 and 61.6 × 20 = 1,232 admissions averted, 2,062 in total. At an illustrative GBP 2,500 per admission, the hospital cost offset is 2,062 × 2,500 = GBP 5,155,000 over two years.

7. Interpretation. The estimate is the effect on Region A in these two years, conditional on the weights, the donor pool and no other changes in A. The cost offset can enter a budget impact analysis or the cost side of a cost-effectiveness analysis. The placebo rank is not a standard error, so a probabilistic sensitivity analysis needs another way to express uncertainty, such as confidence intervals from inverting the placebo test, which Abadie mentions.

Evidence from UK health policy evaluation

Kreif and colleagues re-evaluated Advancing Quality, the hospital pay-for-performance scheme introduced in North West England in October 2008, and compared the synthetic control method with difference-in-differences. Aggregating the 24 treated hospitals to regions would have left nine control regions and a minimum two-sided p-value of 0.22, so they weighted control hospitals directly to build a synthetic North West and drew placebo regions by resampling hospitals. On a balanced panel of 24 treated and 122 control hospitals, difference-in-differences estimated a fall in risk-adjusted 30-day mortality of 0.88 percentage points for the incentivised conditions (p < 0.01), while the synthetic control estimate was 0.45 percentage points and not significant (p = 0.21). For conditions outside the scheme, the synthetic control method found a rise of 0.90 percentage points (p = 0.02). Both analyses used the same patient-level risk adjustment, so the contrast shows how the design alone can change the evidence behind a cost-effectiveness judgement.

Bouttell and colleagues found 38 health-related studies using the method, most evaluating interventions at US state or national level. Later UK work includes a generalised synthetic control analysis of devolution in Greater Manchester.

Where synthetic control sits among neighbouring designs

Difference-in-differences relies on parallel trends, whereas synthetic control chooses weights to remove pre-policy differences and allows unobserved factors to have changing effects. Interrupted time series needs no comparison group, but Abadie notes that other shocks to the outcome complicate its medium and long-term estimates. Propensity score matching balances observed characteristics of many treated individuals, whereas synthetic control is built for few treated aggregate units. Abadie also describes bias-corrected and augmented versions that add a regression adjustment when donors cannot reproduce the treated unit's predictors.

In drug development the phrase has a second meaning. Thorlund and colleagues define synthetic controls in clinical research as cohorts of patients from external data, including real-world data from health records, claims or registries, adjusted by statistical methods and used for example as the comparator for a single-arm trial. That is an external control arm built from individual patients, resting on patient-level adjustment for confounding, a different design from the aggregate method here.

Common misreadings of synthetic control results

Most errors come from treating a close pre-policy fit as proof. Others come from reading the placebo test like a conventional test.

1. Taking a short good fit as evidence of no bias. With few pre-policy periods a close match can be spurious.
2. Filling the donor pool with any available unit. Dissimilar donors add bias and raise the risk of overfitting.
3. Ignoring the floor on the p-value. With $J$ donors the smallest achievable p-value is one divided by $J+1$.
4. Reading the estimate as an average effect for all regions. The method estimates the effect on the treated unit, not a population average treatment effect.

Sources

• Abadie A. Using synthetic controls: feasibility, data requirements, and methodological aspects. Journal of Economic Literature. 2021;59(2):391-425. https://doi.org/10.1257/jel.20191450
• Abadie A, Diamond A, Hainmueller J. Synthetic control methods for comparative case studies: estimating the effect of California's tobacco control program. NBER Working Paper 12831; 2007. Published in Journal of the American Statistical Association. 2010;105(490):493-505. https://www.nber.org/papers/w12831
• Abadie A, Gardeazabal J. The economic costs of conflict: a case study of the Basque Country. American Economic Review. 2003;93(1):113-132. https://doi.org/10.1257/000282803321455188
• Bouttell J, Craig P, Lewsey J, Robinson M, Popham F. Synthetic control methodology as a tool for evaluating population-level health interventions. Journal of Epidemiology and Community Health. 2018;72(8):673-678. https://doi.org/10.1136/jech-2017-210106
• Britteon P, Fatimah A, Lau YS, et al. The effect of devolution on health: a generalised synthetic control analysis of Greater Manchester, England. Lancet Public Health. 2022;7(10):e844-e852. https://doi.org/10.1016/S2468-2667(22)00198-0
• Kreif N, Grieve R, Hangartner D, et al. Examination of the synthetic control method for evaluating health policies with multiple treated units. Health Economics. 2016;25(12):1514-1528. https://doi.org/10.1002/hec.3258
• Thorlund K, Dron L, Park JJH, Mills EJ. Synthetic and external controls in clinical trials: a primer for researchers. Clinical Epidemiology. 2020;12:457-467. https://doi.org/10.2147/CLEP.S242097