Quasi-Experimental Design

Quasi-Experimental Design in Health Policy and Programme Evaluation

A quasi-experimental design is the way health researchers estimate what a policy, programme or treatment did when nobody could randomise who received it. In place of chance allocation, the design borrows its comparison from somewhere else, such as an area the policy did not reach, the trend before the policy began, or people who fell just on the other side of an eligibility rule. Health economists depend on these designs for the many decisions that a randomised controlled trial cannot inform, such as smoking bans, alcohol taxes and payment reforms. The credibility of the result rests on how exposure came about, so the same routine data can support a strong design or a weak one depending on what the analyst knows about that process.

Where the comparison comes from when chance does not decide

Every causal question compares two potential outcomes, with and without the intervention, of which only one is ever observed. In the potential outcomes framework used throughout causal inference, the average effect is the difference between the two averages.

$$\tau = E[Y(1)] - E[Y(0)]$$

where $\tau$ is the average causal effect, $Y(1)$ is the outcome a unit would have with the intervention, $Y(0)$ is the outcome it would have without it, and $E$ denotes the average over the population of interest.

Craig and colleagues, reviewing natural experiment methods, explain that random assignment makes the groups exchangeable, while in an observational study knowledge of the assignment mechanism can make them conditionally exchangeable. The UK Medical Research Council (MRC) guidance on natural experiments adds that randomised trials have a general method for minimising bias from selective exposure, while non-randomised studies have only partial solutions that work in restricted circumstances. Understanding the process that produced the variation in exposure is therefore critical to the design.

The labels overlap, and the MRC guidance notes that the term natural experiment lacks an exact definition. What separates these studies from an ordinary observational study is a clearly identified intervention and an assignment process that can be described, whereas many observational comparisons have no defined intervention and may suffer reverse causation as well as confounding. In a non-randomised controlled trial the investigators choose who is treated, a design that Handley and colleagues also count as quasi-experimental.

The main designs and the assumption each one needs

Each design finds its counterfactual in a different place, and each is only as good as the assumption that makes that counterfactual credible. The table summarises the designs most used in health policy evaluation.

The pre-post comparison with a control group is usually analysed by difference-in-differences. An interrupted time series can work with one or a few groups because each acts as its own control, though it needs many measurements. A synthetic control suits a single region or provider, and instrumental variable analysis exploits factors such as distance from a specialist centre.

Threats to internal validity in routine health data

Handley and colleagues define internal validity as the extent to which a study can establish causality by minimising bias, and external validity as how far its conclusion generalises to other populations or settings. Common threats include history bias from other events at the same time, selection bias from systematic differences between groups, maturation, and differential drop-out.

These threats are sharper in health economics because admissions, prescribing and spending move with secular trends, coding changes and other policies. Handley and colleagues note that pre-post designs with a control group carry a substantial risk of temporal biases, including regression to the mean. The MRC guidance adds that natural experimental studies are more susceptible to bias and confounding than planned experiments.

Regression discontinuity at clinical and eligibility thresholds

Many health care decisions follow a threshold, such as a laboratory value, an age, an income limit or a date, and regression discontinuity compares people just either side of it. Venkataramani, Bor and Jena, reviewing the design in the BMJ, give examples including decisions to start antihypertensive treatment, transfuse blood products or screen for colorectal cancer, and Jones and Rice note that eligibility for free prescription drugs may depend on age alone.

In a sharp design everyone on one side of the cut-off is treated and nobody on the other side is. In a fuzzy design, which the BMJ review calls more relevant to most clinical and policy questions, the probability of treatment rises at the cut-off but some people on each side cross over. The comparison at the threshold then captures the effect of assignment by the rule, and instrumental variable methods recover the effect among compliers.

$$\tau_{FRD} = \frac{\lim_{x \downarrow c} E[Y \mid X = x] - \lim_{x \uparrow c} E[Y \mid X = x]}{\lim_{x \downarrow c} E[D \mid X = x] - \lim_{x \uparrow c} E[D \mid X = x]}$$

where $\tau_{FRD}$ is the fuzzy regression discontinuity effect among compliers at the cut-off, $X$ is the running variable that sets eligibility, $c$ is the cut-off, $Y$ is the outcome, $D$ equals 1 for people who actually receive the intervention, and the limits are the fitted values approached from just above and just below the cut-off.

The BMJ review reports that Bor and colleagues, using the South African guideline to start antiretroviral treatment at CD4 counts below 200 cells per cubic millimetre, found a 35% lower hazard of death just below the threshold than just above it, with patients either side statistically identical on baseline characteristics. Craig and colleagues describe usual practice as local linear regressions within a narrow band either side of the cut-off, which trades bias against precision, with a histogram of the running variable to detect bunching that would suggest manipulation.

Which effect the design estimates

A quasi-experimental estimate often answers a narrower question than its headline suggests. Craig and colleagues distinguish the intention-to-treat effect of being assigned to an intervention, which equals the effect of the intervention only under perfect compliance, from the complier average causal effect estimated by fuzzy regression discontinuity and instrumental variables. Under certain assumptions the complier effect equals the intention-to-treat effect divided by the proportion of compliers.

Regression discontinuity also estimates effects at one value of the assignment variable, known as local average treatment effects, so extrapolation needs care. A policy evaluation that wants the effect across a whole population, the target of an intention-to-treat analysis, may find a local effect less useful. The average treatment effect for everyone eligible can differ again when there is a heterogeneous treatment effect.

Worked example with an illustrative age threshold for a falls service

All numbers in this example are illustrative. A health board offers a home falls-prevention assessment to people aged 75 and over, but general practitioners also refer some younger patients and not everyone eligible accepts. Local linear regressions fitted within two years of age either side of 75 give the values at the cut-off.

1. Outcome jump. Fitted emergency admissions for falls are 148 per 1,000 people a year just below age 75 and 133 per 1,000 just above, a difference of minus 15 per 1,000 people.

2. Uptake jump. The share receiving the assessment is 10% just below and 60% just above, so the threshold raises uptake by 0.50.

3. Effect among compliers. The complier effect is the outcome jump divided by the uptake jump.

$$\tau_{FRD} = \frac{-15}{0.50} = -30$$

where minus 15 is the outcome jump in admissions per 1,000 people, 0.50 is the uptake jump and minus 30 is the change in admissions per 1,000 compliers, or 0.030 fewer admissions per complier.

4. Cost per complier. At an illustrative GBP 3,000 per emergency admission and GBP 120 per assessment, the avoided admission cost per complier is 0.030 × 3,000 = GBP 90, leaving a net cost of GBP 30 per complier.

5. Cost per person offered. At the cut-off, offering the service adds 0.50 × 120 = GBP 60 of assessment cost per person and saves 0.015 × 3,000 = GBP 45 in admissions, a net GBP 15, which equals 0.50 × 30 = GBP 15.

6. Interpretation. The estimate applies to people near age 75 whose uptake depended on the rule. Older people at higher risk may gain more, so applying the GBP 30 net cost to everyone aged 75 and over is an extrapolation that a budget impact analysis would have to state. The design also fails if another service starts at age 75.

Using quasi-experimental estimates in economic evaluation and HTA

The NICE real-world evidence framework (ECD9), published in 2022, states that quasi-experimental studies and natural experiments exploit external variation in exposure across people or over time that is otherwise unrelated to the outcome. It lists instrumental variable analysis, regression discontinuity, interrupted time series and difference-in-difference estimation as common methods, which are used often in public health settings. ECD9 adds that these methods usually depend strongly on assumptions that are difficult to test, so a clear case for validity based on substantive knowledge and empirical justification is required.

For health technology assessment, this places quasi-experimental studies within real-world evidence, beside the cohort designs that ECD9 identifies as the most common non-randomised studies informing NICE guidance. When a quasi-experimental estimate enters a cost-effectiveness analysis, the model should keep the population, comparator, follow-up and type of effect the design identified. The MRC guidance sees the strongest case for a natural experimental study where a health impact is expected but its size is uncertain, a trial would be impractical or unethical, and the intervention could be replicated or scaled.

Strengthening and reporting a quasi-experimental study

No single test proves a quasi-experimental estimate, so credible studies combine design features that make alternative explanations less likely. The MRC guidance lists multiple measures before and after the intervention, multiple exposed and unexposed groups, accurate measurement of potential confounders and combinations of methods addressing different sources of bias.

Craig and colleagues add falsification tests, such as outcomes not expected to respond and false implementation dates, an analysis protocol written before any analysis, and reporting guidelines such as STROBE or TREND. They also conclude that the choice of method should follow the features of the intervention, such as the allocation process and the comparators available, not a general ranking of which method is strongest.

Common misreadings of quasi-experimental results

Most errors with these designs come from treating the label as a guarantee of causal validity. The MRC guidance and the Craig review describe each of the following.

1. Treating adjustment as design. Regression adjustment and propensity score methods assume that every factor driving both exposure and outcome has been measured accurately, so they do not deal with unobserved confounding.
2. Reading a large effect as proof. A large effect that follows implementation can still contain a large element of bias from selective exposure.
3. Generalising a local estimate. A regression discontinuity or instrumental variable estimate describes compliers near the cut-off or moved by the instrument, not every eligible patient.
4. Assuming a natural experiment always exists. The MRC guidance calls it misleading to assume that a natural experimental study is waiting whenever a planned experiment is impossible.

Sources

• Craig P, Cooper C, Gunnell D, Haw S, Lawson K, Macintyre S, et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. Journal of Epidemiology and Community Health. 2012;66(12):1182-1186. https://doi.org/10.1136/jech-2011-200375
• Craig P, Katikireddi SV, Leyland A, Popham F. Natural experiments: an overview of methods, approaches, and contributions to public health intervention research. Annual Review of Public Health. 2017;38:39-56. https://doi.org/10.1146/annurev-publhealth-031816-044327
• Handley MA, Lyles CR, McCulloch C, Cattamanchi A. Selecting and improving quasi-experimental designs in effectiveness and implementation research. Annual Review of Public Health. 2018;39:5-25. https://doi.org/10.1146/annurev-publhealth-040617-014128
• Jones AM, Rice N. Econometric evaluation of health policies. HEDG Working Paper 09/09. University of York; May 2009. https://www.york.ac.uk/media/economics/documents/herc/wp/09_09.pdf
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). Published 23 June 2022. https://www.nice.org.uk/corporate/ecd9/chapter/methods-for-real-world-studies-of-comparative-effects
• Venkataramani AS, Bor J, Jena AB. Regression discontinuity designs in healthcare research. BMJ. 2016;352:i1216. https://doi.org/10.1136/bmj.i1216