Before-and-After Study

Before-and-After Study in Service and Policy Evaluation

A before-and-after study reads the change in an outcome between a period before an intervention and a period after it as the effect of that intervention. Without a control group, the design assumes that nothing else would have moved the outcome, an assumption that secular trends, regression to the mean and other changes made at the same time routinely break. Health economists meet the design constantly, because service reorganisations, payment reforms and local programmes are often evaluated by comparing activity or spending in the year before launch with the year after. Those comparisons become claimed savings in business cases and in budget impact analysis, so the choice of comparison decides whether a saving is real or an artefact.

The counterfactual behind a before-and-after comparison

Every effect estimate compares what happened with what would have happened without the intervention. A before-and-after study uses the pre-intervention measurement as that estimate, so in causal inference terms the past stands in for the counterfactual.

$$\hat{\Delta} = \bar{Y}_{post} - \bar{Y}_{pre} = \Delta + \left(\bar{Y}^{0}_{post} - \bar{Y}_{pre}\right)$$

where $\bar{Y}_{pre}$ is the mean outcome before the intervention, $\bar{Y}_{post}$ is the mean outcome after it, $\bar{Y}^{0}_{post}$ is the mean outcome the same group would have had in the later period without the intervention, and $\Delta = \bar{Y}_{post} - \bar{Y}^{0}_{post}$ is the true effect. The bracketed term is the change that would have happened anyway. The uncontrolled design can only recover the true effect when that term is zero, and the data from one group give no way to check it.

Classification varies. Eccles and colleagues (2003) call before-and-after and time series designs non-experimental, while the Cochrane Handbook lists controlled before-and-after studies among non-randomised studies. Because it estimates an intervention's effect without random assignment, it falls within the broad definition of a quasi-experimental design, as interrupted time series and difference-in-differences do.

Uncontrolled and controlled versions in the EPOC definitions

The Cochrane Effective Practice and Organisation of Care group (EPOC), which reviews health system interventions, gives standard definitions. It defines a controlled before-after study as one in which observations are made before and after an intervention is implemented, both in a group that receives it and in a control group that does not. Allocation to the groups is not made by the investigators, and EPOC judges these studies at high risk of bias because unidentified differences between the groups may affect changes in the outcome.

EPOC sets minimum standards for the controlled version. It recommends including only studies with at least two intervention sites and two control sites, because with one site of each the intervention is completely confounded by study site. It also excludes controlled before-after studies whose data collection was not contemporaneous in study and control sites, or which did not use identical methods of measurement. Its risk of bias criteria then ask whether baseline outcome measurements and characteristics were similar across groups.

The uncontrolled version has no comparison group at all. EPOC strongly discourages including uncontrolled before-after studies in its reviews, stating that it is difficult, if not impossible, to attribute causation from them.

What moves the outcome apart from the intervention

Three mechanisms produce spurious before-and-after effects, and each feeds the bracketed term above. One group measured twice cannot separate them.

Secular trends come first. Soumerai, Starr and Majumdar (2015) describe history bias, where events before or during the intervention have a greater effect than the intervention itself. In the control hospitals of a trial of opinion leaders for acute myocardial infarction, beta blocker use rose from 60% to 78% without the intervention, during a period of new guidelines and a national programme. The same authors show that hospital mortality in the United States was already falling for years before the 100,000 Lives Campaign began in January 2005 and continued at roughly the same rate during it, although its organisers had claimed 122,300 lives saved in 18 months.

Confounding by co-interventions comes second. A service change rarely arrives alone, since staffing, coding practice, payment rules and national policy shift around it. The MRC guidance on natural experiments notes that substantial, rapid changes in exposure across a whole population reduce the risk of confounding by secular trends, a protection gradual local changes lack.

Regression to the mean comes third. Morton and Torgerson (2003) explain that it occurs whenever a group is selected with extreme values on one measurement and then measured again, so that any intervention aimed at a group very different from the average will appear successful. Barnett, van der Pols and Dobson (2005) add that the effect becomes more noticeable with increasing measurement error and when follow-up measurements are only examined on a sub-sample selected using a baseline value. Health services select this way routinely, enrolling high users, poorly performing practices or wards after a spike in infections.

Emergency admissions and the evidence on regression to the mean

A well-documented health economics example concerns case management for older people with frequent emergency admissions. Roland, Dusheiko, Gravelle and Parker (2005) used Hospital Episode Statistics for England to follow 227,206 patients aged 65 or over who had two or more emergency admissions in 1997-8, with no intervention under study.

The cohort's emergency admission rate was 2.568 per person in the index year and fell to 0.641 in the following year and to 0.194 by 2002-3, while the rate for all people of the same age moved only from 0.198 to 0.177. Emergency bed days per person fell from 29.6 to 9.0 in the following year. The authors did not exclude deaths and attribute the fall probably to mortality in the cohort as well as regression toward the mean among survivors.

The authors concluded that simply monitoring admission rates cannot assess interventions to reduce admission among frail older people, because rates fall without any intervention, and that a matched control group is necessary. A programme that enrols patients after two admissions will appear to save most of the bed days used in the qualifying year.

Worked example of an uncontrolled saving claim

The following figures are illustrative only. A commissioner enrols 1,000 patients in a case management programme after each has had at least two emergency admissions in a year, with a mean of 2.5 admissions per patient in that year. In the year after enrolment the mean is 0.8. Each admission costs GBP 2,000 and the programme costs GBP 600 per patient.

The uncontrolled estimate is a fall of 2.5 minus 0.8, or 1.7 admissions per patient. Across 1,000 patients that is 1,700 admissions, a gross saving of 1,700 times GBP 2,000, or GBP 3,400,000. After the programme cost of GBP 600,000, the claimed net saving is GBP 2,800,000.

A comparison group selected on the same rule in a neighbouring area, without the programme, also starts at 2.5 admissions and falls to 0.9. The controlled estimate is the difference in the two changes.

$$\hat{\Delta}_{DiD} = (0.8 - 2.5) - (0.9 - 2.5) = -1.7 - (-1.6) = -0.1$$

where the first bracket is the change in the programme group and the second is the change in the comparison group, both in admissions per patient per year. The programme avoids 0.1 admissions per patient, or 100 admissions across 1,000 patients, worth 100 times GBP 2,000, or GBP 200,000. Set against the programme cost of GBP 600,000, the result is a net cost of GBP 400,000.

The uncontrolled comparison overstated the effect 17-fold, 1.7 against 0.1 admissions per patient, and would have entered a budget impact analysis as a saving, not a cost.

Difference-in-differences as the controlled analysis

Adding a control group turns one before-and-after change into two, and the analysis subtracts one from the other. That is the logic of difference-in-differences, usually estimated by regression with group, period and interaction terms.

$$\hat{\Delta}_{DiD} = \left(\bar{Y}_{T,post} - \bar{Y}_{T,pre}\right) - \left(\bar{Y}_{C,post} - \bar{Y}_{C,pre}\right)$$

where $\bar{Y}_{T,pre}$ and $\bar{Y}_{T,post}$ are the mean outcomes in the intervention group before and after, and $\bar{Y}_{C,pre}$ and $\bar{Y}_{C,post}$ are the corresponding means in the control group.

The MRC guidance explains that the differencing controls for unobserved individual differences and common trends, assuming these are fixed and that the groups would change in the same way without the intervention, and that it is vulnerable to changes in group composition. With one pre-period that assumption cannot be checked, and Soumerai and colleagues rank the before-and-after design with a comparison group as intermediate because the comparability of baseline trends is often unknown. A comparison group chosen by the same entry rule shares the regression to the mean, as in the worked example.

Before-and-after study versus interrupted time series

An interrupted time series replaces the single pre and post measurements with many. EPOC describes a design that uses observations at multiple time points before and after an intervention and asks whether the effect is significantly greater than the underlying trend, and it excludes time series without a clearly defined intervention point and at least three data points before and three after.

The extra points make the secular trend visible. Soumerai and colleagues show that when an outcome is already falling, single points before and after an intervention suggest a large effect where the trend shows none. EPOC's risk of bias guidance adds that a time series analysed with a simple t-test of pre versus post periods, ignoring secular change, should not be included unless reanalysis is possible, since that analysis turns it back into a before-and-after study. In the hierarchy that Soumerai and colleagues propose, an interrupted time series with a control series counts as a strong design, a single interrupted time series and a controlled before-and-after study as intermediate, and the uncontrolled before-and-after study as weak.

Use in economic evaluation and health technology assessment

Before-and-after evidence enters economic work in two ways. The first is as the effect estimate for a service or policy change, where a randomised controlled trial is often impractical. Any bias in the estimate passes directly into the incremental costs and outcomes of the economic evaluation, and probabilistic analysis around a biased point estimate does not reveal it.

The second is descriptive. The NICE manual for technology appraisal and highly specialised technologies (PMG36) states that diagnostic before-and-after studies provide useful information on any change in management after an index test is introduced. Here the design describes changes in care pathways for a diagnostic model, not a health effect.

In systematic reviews, labels can mislead. The Cochrane Handbook chapter on non-randomised studies cites a review finding that Cochrane reviews labelled controlled before-and-after and interrupted time series studies inconsistently, and proposes setting review eligibility on study design features rather than on labels.

Common misreadings of before-and-after results

A large fall after an intervention is weak evidence when patients, practices or areas were selected because their values were high. In the Roland cohort, high users' admissions fell by about three-quarters in a year without any intervention under study, through deaths as well as regression to the mean.

A statistically significant pre-post difference does not address bias. Significance concerns chance, while the bracketed term in the first equation is systematic and does not shrink with more patients.

Sources

• Barnett AG, van der Pols JC, Dobson AJ. Regression to the mean: what it is and how to deal with it. International Journal of Epidemiology. 2005;34(1):215-220. https://doi.org/10.1093/ije/dyh299
• Cochrane Effective Practice and Organisation of Care (EPOC). What study designs can be considered for inclusion in an EPOC review and what should they be called? EPOC Resources for review authors; 2017. https://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/Resources-for-authors2017/what_study_designs_should_be_included_in_an_epoc_review.pdf
• Cochrane Effective Practice and Organisation of Care (EPOC). Suggested risk of bias criteria for EPOC reviews. EPOC Resources for review authors; 2017. https://epoc.cochrane.org/sites/epoc.cochrane.org/files/uploads/Resources-for-authors2017/suggested_risk_of_bias_criteria_for_epoc_reviews.pdf
• Craig P, Cooper C, Gunnell D, et al. Using natural experiments to evaluate population health interventions: new Medical Research Council guidance. Journal of Epidemiology and Community Health. 2012;66(12):1182-1186. https://doi.org/10.1136/jech-2011-200375
• Eccles M, Grimshaw J, Campbell M, Ramsay C. Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality and Safety in Health Care. 2003;12(1):47-52. https://doi.org/10.1136/qhc.12.1.47
• Morton V, Torgerson DJ. Effect of regression to the mean on decision making in health care. BMJ. 2003;326(7398):1083-1084. https://doi.org/10.1136/bmj.326.7398.1083
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Reeves BC, Deeks JJ, Higgins JPT, Shea B, Tugwell P, Wells GA. Chapter 24: Including non-randomized studies on intervention effects. In: Higgins JPT, Thomas J, Chandler J, et al, editors. Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. https://training.cochrane.org/handbook/current/chapter-24
• Roland M, Dusheiko M, Gravelle H, Parker S. Follow up of people aged 65 and over with a history of emergency admissions: analysis of routine admission data. BMJ. 2005;330(7486):289-292. https://doi.org/10.1136/bmj.330.7486.289
• Soumerai SB, Starr D, Majumdar SR. How do you know which health care effectiveness research you can trust? A guide to study design for the perplexed. Preventing Chronic Disease. 2015;12:E101. https://doi.org/10.5888/pcd12.150187