Interrupted Time Series

Interrupted Time Series in Health Policy Evaluation

An interrupted time series study judges a health policy by comparing what happened after its start date with where the earlier run of monthly or quarterly figures was already heading. The gap between the observed series and that projected path is read as the policy effect, split into an immediate jump in level and a gradual change in slope. Health economists use the design when a reimbursement limit or a pay for performance scheme reaches a whole population on one date, so no randomised comparison exists. Wagner and colleagues, writing on medication use research, call it the strongest quasi-experimental approach for evaluating the longitudinal effects of interventions. Its estimates of changes in drug use and spending can feed budget impact and cost-effectiveness work once the threats to validity are addressed.

The earlier trend stands in for the missing comparison group

Like any quasi-experimental design, the method needs an estimate of the outcome that would have been seen without the intervention. Lopez Bernal, Cummins and Gasparrini call the expected continuation of the pre-existing trend, in the absence of the intervention, the counterfactual. Wagner and colleagues describe each segment by two parameters, its level at the start of the segment and its trend, meaning the slope during the segment.

A change in level is an abrupt effect, such as a drop in prescriptions when a payment limit starts. A change in trend is a gradual effect, such as uptake of a guideline that builds month by month. Because many points are measured on each side, the design controls for the secular trend, whereas Wagner and colleagues warn that pre and post studies comparing two points may easily lead to invalid results through failure to control for pre-existing trends.

Segmented regression estimates the level and slope changes

The standard analysis fits a regression line to each segment. The form below follows the published correction to the Lopez Bernal tutorial, in which the slope term counts time from the start of the intervention.

$$Y_t = \beta_0 + \beta_1 T + \beta_2 X_t + \beta_3 (T - T_0) X_t + \varepsilon_t$$

where $Y_t$ is the outcome at time $t$, $T$ is time elapsed since the start of the series, $X_t$ equals 0 before the intervention and 1 after it, $T_0$ is the time at which the intervention begins, $\beta_0$ is the baseline level at $T = 0$, $\beta_1$ is the pre-intervention trend per period, $\beta_2$ is the level change at the start of the intervention, $\beta_3$ is the change in slope after it, and $\varepsilon_t$ is an error term that may be correlated over time.

The post-intervention slope is $\beta_1 + \beta_3$. The effect at a post-intervention time is the fitted value with the intervention minus the fitted value from baseline level and trend alone.

$$\Delta_t = \hat{Y}_t - \hat{Y}^{\text{cf}}_t = \hat{\beta}_2 + \hat{\beta}_3 (T - T_0)$$

where $\Delta_t$ is the absolute effect at time $t$, $\hat{Y}_t$ is the fitted outcome with the intervention terms, and $\hat{Y}^{\text{cf}}_t = \hat{\beta}_0 + \hat{\beta}_1 T$ is the fitted counterfactual. Wagner and colleagues also express the effect as a percentage of the counterfactual value.

The coding of the slope term matters. If the interaction is written as $T$ times $X_t$ without subtracting $T_0$, the fitted line is the same but the coefficient on $X_t$ no longer measures the jump at the intervention date, and the correction notice states that the original definition could lead to erroneous interpretations of the estimated parameters. Counts such as admissions are often modelled with Poisson regression, with a population offset and a scaling adjustment for over-dispersion, while continuous outcomes use linear regression.

Choosing the impact model before looking at the results

Lopez Bernal and colleagues treat the impact model as a choice made before analysis. The analyst states whether a level change, a slope change or both are expected, and whether the effect is immediate, lagged or temporary. Letting the outcome data choose the model, they warn, raises the chance of finding an effect produced by random fluctuation.

Wagner and colleagues describe a brief rise in prescriptions in the month before the New Hampshire Medicaid three-drug payment limit, an anticipatory effect they excluded from the model. Where an effect takes months to appear, as when patients moved to substitute drugs after Medicaid stopped paying for drugs deemed ineffective, the lag period can be excluded or modelled as its own segment.

How many data points and how to handle correlated errors

There is no single minimum length. The Cochrane Effective Practice and Organisation of Care (EPOC) group suggests excluding studies that lack a clearly defined point in time when the intervention occurred and at least three data points before and three after it. Wagner and colleagues cite a general recommendation of 12 monthly points before and 12 after, noting that it is not based on power and that 24 monthly points allow seasonal variation to be assessed. In simulations of autoregressive error models by Zhang, Wagner and Ross-Degnan, power rose with sample size and effect size, tended to fall as autocorrelation increased, and was lower when points were split unevenly around the intervention.

Ordinary least squares assumes independent errors, yet neighbouring months tend to resemble each other. Wagner and colleagues note that ignoring this autocorrelation may lead to underestimated standard errors and overstated significance, and they test for it with the Durbin-Watson statistic. Lopez Bernal and colleagues recommend checking residuals and the partial autocorrelation function, adjusting for seasonality with calendar-month terms, Fourier terms or splines, and using Prais regression or ARIMA models where residual autocorrelation remains.

Co-interventions and the case for a control series

Wagner and colleagues state that, with correctly specified segments, confounding is limited to factors related to the outcome that changed at the time of the intervention, such as simultaneous interventions, shifts in the study population or changes in how the outcome is measured. The consistency of recording needs particular scrutiny in routinely collected data.

EPOC notes that a single series offers no way to assess the impact of concurrent events. Wagner and colleagues describe several kinds of control, among them a different group not exposed to the intervention and a related outcome in the same group that the intervention should not affect. In a study of Medicaid reimbursement for drugs deemed ineffective, use of insulin, digoxin and eye ointments stayed stable, supporting the reading that the change in the study drugs came from the policy. A control group turns the analysis into a close relative of difference-in-differences, and where no single comparison unit fits, a weighted synthetic control can be built. Lopez Bernal, Cummins and Gasparrini caution that a control can exclude some concurrent events while introducing new sources of bias.

Interrupted time series as evidence for health economic decisions

Wagner and colleagues frame segmented regression as a way to evaluate interventions intended to improve the quality of medication use or contain costs, drawing on dispensing files, insurance claims and cost data. In their New Hampshire example the three-drug limit cut prescriptions by 2.4 per patient per month, or 46%, six months after it started, relative to the counterfactual. Yet the percentage of chronically ill enrolees living in nursing homes rose more steeply in New Hampshire than in New Jersey, which had no limit. A saving on the drug budget can coincide with higher costs elsewhere, so the perspective of an economic analysis built on such estimates has to be stated.

UK pay for performance gives a second example. Kontopantelis and colleagues used the Quality and Outcomes Framework (QOF), introduced in the 2004-05 financial year, to show an effect on recorded quality of care for diabetes and asthma but not for heart disease. Serumaga and colleagues studied 470,725 patients with hypertension over 48 months before QOF and 36 months after. Care was already stable or improving beforehand, and pay for performance had no discernible effect on processes of care or on hypertension related outcomes.

NICE lists interrupted time series among the common quasi-experimental methods in its real-world evidence framework (ECD9), noting that such methods are frequently used in public health settings. The results are a form of real-world evidence. Monthly effects can be summed over the observed period for a budget impact analysis, but projecting the post-intervention slope beyond the observed window is an extrapolation that assumes the slope stays unchanged.

Worked example with an illustrative prescribing restriction

All figures are illustrative. A health system restricts prescribing of a high-cost medicine and records monthly spending for 36 months before and 24 months after, so $T$ runs 1 to 60 and $T_0 = 37$. The fitted model gives $\hat{\beta}_0$ = GBP 4,000,000, $\hat{\beta}_1$ = GBP 120,000 per month, $\hat{\beta}_2$ = minus GBP 2,400,000 and $\hat{\beta}_3$ = minus GBP 80,000 per month.

1. Counterfactual at month 49, 12 months after the start. 4,000,000 + 120,000 × 49 = GBP 9,880,000.

2. Fitted spending with the restriction at month 49. The slope change contributes 80,000 × 12 = 960,000, so spending is 9,880,000 minus 2,400,000 minus 960,000 = GBP 6,520,000.

3. Absolute and relative effect. The effect is minus 2,400,000 minus 960,000 = minus GBP 3,360,000 per month, and 3,360,000 / 9,880,000 = 0.340, a fall of 34.0% against the counterfactual.

4. Post-restriction slope. 120,000 minus 80,000 = GBP 40,000 per month, so spending still rises, only more slowly.

5. Cumulative effect over the 24 observed months. The monthly saving is 2,400,000 plus 80,000 times $(T - 37)$ for $T$ = 37 to 60. The level part is 24 × 2,400,000 = 57,600,000. The slope part is 80,000 × 276 = 22,080,000, since 0 + 1 + ... + 23 = 276. The total is 57,600,000 + 22,080,000 = GBP 79,680,000.

6. A wider perspective. Suppose a second illustrative series of hospital costs for the same patients shows a level increase of GBP 900,000 per month. At month 49 the net saving for the health system is 3,360,000 minus 900,000 = GBP 2,460,000 per month.

7. The coding trap. With the slope term coded as $T \times X_t$, the same fitted line gives a coefficient on $X_t$ of $\hat{\beta}_2 - \hat{\beta}_3 T_0$ = minus 2,400,000 + 80,000 × 37 = minus 2,400,000 + 2,960,000 = plus GBP 560,000. Read as a level change, it would suggest the restriction raised spending.

The restriction is associated with a large and growing fall in drug spending against the projected trend, partly offset by higher hospital costs. The causal reading holds only if no other event changed spending in month 37.

Limits and common misreadings

Most errors come from forgetting that the counterfactual is a projection. It rests on the assumption that the earlier trend would have continued unchanged.

1. Assuming linearity far from the interruption. Wagner and colleagues note that a linear trend often holds only over short intervals.
2. Treating the jump as causal without checking concurrent events. Co-interventions and coding changes at the same date can produce the same pattern.
3. Reading population results as individual effects. Segmented regression usually aggregates data by time point and, as Wagner and colleagues state, does not allow control for individual-level covariates.
4. Selecting the model by stepwise elimination alone. Wagner and colleagues advise keeping important measured confounders and the baseline trend regardless of statistical significance.
5. Judging economic importance by p-values alone. In step 5 the slope change accounts for GBP 22,080,000, so a slope change that is not significant on its own can still matter for a budget.

Compared with a randomised controlled trial, the design gives up protection against events that coincide with the intervention in exchange for routine data covering whole populations. Kontopantelis and colleagues call it arguably the next best approach when randomisation is not possible.

Sources

• Cochrane Effective Practice and Organisation of Care (EPOC). What study designs should be included in an EPOC review and what should they be called? EPOC resources for review authors; 2017. https://epoc.cochrane.org/sites/epoc.cochrane.org/files/public/uploads/Resources-for-authors2017/what_study_designs_should_be_included_in_an_epoc_review.pdf
• Kontopantelis E, Doran T, Springate DA, Buchan I, Reeves D. Regression based quasi-experimental approach when randomisation is not an option: interrupted time series analysis. BMJ. 2015;350:h2750. https://doi.org/10.1136/bmj.h2750
• Lopez Bernal J, Cummins S, Gasparrini A. Interrupted time series regression for the evaluation of public health interventions: a tutorial. International Journal of Epidemiology. 2017;46(1):348-355. https://doi.org/10.1093/ije/dyw098
• Lopez Bernal J, Cummins S, Gasparrini A. Corrigendum to: Interrupted time series regression for the evaluation of public health interventions: a tutorial. International Journal of Epidemiology. 2021;50(3):1045. https://doi.org/10.1093/ije/dyaa118
• Lopez Bernal J, Cummins S, Gasparrini A. The use of controls in interrupted time series studies of public health interventions. International Journal of Epidemiology. 2018;47(6):2082-2093. https://doi.org/10.1093/ije/dyy135
• National Institute for Health and Care Excellence. NICE real-world evidence framework (ECD9). Published 23 June 2022. https://www.nice.org.uk/corporate/ecd9/chapter/methods-for-real-world-studies-of-comparative-effects
• Serumaga B, Ross-Degnan D, Avery AJ, Elliott RA, Majumdar SR, Zhang F, Soumerai SB. Effect of pay for performance on the management and outcomes of hypertension in the United Kingdom: interrupted time series study. BMJ. 2011;342:d108. https://doi.org/10.1136/bmj.d108
• Wagner AK, Soumerai SB, Zhang F, Ross-Degnan D. Segmented regression analysis of interrupted time series studies in medication use research. Journal of Clinical Pharmacy and Therapeutics. 2002;27(4):299-309. https://doi.org/10.1046/j.1365-2710.2002.00430.x
• Zhang F, Wagner AK, Ross-Degnan D. Simulation-based power calculation for designing interrupted time series analyses of health policy interventions. Journal of Clinical Epidemiology. 2011;64(11):1252-1261. https://doi.org/10.1016/j.jclinepi.2011.02.007