Last Observation Carried Forward

Last Observation Carried Forward, Its Bias and Alternatives in Trial-Based Economic Evaluation

Last observation carried forward was for many years a routine way to complete longitudinal trial data, and it still appears in clinical and economic analyses. Its simplicity hides a strong assumption about people who leave a study, one that can push a treatment effect, a QALY estimate or an incremental cost-effectiveness ratio in either direction while overstating precision. This page states the rule and its assumption, shows with an illustrative trial how it shifts incremental QALYs and the ICER, and covers the regulatory position, the alternatives and missing cost and utility data.

How the carry-forward rule completes a longitudinal dataset

Last observation carried forward (LOCF) is a form of imputation for repeated measures. Each participant who stops providing data has every later scheduled value filled with the value recorded at their last completed visit, and the completed dataset is then analysed as if nothing were missing. Written as a rule, it takes the form below.

$$\tilde{y}_{ik} = y_{i d_i} \quad \text{for all } k > d_i$$

where $\tilde{y}_{ik}$ is the value used for participant $i$ at scheduled visit $k$, $y_{i d_i}$ is the participant's last observed value and $d_i$ is the index of the last visit at which the participant was observed. Visits up to $d_i$ keep their observed values.

The rule is a single imputation, in which one value replaces each missing value and is then treated as if it had been measured. Its appeal is that everyone randomised keeps an outcome at every visit without any model of why data are missing. The National Research Council panel recorded that it had been common in chronic pain trials.

The assumption that nothing changes after dropout

Every method for missing data rests on an assumption about the unobserved values. For LOCF the National Research Council panel called it the strong assumption that a participant's outcome does not change after dropout. Formally, for a participant missing only the final visit $K$, LOCF predicts the final value as the previous one with no uncertainty, as follows.

$$\mathbb{E}\left[Y_K \mid Y_1, \ldots, Y_{K-1}\right] = Y_{K-1}, \qquad \operatorname{Var}\left(Y_K \mid Y_1, \ldots, Y_{K-1}\right) = 0$$

where $Y_k$ is the outcome at visit $k$ and $K$ is the final scheduled visit.

LOCF is sometimes mistakenly considered valid when data are missing completely at random or missing at random. The National Research Council panel explained that, in general, it makes a missing not at random assumption. It predicts a dropout's final value as their last value with certainty, so the distribution of the final outcome for a dropout differs from that for a participant with the same observed history who stayed, and the assumption is neither missing completely at random nor missing at random.

Lachin gave the resulting bias a simple form. When a fraction of final values are missing completely at random and replaced by earlier values, the analysed final-visit mean is the mixture below.

$$\tilde{\mu} = \tau \mu_{\text{last}} + (1 - \tau)\, \mu_{\text{end}}$$

where $\tilde{\mu}$ is the mean of the observed and imputed final values, $\tau$ is the fraction imputed, $\mu_{\text{last}}$ is the mean at the earlier visit whose values are carried forward and $\mu_{\text{end}}$ is the true mean at the final visit. The estimate is unbiased only when the carried-forward values have the same distribution as the missing final values, which is the one thing the analyst cannot observe.

When LOCF flatters a treatment and when it penalises one

The direction of the bias depends on the course of the condition and on who leaves, and when. Lachin showed that, when the same fraction of each arm is missing completely at random, the LOCF difference is biased upwards when the between-arm difference at the earlier visit exceeds the true final difference, and downwards when it is smaller. With unequal dropout each arm carries its own error, as follows.

$$\tilde{\Delta} - \Delta = \tau_T\left(\mu_{T,\text{last}} - \mu_{T,\text{end}}\right) - \tau_C\left(\mu_{C,\text{last}} - \mu_{C,\text{end}}\right)$$

where $\tilde{\Delta}$ is the analysed between-arm difference at the final visit, $\Delta$ is the true difference, $\tau_T$ and $\tau_C$ are the fractions imputed in the treatment and control arms, and the $\mu$ terms are each arm's means at the last observed visit and at the final visit. Even a widening true difference can then be overstated, as the worked example shows.

The EMA guideline gives two practical cases. In a deteriorating condition such as Alzheimer's disease, LOCF is very likely to be overly optimistic in both arms, and if active-arm participants withdraw earlier, for example because of adverse events, the comparison may favour the test product, so LOCF is not considered appropriate there. In a condition expected to improve spontaneously, such as depression, LOCF might be conservative when experimental-arm participants withdraw earlier and more often. The National Research Council panel cautioned that LOCF is anticonservative where participants off study treatment generally do worse over time, so that a poorly tolerated treatment can be made to look much better than its control.

Precision suffers too. The panel noted that single imputation overstates precision because imputed values are treated as true, and that LOCF can artificially increase the sample size for some analyses.

Worked example of differential dropout, utilities and incremental QALYs

The figures are illustrative. A 12-month randomised controlled trial in a slowly deteriorating condition records a preference-based utility at baseline, 6 months and 12 months. If every participant had been followed, the arm means would be as shown below.

After the 6-month visit, 40% of the treatment arm and 10% of the control arm stop providing data, for reasons the example assumes are unrelated to utility. With visits at 0, 0.5 and 1 year, the trapezoidal rule gives QALYs over the year as follows.

$$Q = 0.25\left(u_0 + 2u_6 + u_{12}\right)$$

where $Q$ is mean QALYs per patient over the year and $u_0$, $u_6$ and $u_{12}$ are the mean utilities at baseline, 6 months and 12 months.

1. Apply LOCF to the 12-month means. In the treatment arm the analysed 12-month mean becomes 0.4 × 0.68 + 0.6 × 0.64 = 0.656, against a true 0.64. In the control arm it becomes 0.1 × 0.64 + 0.9 × 0.58 = 0.586, against a true 0.58. The 12-month difference rises from 0.06 to 0.07, an overstatement of about 17%, matching the formula above, since 0.4 × 0.04 − 0.1 × 0.06 = 0.010.

2. Calculate QALYs. With full data the treatment arm has 0.25 × 2.70 = 0.675 QALYs and the control arm 0.25 × 2.56 = 0.64 QALYs, an incremental gain of 0.035. With LOCF the treatment arm has 0.25 × 2.716 = 0.679 QALYs and the control arm 0.25 × 2.566 = 0.6415 QALYs, an incremental gain of 0.0375, about 7% larger.

3. Carry the error into the ICER. With an illustrative incremental cost of £1,500 per patient, observed for everyone, the ICER with full data is about £42,900 per QALY (1,500 / 0.035 = 42,857). Under LOCF it is 1,500 / 0.0375 = 40,000, or £40,000 per QALY.

4. Interpret. In both arms dropouts are, on average, frozen at a 6-month value above the arm's true 12-month mean, so both arms gain from the imputation, but the treatment arm, with four times the dropout, gains more (0.016 against 0.006 at 12 months). The true difference widens from 0.04 to 0.06 over the second half-year, yet LOCF still overstates it. Because dropout here is unrelated to utility, the completers would recover the true 12-month means; LOCF is biased even so. Sicker dropouts or an improving condition would change the size and even the sign of the error.

The regulatory and methodological position

The National Research Council panel, convened at the request of the US Food and Drug Administration, recommended in 2010 that single imputation methods such as last and baseline observation carried forward should not be used as the primary approach to missing data unless the assumptions behind them are scientifically justified. The panel stated its view that in nearly all cases there are better alternatives resting on more reasonable assumptions.

The EMA Guideline on Missing Data in Confirmatory Clinical Trials (EMA/CPMP/EWP/1776/99 Rev. 1), adopted in June 2010 and in effect from 1 January 2011, states that LOCF produces an unbiased estimate of the treatment effect only under certain restrictive assumptions. Its test for any primary analysis, LOCF included, is that an important bias in favour of the experimental treatment can reasonably be excluded and the confidence interval does not underestimate the variability of the estimated effect to an important extent.

The ICH E9(R1) addendum on estimands and sensitivity analysis, adopted in November 2019, does not name LOCF. It requires the estimand, the treatment effect of interest, to be defined first, including how intercurrent events such as discontinuation of randomised treatment are handled. Missing data are then defined relative to that estimand, their handling should rest on clinically plausible assumptions guided, where possible, by the estimand's strategies, and a sensitivity analysis, such as a tipping point analysis, should test whether conclusions survive departures from those assumptions. Values after death are not missing data. ICH E9(R1) distinguishes data that do not exist because of an intercurrent event, and carrying a utility forward past death would credit QALYs that were never lived.

Alternatives that state the assumption and carry the uncertainty

The alternatives differ from LOCF in two ways. They state the assumption about the missing values openly, and they carry the uncertainty about those values into the result.

• Multiple imputation. Several completed datasets are drawn and combined, so the variance includes the between-imputation component that single imputation misses.
• Likelihood-based mixed-effects models. The EMA guideline names mixed-effect models for repeated measures (MMRM) and states that likelihood-based methods are applicable under missing completely at random and missing at random assumptions.
• Departures from missing at random. The EMA guideline rejects presenting missing at random models as a comprehensive solution and lists pattern mixture models, selection models and shared parameter models for exploring other scenarios. Cro and colleagues describe delta-based imputation, which typically adds an offset to the expected value of the missing data, and reference-based imputation, which draws on observed data in other trial groups, typically other arms.

Missing costs and utilities in trial-based economic evaluation

Trial-based cost-effectiveness analysis is exposed to missing costs and outcomes. Faria and colleagues note that costs and QALYs are cumulative measures built from longitudinal data, so a missing value at one time point leaves the total missing. Carrying the last utility or cost forward gives every patient a total, but assumes that health-related quality of life, or resource use per period, stayed constant after the last questionnaire.

Their guidance sets three principles, namely a plausible assumption about the missing data mechanism, a base-case method that fits it, and sensitivity analysis under other assumptions. They conclude that single imputation methods are not appropriate for missing outcome data and class last-value carried forward among the ad hoc methods to avoid, because they cannot reflect the uncertainty in the missing data and often make implausible assumptions.

The National Research Council panel noted that when imputed values are used to estimate an area under the curve, as QALYs are, LOCF may understate statistical uncertainty, which flows through to the cost-effectiveness acceptability curve. A 2017 review by Gabrio, Mason and Baio of 81 trial-based cost-effectiveness analyses published between 2009 and 2015 concluded that missing data were still often inadequately handled.

LOCF beside nearby approaches

Several methods address the same gaps in follow-up but rest on different assumptions. Baseline observation carried forward assumes a return to the starting value; the EMA guideline suggests it may suit chronic pain trials. Complete case analysis drops participants with missing outcomes; the EMA guideline notes that it violates the intention to treat principle and cannot be recommended as the primary analysis in a confirmatory trial. Intention-to-treat analysis is a principle about whom to analyse; LOCF is one way of supplying the outcomes it needs, and its validity rests on its own assumption.

Common misreadings of LOCF

LOCF is often defended as conservative, but that holds only for particular combinations of disease course and dropout, and the EMA guideline and the National Research Council panel each describe settings where it favours the experimental treatment. Nor is it safe when data are missing completely at random, because a single-arm mean is still biased whenever the mean changes after the last observed visit, and a between-arm difference whenever that difference changes or dropout differs between arms. Finally, the imputed values carry no uncertainty, so standard errors and probabilities of cost-effectiveness can be too confident.

Sources

• Cro S, Morris TP, Kenward MG, Carpenter JR. Sensitivity analysis for clinical trials with missing continuous outcome data using controlled multiple imputation: a practical guide. Statistics in Medicine. 2020;39(21):2815-2842. doi:10.1002/sim.8569
• European Medicines Agency, Committee for Medicinal Products for Human Use. Guideline on Missing Data in Confirmatory Clinical Trials. EMA/CPMP/EWP/1776/99 Rev. 1. London: EMA; 2010.
• Faria R, Gomes M, Epstein D, White IR. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. PharmacoEconomics. 2014;32(12):1157-1170. doi:10.1007/s40273-014-0193-3
• Gabrio A, Mason AJ, Baio G. Handling missing data in within-trial cost-effectiveness analysis: a review with future recommendations. PharmacoEconomics Open. 2017;1(2):79-97. doi:10.1007/s41669-017-0015-6
• International Council for Harmonisation. ICH E9(R1) Addendum on Estimands and Sensitivity Analysis in Clinical Trials to the Guideline on Statistical Principles for Clinical Trials. Step 4, 20 November 2019.
• Lachin JM. Fallacies of last observation carried forward analyses. Clinical Trials. 2016;13(2):161-168. doi:10.1177/1740774515602688
• National Research Council, Panel on Handling Missing Data in Clinical Trials. The Prevention and Treatment of Missing Data in Clinical Trials. Washington, DC: National Academies Press; 2010. doi:10.17226/12955