Listwise Deletion

Listwise Deletion in Trial-Based Economic Evaluation

Listwise deletion, usually called complete-case analysis in health economics, restricts an analysis to the participants who have a recorded value for every variable it uses. In a trial-based cost-effectiveness analysis, one unreturned resource-use questionnaire or one missing EQ-5D score at any visit removes a patient from the whole cost and QALY comparison. It is the default in many statistical packages, and it remained the most common primary approach in trial-based evaluations in the NIHR Health Technology Assessment journal for 2013 to 2015. For regression estimates, its validity turns mainly on whether the chance of being a complete case depends on the outcome once covariates are allowed for. Any bias or lost precision then carries through to the net monetary benefit behind the reimbursement decision.

How the complete-case rule shrinks a cost-effectiveness sample

The rule is applied to the variables in the analysis model, so one dataset can lose different people in different analyses. A participant counts as a complete case only if every one of those variables is recorded. In symbols,

$$C_i = \prod_{j=1}^{p} (1-M_{ij}), \qquad n_{cc} = \sum_{i=1}^{n} C_i$$

where $M_{ij}$ equals 1 if variable $j$ is missing for participant $i$ and 0 if it is observed, $p$ is the number of analysis variables (each cost component, EQ-5D visit and covariate), $C_i$ equals 1 only when participant $i$ is observed on all of them, $n$ is the number randomised and $n_{cc}$ is the number of complete cases. The complete-case mean is then $\bar{Y}_{cc} = \frac{1}{n_{cc}} \sum_{i} C_i Y_i$, where $Y_i$ is participant $i$'s total cost or total QALYs.

Faria and colleagues explain why economic outcomes are exposed. Costs and QALYs are cumulative measures built from longitudinal data, each component may have its own missing data pattern, and a missing value at one time point or for one component makes the total missing too. They describe the complete-case analysis as inefficient whenever there is more than one follow-up assessment. Leurent and colleagues found a median of 63% of trial participants with complete cost-effectiveness data, with the proportion falling as the number of follow-up assessments rose.

Van Buuren adds that different tables from one dataset often rest on different subsamples, so separate cost and QALY regressions can describe different patients, the problem covered on the available case analysis page, whereas a joint model such as seemingly unrelated regression keeps one set.

Conditions under which complete cases give unbiased estimates

Van Buuren states that if data are missing completely at random, listwise deletion gives unbiased means, variances and regression weights, with standard errors correct for the reduced subset. Faria and colleagues extend this to covariate-dependent missingness, where missingness depends on observed baseline covariates but not on outcomes, provided the analysis model includes every baseline variable that predicts both outcome and missingness.

Hughes and colleagues give a more general rule. For most regression models, a complete-case analysis is unbiased when the chance of being a complete case does not depend on the outcome after the covariates are taken into account, which includes some data missing not at random. The rule protects regression weights, but Van Buuren notes that quantities depending on the marginal distribution of the outcome, such as its mean, need the stronger assumption of missing completely at random.

In large samples, the source of bias shows when a trial arm is split into subgroups of patients.

$$\bar{Y}_{cc} \approx \frac{\sum_{s} \pi_s r_s \mu_s}{\sum_{s} \pi_s r_s}$$

Here $s$ indexes subgroups within the arm, $\pi_s$ is the share of the arm's randomised patients in subgroup $s$, $r_s$ is the probability that a member of $s$ is a complete case and $\mu_s$ is the subgroup's mean outcome. The complete-case mean equals the true arm mean, the sum of $\pi_s \mu_s$, only when completion rates and subgroup means are uncorrelated across subgroups, weighting by $\pi_s$, for example when either is the same in every subgroup. Van Buuren, citing Little and Rubin, notes that bias in a mean grows with the gap between observed and missing cases and with the proportion missing. Faria and colleagues add that data are unlikely to be missing completely at random if the proportion missing differs between treatment groups.

Precision lost with each discarded record

Even when listwise deletion is unbiased, it estimates from fewer patients. Under missing completely at random, with the outcome's variance unchanged, the standard error grows roughly as follows.

$$\frac{SE_{cc}}{SE_{full}} \approx \sqrt{\frac{n}{n_{cc}}}$$

Here $SE_{cc}$ is the standard error from the complete cases and $SE_{full}$ the one had all $n$ participants been observed. Sterne and colleagues note that missing values across several variables often exclude a substantial share of the sample, with a matching loss of statistical power.

How much the discarded records could have added depends on the pattern. Hughes and colleagues explain that when only the outcome is missing and no auxiliary variables are available, incomplete cases hold no information about the regression coefficients and complete-case analysis is the best choice, while they hold most when many covariates each have small amounts missing on different people. Leurent and colleagues judged a complete-case primary analysis hard to justify with repeated measurements, because the loss of power tends to be large.

Published evaluations where complete cases changed the estimate

Noble and colleagues found complete-case analysis the most popular approach in trial-based evaluations published since 2003, and its use was increasing. Among the 47 evaluations in Leurent and colleagues' review with at least 5% of participants incomplete, it was the primary method in 20 (43%), against 14 (30%) for multiple imputation, and only 2 (4%) examined departures from missing at random.

In the REFLUX trial of surgery for gastro-oesophageal reflux disease, analysed by Faria and colleagues, the complete-case incremental cost was GBP 1,668, against GBP 1,305 after multiple imputation and GBP 1,338 from a mixed model. Baseline-adjusted incremental QALYs were 0.301, 0.244 and 0.227. The authors read the higher complete-case estimates as bias from assuming missing completely at random, although all three methods found surgery cost-effective. Sterne and colleagues describe a lung cancer trial by Burton and colleagues in which costs were complete for 82 of 115 patients. The complete-case mean net monetary benefit was minus GBP 3,346, implying chemotherapy was not cost-effective, while multiple imputation gave GBP 1,186, implying it was, with wide confidence intervals.

Worked example with admissions under-represented among complete cases

All numbers in this example are illustrative. A trial randomises 300 patients to a new care pathway and 300 to usual care. In usual care 60 patients (20%) are admitted to hospital, with a mean 12-month cost of GBP 9,000, and the other 240 cost GBP 1,000 on average. The pathway costs GBP 700 per patient and halves admissions, so 30 admitted patients cost GBP 9,700 and 270 others GBP 1,700. In both arms, admitted patients return all their questionnaires half the time, against 80% for everyone else.

1. True arm means. Usual care costs 60 × 9,000 + 240 × 1,000 = 780,000, or GBP 2,600 per patient. The pathway costs 30 × 9,700 + 270 × 1,700 = 750,000, or GBP 2,500 per patient. The true incremental cost is minus GBP 100.

2. Complete cases. Usual care keeps 30 admitted and 192 other patients, 222 in all. The pathway keeps 15 and 216, 231 in all.

3. Complete-case means. Usual care gives 30 × 9,000 + 192 × 1,000 = 462,000, and 462,000 / 222 = 2,081, or GBP 2,081. The pathway gives 15 × 9,700 + 216 × 1,700 = 512,700, and 512,700 / 231 = 2,219, or GBP 2,219. The complete-case incremental cost is plus GBP 138, so the saving becomes an extra cost.

4. Source of the bias. Admitted patients are 20% of usual care but 30 / 222, or 13.5%, of its complete cases, and 10% of the pathway but 15 / 231, or 6.5%, of its complete cases. The gap in admission rates is 10 percentage points in truth and 7.0 among complete cases. At GBP 8,000 per avoided admission, the saving is GBP 800 per patient in truth but about GBP 562 among complete cases, which no longer covers the GBP 700 pathway cost.

5. Decision. Suppose the QALY gain is 0.004 in both analyses. At an illustrative GBP 25,000 per QALY, the true incremental net monetary benefit is 25,000 × 0.004 + 100 = 200, or GBP 200, but the complete-case value is 100 minus 138, about minus GBP 38. A cost-effective pathway is rejected.

6. Precision. Had 453 of 600 patients been kept completely at random, standard errors would be about 15% larger, since the square root of 600 / 453 is about 1.15.

Adjusting for baseline covariates would not repair this, because completion depends on the cost outcome itself. Admissions recorded in hospital data for everyone could enter an imputation model as an auxiliary variable, an advantage of multiple imputation that Hughes and colleagues describe.

Positions of regulators and methods guides

The European Medicines Agency guideline on missing data in confirmatory clinical trials (EMA/CPMP/EWP/1776/99 Rev. 1, adopted in 2010) states that complete-case analysis cannot be recommended as the primary analysis in a confirmatory trial, since in general it violates the intention-to-treat principle and is subject to bias. It accepts the approach in exploratory studies and as a supportive sensitivity analysis. The guideline concerns efficacy, but an intention-to-treat analysis of costs and QALYs faces the same problem.

Faria and colleagues call complete-case analysis a useful starting point and benchmark that should not form the base case of a within-trial evaluation, and state that removing individuals with missing baseline values is rarely adequate. Sterne and colleagues recommend reporting complete-case results beside multiple imputation results and explaining important differences. Van Buuren asks that listwise deletion be a conscious, stated decision.

Listwise deletion beside pairwise deletion, imputation and weighting

Pairwise deletion keeps every observed value for each quantity but lets estimates rest on different patients. Mean imputation and last observation carried forward keep everyone, but Faria and colleagues group them with complete-case analysis as ad hoc methods that cannot reflect the uncertainty from missing values. Multiple imputation and likelihood-based mixed-effects models use the partial records under missing at random.

In simulations with missing covariate values, White and Carlin found that under missing completely at random both had negligible bias and multiple imputation was more efficient. When missingness was independent of the outcome given the covariates, the complete-case analysis had negligible bias and multiple imputation was biased away from the null, although with more general mechanisms bias tended to be smaller for multiple imputation. Inverse probability weighting keeps the complete cases but weights each by the inverse of its probability of being complete, and Faria and colleagues consider it inappropriate when patients miss a visit and later return. Departures from missing at random need sensitivity methods such as a selection model.

Misreadings of complete-case results

Most errors treat the retained sample size, or agreement between methods, as evidence about the missing data mechanism. Five recur in trial-based evaluations.

1. Missing completely at random is required for any unbiased result. Hughes and colleagues call this a widespread belief and show that complete-case regression can be unbiased under some missing at random and missing not at random mechanisms.
2. Agreement with multiple imputation shows the result is insensitive to missing data. Leurent and colleagues call this misleading, because both methods assume missingness is independent of the unobserved values.
3. The method with the smaller standard error is better. White and Carlin state that the choice should not rest on comparing standard errors.
4. A complete-case analysis is an intention-to-treat analysis. It omits randomised patients with partial data, as Faria and colleagues and the EMA guideline point out.
5. The observed data can confirm the assumption. Faria and colleagues note that missingness not at random can usually not be ruled out, since the unobserved values are unknown.

Sources

• European Medicines Agency, Committee for Medicinal Products for Human Use. Guideline on Missing Data in Confirmatory Clinical Trials. EMA/CPMP/EWP/1776/99 Rev. 1. 2 July 2010, effective 1 January 2011. https://www.ema.europa.eu/en/documents/scientific-guideline/guideline-missing-data-confirmatory-clinical-trials_en.pdf
• Faria R, Gomes M, Epstein D, White IR. A guide to handling missing data in cost-effectiveness analysis conducted within randomised controlled trials. PharmacoEconomics. 2014;32(12):1157-1170. https://doi.org/10.1007/s40273-014-0193-3
• Hughes RA, Heron J, Sterne JAC, Tilling K. Accounting for missing data in statistical analyses: multiple imputation is not always the answer. International Journal of Epidemiology. 2019;48(4):1294-1304. https://doi.org/10.1093/ije/dyz032
• Leurent B, Gomes M, Carpenter JR. Missing data in trial-based cost-effectiveness analysis: an incomplete journey. Health Economics. 2018;27(6):1024-1040. https://doi.org/10.1002/hec.3654
• Noble SM, Hollingworth W, Tilling K. Missing data in trial-based cost-effectiveness analysis: the current state of play. Health Economics. 2012;21(2):187-200. https://doi.org/10.1002/hec.1693
• Sterne JAC, White IR, Carlin JB, Spratt M, Royston P, Kenward MG, Wood AM, Carpenter JR. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338:b2393. https://doi.org/10.1136/bmj.b2393
• Van Buuren S. Flexible Imputation of Missing Data. 2nd ed. CRC Press; 2018. Sections 1.3.1 and 2.7. https://stefvanbuuren.name/fimd/
• White IR, Carlin JB. Bias and efficiency of multiple imputation compared with complete-case analysis for missing covariate values. Statistics in Medicine. 2010;29(28):2920-2931. https://doi.org/10.1002/sim.3944