Leave-One-Out

Leave-One-Out Cross-Validation in Health Economic Models

Leave-one-out cross-validation measures how well a model predicts each unit in a dataset when that unit has played no part in fitting the model. The unit can be a patient in a cost regression, an observed event time in a survival model or a whole trial in a meta-analysis, and the method is the most extreme form of K-fold cross-validation, with as many folds as units. In health economics it is mainly used for prediction models built on small samples, for the LOOIC statistic that compares Bayesian survival models before extrapolation and for checking whether one trial in a synthesis is out of line with the rest. Fold design, leakage and nested tuning work as for any cross-validation, and the distinctive features of leaving out a single unit are an estimate free of random splitting and a shortcut that avoids refitting linear models.

Predicting each observation from all the others

The procedure fits the model $n$ times. On pass $i$ the fit uses every observation except observation $i$, the fitted model predicts observation $i$, and the loss for that single prediction is recorded. Hastie, Tibshirani and Friedman describe it as the case of K-fold cross-validation in which the number of folds equals the sample size, so that for the $i$th observation the fit is computed using all the data except the $i$th.

$$\mathrm{CV}_{(n)} = \frac{1}{n} \sum_{i=1}^{n} L\big(y_i, \hat f^{-i}(x_i)\big)$$

where $\mathrm{CV}_{(n)}$ is the leave-one-out estimate of prediction error, $n$ is the number of observations, $y_i$ and $x_i$ are the outcome and predictors for observation $i$, $\hat f^{-i}$ is the model fitted with observation $i$ removed and $L$ is the loss function, for example squared error for a cost or the Brier score for a predicted probability. With squared-error loss the result is a held-out mean squared error.

Because there is only one way to leave out each unit, the estimate involves no random partition. Two analysts with the same data and the same model-building procedure obtain the same value, whereas K-fold results change with the random seed that allocates folds. Every learned step, including variable selection and tuning, still has to be repeated inside each of the $n$ fits, or the held-out observation leaks into its own prediction.

A shortcut for linear cost models

Refitting a model $n$ times can be slow. Jones, reviewing econometric models for health care costs, notes that setting the left-out group to one observation means estimating the model $n$ times, which may be computationally expensive. For linear fitting methods under squared-error loss no refitting is needed, because the leave-one-out residuals can be recovered by fitting all the data once.

$$\frac{1}{n} \sum_{i=1}^{n} \big[y_i - \hat f^{-i}(x_i)\big]^2 = \frac{1}{n} \sum_{i=1}^{n} \left[\frac{y_i - \hat f(x_i)}{1 - S_{ii}}\right]^2$$

where $\hat f(x_i)$ is the fitted value from the model estimated on all $n$ observations, $S$ is the matrix that turns the observed outcomes into fitted values, and $S_{ii}$ is its $i$th diagonal element, which measures how strongly observation $i$ pulls its own fitted value towards itself. Hastie and colleagues give this identity for many linear fitting methods, including ordinary least squares linear regression. A patient with an unusual predictor profile has a large $S_{ii}$, so its ordinary residual can look small while its leave-one-out residual is much larger.

Generalised cross-validation (GCV) replaces each $S_{ii}$ with the average, $\mathrm{trace}(S)/n$, where the trace is the effective number of parameters, and Hastie and colleagues present it as a convenient approximation to leave-one-out cross-validation for linear fitting.

Low bias, high variance and the choice against K-fold

Each training set contains $n-1$ observations, almost the full sample. Hastie and colleagues state that the estimator is therefore approximately unbiased for the true expected prediction error, but that it can have high variance because the $n$ training sets are so similar to one another. With five or ten folds the training sets are smaller, so the estimate can overstate error when performance is still improving steeply with sample size, but its variance is lower, and they recommend five- or tenfold cross-validation as a good compromise.

Kohavi describes leave-one-out as almost unbiased but with high variance, leading to unreliable estimates, and his experiments on real datasets favoured stratified tenfold cross-validation for model selection. He also cites results showing that, for linear models, leave-one-out used for model selection is asymptotically inconsistent, because the probability of choosing the model with the best predictive power does not tend to one as the sample grows.

The method therefore suits small development samples, such as a cost or utility model built on one trial, where withholding a fifth of the data would weaken each fit, more than large administrative datasets.

Worked example with a small cost regression

The data in this example are illustrative. A health economist predicts annual hospital cost in GBP thousand from the number of long-term conditions for six patients, using ordinary least squares. The numbers of conditions are 0, 1, 2, 3, 4 and 8, and the costs are 1.0, 2.2, 2.4, 3.6, 3.8 and 6.0.

1. Fit the model to all six patients. The mean number of conditions is 18 / 6 = 3 and the mean cost is 19 / 6 = 3.1667. The sum of squared deviations of conditions is 9 + 4 + 1 + 0 + 1 + 25 = 40 and the sum of cross-products is 24, so the slope is 24 / 40 = 0.6 and the intercept is 3.1667 − 0.6 × 3 = 1.3667. Each extra condition adds GBP 600 to predicted cost.

2. Residuals and apparent error. The fitted values are 1.367, 1.967, 2.567, 3.167, 3.767 and 6.167, giving residuals of −0.367, 0.233, −0.167, 0.433, 0.033 and −0.167. The squared residuals sum to 0.4333, so the apparent mean squared error is 0.4333 / 6 = 0.0722, a root mean squared error of about GBP 269.

3. Diagonal elements. For a straight line, $S_{ii}$ is one sixth plus the squared deviation of the patient's conditions from 3, divided by 40. The six values are 0.392, 0.267, 0.192, 0.167, 0.192 and 0.792, summing to 2, the number of fitted parameters. For the patient with 8 conditions the value is 0.1667 + 25 / 40 = 0.7917.

4. Leave-one-out residuals. Dividing each residual by one minus its diagonal element gives −0.603, 0.318, −0.206, 0.520, 0.041 and −0.800. For the patient with 8 conditions, −0.1667 / 0.2083 = −0.800. Refitting without that patient confirms it, since the other five give an intercept of 1.2 and a slope of 0.7, a prediction of 1.2 + 0.7 × 8 = 6.8 and a held-out residual of 6.0 − 6.8 = −0.8.

5. Leave-one-out error. The squared leave-one-out residuals are 0.363, 0.101, 0.043, 0.270, 0.002 and 0.640, summing to 1.419, so the leave-one-out mean squared error is 1.419 / 6 = 0.2365, a root mean squared error of about GBP 486. GCV, using the average diagonal of 2 / 6, gives 0.0722 / 0.4444 = 0.1625, a root mean squared error of about GBP 403.

6. Interpretation. Held-out error is more than three times the apparent error on the squared scale, and the patient with 8 conditions supplies 45% of it (0.640 out of 1.419). A model that predicts each patient's cost as the mean of the other five has a leave-one-out mean squared error of 3.56, so the regression still predicts far better than the mean. With six patients both figures are very uncertain, and a decision model would need an equation estimated on a far larger sample.

Bayesian leave-one-out and LOOIC for survival models

In a Bayesian analysis the natural loss is the log predictive density of the left-out observation. Vehtari, Gelman and Gabry define the leave-one-out estimate of out-of-sample predictive fit as a sum over observations.

$$\widehat{\mathrm{elpd}}_{\mathrm{loo}} = \sum_{i=1}^{n} \log p(y_i \mid y_{-i})$$

where $\widehat{\mathrm{elpd}}_{\mathrm{loo}}$ is the estimated expected log pointwise predictive density, $y_i$ is observation $i$, $y_{-i}$ is the data without observation $i$, and $p(y_i \mid y_{-i})$ is the posterior predictive density of $y_i$ given the remaining data. Multiplying by minus 2 puts the result on the deviance scale, and this quantity is reported as LOOIC, where lower values indicate better predictive ability.

Exact leave-one-out would mean refitting a Markov chain Monte Carlo model $n$ times, which is slow. The authors instead reweight the existing posterior draws by Pareto smoothed importance sampling (PSIS-LOO). The fitted Pareto shape $\hat k$ is a reliability check for each observation, and when $\hat k$ exceeds 0.7 they advise sampling directly for that observation, switching to K-fold cross-validation or revising the model.

Jackson's survextrap package for survival extrapolation in health technology assessment uses this method, with each event or censoring time as a left-out observation. In his cetuximab case study, three models of the trial data alone had LOOIC values of 1156, 1157 and 1160 and almost identical five-year restricted mean survival time. Over 20 years the 95% credible interval for the gain in mean survival ran from −0.22 to 2.61 years under proportional hazards but from −1.24 to 4.12 years with the arms modelled separately. Leave-one-out fit within the observed follow-up says little about which extrapolation is right, because no left-out observation lies beyond the end of the data.

Leaving out one trial in a meta-analysis

In evidence synthesis the unit left out is a whole study, and the NICE Decision Support Unit's Technical Support Document 3 calls cross-validation based on a leave one out approach the appropriate tool for examining single trials in a meta-analysis. The trial is removed, a random-effects model is fitted to the rest, and the observed result is compared with the predictive distribution for a new trial of the same size and baseline risk, giving a Bayesian p-value.

TSD 3 applies this to the ISIS-4 trial in a meta-analysis of 16 trials of intravenous magnesium against placebo for acute myocardial infarction and obtains a p-value of 0.056. Because the most extreme of 16 trials was chosen, it compares that value with the expected smallest p-value of 1 / (16 + 1) = 0.059 and concludes that ISIS-4 is not necessarily incompatible with a model fitted to the remaining trials. In a network meta-analysis example the same check gave 0.004 for one trial. The document stresses that the result is statistical only and cannot show whether the left-out trial or the others are deviant. In frequentist work, the metafor package for R provides leave-one-out diagnostics for meta-analysis models without moderators.

Neighbouring methods that also delete one unit

Several methods delete one unit but answer different questions. The jackknife recomputes an estimate, such as a mean cost or a ratio, with each observation removed in order to estimate its bias and standard error, while leave-one-out cross-validation predicts the removed observation in order to estimate prediction error. Cook's distance measures how far the fitted coefficients move when one observation is deleted, so it flags influential patients without scoring predictions. Bootstrapping instead resamples with replacement, so each resample keeps the full sample size.

The unit deleted decides the question answered. Leaving out one record from a patient with several lets the model learn that patient from the other records, so repeated measures call for leaving out whole patients, and transport to new centres calls for leaving out whole centres, as discussed under model validation.

Where leave-one-out estimates mislead

The low bias comes with instability that the estimate itself can hide. Kohavi gives the example of a classifier that always predicts the majority class on a dataset with three equally common classes. Removing any observation makes its own class the minority, so every prediction is wrong and the leave-one-out accuracy is 0%, against an expected 33%, with a fold standard deviation of 0% that gives false assurance.

Choosing among many candidate models by minimum leave-one-out error and then reporting that minimum overstates performance, as for any cross-validated selection. Approximate Bayesian versions are only as good as their $\hat k$ diagnostics. A good leave-one-out score shows that a procedure predicts well for units like those in the data, and it cannot test extrapolation beyond follow-up, the causal effect of a treatment or performance in another health system, which need external data and attention to overfitting.

Sources

• Dias S, Sutton AJ, Welton NJ, Ades AE. NICE DSU Technical Support Document 3: Heterogeneity: subgroups, meta-regression, bias and bias-adjustment. Decision Support Unit; September 2011, last updated April 2012. https://www.sheffield.ac.uk/sites/default/files/2022-02/TSD3-Heterogeneity.final-report.08.05.12.pdf
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. 2nd ed. New York: Springer; 2009. Section 7.10 and bibliographic notes to chapter 7. https://hastie.su.domains/ElemStatLearn/
• Jackson CH. survextrap: a package for flexible and transparent survival extrapolation. BMC Medical Research Methodology. 2023;23:282. https://doi.org/10.1186/s12874-023-02094-1
• Jones AM. Models for health care. HEDG Working Paper 10/01. York: University of York; January 2010. https://www.york.ac.uk/media/economics/documents/herc/wp/10_01.pdf
• Kohavi R. A study of cross-validation and bootstrap for accuracy estimation and model selection. In: Proceedings of the 14th International Joint Conference on Artificial Intelligence. 1995;2:1137-1143. https://www.ijcai.org/Proceedings/95-2/Papers/016.pdf
• Vehtari A, Gelman A, Gabry J. Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 2017;27:1413-1432. https://doi.org/10.1007/s11222-016-9696-4
• Viechtbauer W. rma.uni, meta-analysis via linear (mixed-effects) models. metafor package documentation, version 5.3-1, accessed 26 October 2026. https://wviechtb.github.io/metafor/reference/rma.uni.html