Leave-One-Out

Concept

---

Theoretically, Leave-One-Out Cross-Validation (LOOCV) is a special case of K-Fold Cross-Validation in which the number of folds equals the number of observations in the dataset. Each observation is excluded once, the model is fitted using the remaining observations, and the excluded observation is used for validation. LOOCV exists to estimate the predictive performance of statistical and machine learning models while making maximal use of available data, particularly in studies with small sample sizes.

Mathematically, Leave-One-Out Cross-Validation estimates the expected prediction error by averaging the validation loss obtained from all individual leave-one-out iterations. Since each observation serves once as the validation sample and n ? 1 times as part of the training data, the resulting estimate is nearly unbiased for out-of-sample prediction error, although it may exhibit relatively high variance compared with other cross-validation methods.

In practice, LOOCV is implemented by repeatedly fitting the model after omitting a single observation, calculating the prediction error for that omitted observation and averaging the errors across all observations. In health economics, LOOCV is applied to validate predictive risk models, health outcome models, resource utilisation models and machine learning algorithms when datasets are relatively small and retaining as much training data as possible is desirable.

Purpose

---

Used to estimate out-of-sample predictive performance, compare competing statistical models, assess model generalisability and validate prediction models when sample sizes are limited.

---

Mathematical Formulae

Primary Formula

LOOCV = (1/n) ? ????� L?

where:

• LOOCV = average leave-one-out validation error
• n = number of observations
• L? = prediction loss for the omitted observation in iteration i

Supporting Formulae

Mean Squared Error (regression):

MSE = (1/n) ? ?(y? ? ??)�

Root Mean Squared Error:

RMSE = �MSE

Classification Error:

Error = Number of incorrect predictions / Number of observations

Related Mathematical Methods

• K-Fold Cross-Validation
• Repeated K-Fold Cross-Validation
• Bootstrap Validation
• Hold-Out Validation
• Cross-Validation
• Maximum Likelihood Estimation

---

Example

A health economist develops a regression model to predict annual healthcare costs using data from 120 patients.

Using Leave-One-Out Cross-Validation:

• 120 separate models are fitted.
• Each model excludes one patient.
• The excluded patient's cost is predicted.
• The squared prediction errors are averaged.

Suppose the average validation Mean Squared Error is:

LOOCV MSE = 18.6

This value provides an estimate of the model's expected prediction error when applied to new patients and can be compared with competing prediction models.

---

Excel Implementation

Function	Example Formula	Health Economics Application
ROW	=ROW(A2)-1	Identify the observation omitted during each iteration.
AVERAGE	=AVERAGE(B2:B121)	Calculate the mean validation error across all leave-one-out iterations.
SUMPRODUCT	=SUMPRODUCT((Actual-Predicted)^2)/COUNT(Actual)	Calculate Mean Squared Error for validation.
COUNT	=COUNT(A2:A121)	Determine the total number of leave-one-out iterations.
Solver	Optimise model parameters after excluding each observation.	Estimate model performance under leave-one-out validation.

---

VBA (Optional)

A VBA routine can automate repeated leave-one-out model fitting, calculate validation errors for every omitted observation and summarise overall predictive performance.

---

Sources

• Stone M. Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society Series B. 1974;36(2):111?147.
• Allen DM. The Relationship Between Variable Selection and Data Augmentation and a Method for Prediction. Technometrics. 1974;16(1):125?127.
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Springer.
• James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning. Springer.
• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford University Press.
• ISPOR Good Practice Reports.