Cross-Validation

Concept

Theoretically, Cross-Validation is a statistical resampling technique used to evaluate the predictive performance and generalisability of a model by repeatedly partitioning data into training and validation subsets. It estimates how accurately a model is expected to perform on independent data, thereby reducing the risk of overfitting and supporting model selection. In health economics, cross-validation is commonly applied during the development of prediction models, mapping algorithms, risk equations and surrogate models incorporated into economic evaluations.

Mathematically, cross-validation is based on repeated estimation of a prediction error function across multiple data partitions. In k-fold cross-validation, the dataset is divided into k approximately equal subsets. Each subset serves once as the validation set while the remaining k ? 1 subsets form the training data. The overall cross-validation error is calculated as the average prediction error across all folds.

In practice, cross-validation is implemented by selecting an appropriate resampling strategy, fitting the model repeatedly and calculating predictive performance measures such as mean squared error, root mean squared error, mean absolute error or classification accuracy. In health economics, it is frequently used to validate utility mapping algorithms, disease risk prediction models and statistical models informing decision-analytic model parameters.

---

Purpose

Used to assess the predictive performance and generalisability of statistical models, compare competing model specifications and reduce overfitting before models are applied within health economic evaluations.

---

Mathematical Formulae

Primary Formula

For k-fold cross-validation,

CV = (1/k) �???? L?

where:

• k = number of folds
• L? = prediction loss calculated on validation fold i

Supporting Formulae

Mean Squared Error (common loss function):

MSE = (1/n) �???� (y? ? ??)�

Root Mean Squared Error:

RMSE = �MSE

Mean Absolute Error:

MAE = (1/n) �???� |y? ? ??|

Related Mathematical Methods

• k-fold cross-validation
• Leave-one-out cross-validation (LOOCV)
• Repeated k-fold cross-validation
• Bootstrap validation
• Prediction error estimation
• Model selection

---

Example

A health economist develops a mapping algorithm to predict EQ-5D utility values from a disease-specific quality-of-life questionnaire using data from 500 patients.

Ten-fold cross-validation is performed.

Validation fold mean squared errors are:

0.022, 0.020, 0.024, 0.021, 0.023, 0.019, 0.021, 0.022, 0.020 and 0.023.

The cross-validation error is

CV = (0.022 + 0.020 + ? + 0.023) / 10 = 0.0215

The low average prediction error indicates good expected performance on new patient data.

---

Excel Implementation

Function	Example Formula	Health Economics Application
AVERAGE	=AVERAGE(B2:B11)	Calculates the average validation error across folds.
SUM	=SUM(B2:B11)	Sums prediction errors before averaging.
COUNT	=COUNT(B2:B11)	Determines the number of validation folds.
SQRT	=SQRT(B12)	Calculates RMSE from MSE.
ABS	=ABS(C2-D2)	Computes absolute prediction error for MAE calculations.

---

VBA (Optional)

Automate repeated model fitting across validation folds and produce a summary report of cross-validation performance measures.

---

Sources

• Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed.
• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation.
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning.
• James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning.
• ISPOR Good Practice Reports on statistical modelling and model validation.