K-Fold Cross-Validation

Concept

---

Theoretically, K-Fold Cross-Validation is a resampling method used to evaluate the predictive performance and generalisability of statistical and machine learning models. The method partitions a dataset into k mutually exclusive subsets (folds), iteratively using one fold for validation and the remaining k ? 1 folds for model training. It exists to provide an unbiased estimate of out-of-sample predictive performance while making efficient use of available data, particularly when sample sizes are limited.

Mathematically, K-Fold Cross-Validation is represented by repeated estimation of a predefined loss function across k validation folds, followed by calculation of the average validation error. The framework estimates the expected prediction error of a modelling procedure rather than the parameters of the model itself. Common loss functions include mean squared error for regression and classification error, log-loss or area under the receiver operating characteristic curve for classification models.

In practice, the dataset is randomly divided into k approximately equal folds, with the model refitted k times. Each observation serves as validation data exactly once and training data k ? 1 times. Performance metrics are averaged across all folds to assess model robustness, compare alternative models and tune hyperparameters. In health economics, K-Fold Cross-Validation is widely applied in predictive risk models, health outcome prediction, resource utilisation forecasting, disease progression modelling and machine learning applications supporting economic evaluation.

Purpose

---

Used to estimate out-of-sample predictive performance, compare competing models, optimise model complexity, reduce overfitting and assess the generalisability of statistical and machine learning models used in health economics.

---

Mathematical Formulae

Primary Formula

CV = (1/k) ? ????? L?

where:

• CV = average cross-validation error
• k = number of folds
• L? = validation loss for fold i

Supporting Formulae

Mean Squared Error (regression):

MSE = (1/n) ? ?(y? ? ??)�

Classification Error:

Error = Number of incorrect predictions / Number of observations

Root Mean Squared Error:

RMSE = �MSE

Related Mathematical Methods

• Leave-One-Out Cross-Validation (LOOCV)
• Repeated K-Fold Cross-Validation
• Stratified K-Fold Cross-Validation
• Nested Cross-Validation
• Bootstrap Validation
• Hold-Out Validation

---

Example

A health economist develops a machine learning model to predict annual hospital expenditure using data from 1,000 patients.

Ten-fold cross-validation is selected (k = 10).

Validation Mean Squared Errors obtained from each fold are:

15.1, 14.7, 15.6, 14.9, 15.2, 14.8, 15.0, 15.3, 14.6 and 15.1.

The overall cross-validation error is:

CV = (15.1 + 14.7 + 15.6 + 14.9 + 15.2 + 14.8 + 15.0 + 15.3 + 14.6 + 15.1) � 10

CV = 15.03

The average validation error indicates the expected predictive performance of the model on unseen healthcare data and can be compared with competing models during model selection.

---

Excel Implementation

Function	Example Formula	Health Economics Application
RAND	=RAND()	Randomly assign observations to folds.
RANK	=RANK(A2,$A$2:$A$1001)	Randomise observation ordering before fold assignment.
MOD	=MOD(B2,10)+1	Allocate observations to one of k folds.
AVERAGE	=AVERAGE(C2:C11)	Calculate the mean validation error across folds.
SUMPRODUCT	=SUMPRODUCT((Actual-Predicted)^2)/COUNT(Actual)	Calculate validation Mean Squared Error for each fold.

---

VBA (Optional)

A VBA routine can automate fold generation, iterative model evaluation and calculation of average cross-validation performance across multiple candidate models.

---

Sources

• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Springer.
• James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning. Springer.
• Kuhn M, Johnson K. Applied Predictive Modelling. Springer.
• Harrell FE. Regression Modelling Strategies. Springer.
• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford University Press.
• ISPOR Good Practice Reports.