Overfitting

Concept

---

Theoretically, Overfitting is a modelling phenomenon in which a statistical or machine learning model captures random variation and noise in the training data rather than the underlying data-generating process. An overfitted model achieves excellent performance on the data used for estimation but performs poorly when applied to new observations because it has insufficient generalisability. Overfitting exists as a consequence of excessive model complexity relative to the amount of available information and is fundamentally linked to the bias-variance trade-off.

Mathematically, overfitting is characterised by a reduction in training error accompanied by an increase in prediction error on independent data. It is commonly assessed by comparing model performance using validation datasets or cross-validation. Excessive model flexibility increases estimator variance, causing the expected prediction error to rise even though the apparent fit to the training data continues to improve.

In practice, overfitting is identified by evaluating predictive performance on independent validation datasets, using cross-validation, bootstrap validation or information criteria. In health economics, overfitting may occur when developing prediction models for healthcare costs, quality-adjusted life years, disease progression or resource utilisation using too many predictors or overly complex algorithms relative to the sample size. Prevention strategies include penalised regression, model simplification, regularisation and external validation.

Purpose

---

Used to assess whether statistical or machine learning models generalise beyond the training data, identify excessive model complexity and guide the development of robust predictive models in health economic research.

---

Mathematical Formulae

Primary Formula

Expected prediction error:

EPE = Bias� + Variance + ��

where:

• Bias� = squared systematic error
• Variance = variability of the fitted model
• �� = irreducible error

Supporting Formulae

Mean Squared Error:

MSE = (1/n) ? ?(y? ? ??)�

Cross-validation error:

CV = (1/k) ? ????? L?

Akaike Information Criterion:

AIC = ?2ln(L) + 2p

Bayesian Information Criterion:

BIC = ?2ln(L) + p ln(n)

Related Mathematical Methods

• Cross-Validation
• Regularisation
• Ridge Regression
• Lasso Regression
• Bias-Variance Trade-off
• Model Validation
• Bootstrap Validation

---

Example

A health economist develops two models to predict annual healthcare costs using 5,000 patient records.

Model A contains eight clinically justified predictors.

Model B contains 85 predictors, including numerous interaction terms.

Performance is assessed using ten-fold cross-validation.

Model	Training MSE	Validation MSE
A	1.22	1.31
B	0.76	2.48

Although Model B fits the training data substantially better, its much larger validation error indicates overfitting. Model A is therefore preferred because it is expected to perform more reliably when predicting healthcare costs for new patients.

---

Excel Implementation

Function	Example Formula	Health Economics Application
AVERAGE	=AVERAGE(B2:B11)	Calculate the mean cross-validation error across folds.
POWER	=POWER(B2-C2,2)	Calculate squared prediction errors.
SUMPRODUCT	=SUMPRODUCT((Actual-Predicted)^2)/COUNT(Actual)	Calculate Mean Squared Error for model validation.
MIN	=MIN(B2:B20)	Identify the model with the lowest validation error.
Solver	Optimise model complexity while minimising validation error.	Compare candidate prediction models and reduce overfitting.

---

VBA (Optional)

A VBA routine can automate cross-validation, compare training and validation errors across candidate models and identify models exhibiting evidence of overfitting.

---

Sources

• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. Springer.
• James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning. Springer.
• Bishop CM. Pattern Recognition and Machine Learning. Springer.
• Harrell FE. Regression Modeling Strategies. Springer.
• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford University Press.
• ISPOR Good Practice Reports.