Concept Architecture
Concept
Theoretically, Prediction Error is the difference between an observed outcome and the value predicted by a statistical or mathematical model. It quantifies the accuracy of model predictions and forms the basis of model evaluation, estimation, optimisation, and machine learning. Prediction error is fundamental to statistical inference because it measures the discrepancy between modelled and observed data, providing information about model fit and predictive performance.
Mathematically, prediction error is represented as the residual difference between an observed value and its corresponding prediction. Individual prediction errors are commonly aggregated into recognised summary measures, including Mean Absolute Error (MAE), Mean Squared Error (MSE), and Root Mean Squared Error (RMSE), which are used as objective functions during model estimation and validation.
In practice, prediction error is calculated by comparing observed and predicted outcomes from regression models, survival models, health economic simulations, and machine learning algorithms. In health economics it is used to evaluate prediction of costs, quality-adjusted life years, disease progression, healthcare utilisation, and other modelled outcomes during calibration, validation, and comparative model assessment.
Purpose
Used to quantify the discrepancy between observed and predicted outcomes, evaluate predictive accuracy, compare competing models, optimise model parameters, and assess model performance in health economic analyses.
Mathematical Formulae
Primary Formula
e? = y? ? ??
where:
- e? = prediction error
- y? = observed value
- ?? = predicted value
Supporting Formulae
MAE = (1/n) ? ?|y? ? ??|
MSE = (1/n) ? ?(y? ? ??)�
RMSE = �[(1/n) ? ?(y? ? ??)�]
Bias = (1/n) ? ?(y? ? ??)
Related Mathematical Methods
- Residual Analysis
- Mean Absolute Error
- Mean Squared Error
- Root Mean Squared Error
- Cross-Validation
- Regression Analysis
- Calibration Assessment
- Model Validation
Example
A health economic model predicts annual healthcare costs for five patients.
| Patient | Observed (�) | Predicted (�) | Prediction Error (�) |
|---|---|---|---|
| 1 | 8,200 | 8,000 | 200 |
| 2 | 10,100 | 10,400 | ?300 |
| 3 | 7,500 | 7,600 | ?100 |
| 4 | 9,300 | 9,100 | 200 |
| 5 | 8,900 | 9,000 | ?100 |
The Mean Squared Error is:
MSE = (200� + (?300)� + (?100)� + 200� + (?100)�) � 5
= (40,000 + 90,000 + 10,000 + 40,000 + 10,000) � 5
= 38,000
RMSE = �38,000 = �194.94
This indicates that model predictions differ from observed costs by approximately �195 on average in root mean squared terms.
Excel Implementation
| Function | Example Formula | Health Economics Application |
|---|---|---|
| ABS | =ABS(B2-C2) | Calculate the absolute prediction error for each observation. |
| POWER | =POWER(B2-C2,2) | Calculate squared prediction errors. |
| AVERAGE | =AVERAGE(D2:D101) | Calculate the mean prediction error or MSE. |
| SQRT | =SQRT(AVERAGE(E2:E101)) | Calculate RMSE for model validation. |
| SUMXMY2 | =SUMXMY2(B2:B101,C2:C101) | Calculate the sum of squared prediction errors. |
VBA (Optional)
Automate calculation of prediction error metrics across competing health economic models and generate validation summary reports.
Sources
- Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning.
- Bishop CM. Pattern Recognition and Machine Learning.
- James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning.
- Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation.
- Drummond MF, et al. Methods for the Economic Evaluation of Health Care Programmes.
- NICE Health Technology Evaluation Manual.
- ISPOR Good Practice Reports.
Related Concepts (3)
Library
Publications
1
Bayesian Methods in Health Economics — Gianluca Baio, 1st Edition ed., 2012 (Chapman & Hall / CRC Press)
An overview of Bayesian statistical methods for the analysis of health economic data, covering economic evaluation concepts, statistical cost-effectiveness analysis, Bayesian computation and MCMC, and applied health economic evaluation.
BookView source →
Frequently Asked Questions (6)
What is prediction error?
The difference between a value predicted by a statistical model and the value that is actually observed.
Source: Hastie, Tibshirani & Friedman 2009
What does prediction error measure about a model's output?
Prediction error is the gap between what a model predicts and what actually happens, the difference between a predicted value and the observed one. It measures how far the model's forecasts miss reality, and summarised across many cases it gauges the model's overall accuracy. To be meaningful it must be judged on data the model did not learn from, since error measured on the training data flatters the model by ignoring its tendency to fit noise. The miss between prediction and reality is what it captures. Steyerberg (2009) describes this.
Source: Steyerberg 2009
How is prediction error measured?
Prediction error is measured by comparing predicted with observed values across a set of cases and summarising the differences, using measures such as the mean squared error, which averages squared errors and penalises large ones, the mean absolute error, which averages absolute errors, or classification error rates for categorical outcomes. So prediction error is measured through summary metrics of the differences between predictions and outcomes, chosen according to the outcome type and the emphasis desired, and importantly it is assessed on data not used to fit the model, through a test set or cross-validation, since error on the training data understates the true prediction error on new cases.
Source: Hastie, Tibshirani & Friedman 2009
What are the components of prediction error?
Prediction error can be decomposed into components: bias, the systematic error from the model being too simple or misspecified; variance, the error from sensitivity to the particular training sample; and irreducible error, the inherent noise that no model can remove. So prediction error comprises bias, variance, and irreducible noise, and this decomposition underlies the bias-variance tradeoff, since reducing bias by adding flexibility tends to increase variance, meaning the total prediction error is minimised by balancing the two, which is why understanding the components of prediction error guides the choice of model complexity to achieve good predictive performance.
Source: Hastie, Tibshirani & Friedman 2009
Why must prediction error be assessed on new data?
Prediction error must be assessed on new data because error measured on the training data is optimistic, since the model has been fitted to those observations, including their noise, so it understates the error on unseen cases and can hide overfitting. So prediction error is assessed out-of-sample, through a held-out test set or cross-validation, to obtain an honest estimate of how the model will perform on new data, since in-sample error is biased downward, and only evaluation on data not used in fitting reveals the true predictive accuracy and detects overfitting, which is why out-of-sample assessment is standard for prediction error.
Source: Hastie, Tibshirani & Friedman 2009
How is prediction error used?
Prediction error is used to evaluate how well a model predicts, to compare competing models by their errors, and to select model complexity or settings that minimise error on new data, guiding model development. So prediction error is used as the criterion for assessing and choosing predictive models, since the aim of prediction is to minimise error on future cases, and comparing out-of-sample prediction error across models or configurations identifies those that generalise best, which is why estimating prediction error reliably, through cross-validation or a test set, is central to building and selecting models that perform well in practice.
Source: Hastie, Tibshirani & Friedman 2009
Trust Record
Verified by Dr Darrin Baines
British health economist
Professional identity: darrinbaines.org
Verification date: 23 Dec 2025
Content version: 1.0.0
Canonical Identity
- Persistent URI
- https://healtheconomics.wiki/concept/prediction-error
- Term code
- HE-ES-SA-162
Stable URI · Machine-readable · Resolvable · CC BY 4.0