Predictive Accuracy

Concept

Theoretically, Predictive Accuracy is the degree to which a statistical or mathematical model correctly predicts outcomes in new or independent data. It reflects the closeness of predicted values to observed values and provides an assessment of a model's ability to generalise beyond the data used for model development. Predictive accuracy is central to statistical learning and prediction because a model with high explanatory performance may nevertheless perform poorly when applied to external populations. In health economics, predictive accuracy is evaluated when developing risk equations, survival models, utility mapping algorithms and other statistical models used to parameterise economic evaluations.

Mathematically, predictive accuracy is quantified using measures that compare observed outcomes with model predictions. The appropriate measure depends on the outcome type and modelling framework and may include mean squared error, root mean squared error, mean absolute error, classification accuracy, the Brier score or other prediction error statistics. These measures estimate the magnitude of prediction error and facilitate comparison between competing predictive models.

In practice, predictive accuracy is assessed using independent validation datasets, cross-validation or bootstrap resampling to minimise optimism arising from model development. Health economists evaluate predictive accuracy before selecting statistical models used to estimate costs, utilities, treatment effects or disease progression. Assessment is typically combined with calibration and discrimination analyses to provide a comprehensive evaluation of model performance.

---

Purpose

Used to quantify how closely model predictions match observed outcomes and to compare competing predictive models before their use in health economic evaluation.

---

Mathematical Formulae

Primary Formula

For continuous outcomes, a widely recognised measure of predictive accuracy is the mean squared error:

MSE = (1/n) �???� (y? ? ??)�

where:

• y? = observed outcome
• ?? = predicted outcome
• n = number of observations

Supporting Formulae

Root Mean Squared Error:

RMSE = �MSE

Mean Absolute Error:

MAE = (1/n) �???� |y? ? ??|

Related Mathematical Methods

• Cross-validation
• Bootstrap validation
• Calibration assessment
• Discrimination assessment
• Goodness of fit assessment
• Mean squared error
• Root mean squared error
• Mean absolute error
• Brier score

---

Example

A health economist develops a regression model predicting annual healthcare costs for 1,000 patients.

Observed costs (�):

1,950; 2,100; 2,300; 2,450; 2,700

Predicted costs (�):

2,000; 2,050; 2,250; 2,500; 2,650

The mean squared error is

MSE = [(-50)� + 50� + 50� + (-50)� + 50�] / 5 = 2,500

Therefore,

RMSE = �2,500 = 50

The average prediction error is �50, indicating good predictive accuracy relative to the scale of annual healthcare costs.

---

Excel Implementation

Function	Example Formula	Health Economics Application
SUMXMY2	=SUMXMY2(B2:B1001,C2:C1001)/COUNT(B2:B1001)	Calculates mean squared prediction error.
SQRT	=SQRT(D2)	Calculates root mean squared error.
ABS	=ABS(B2-C2)	Calculates absolute prediction error for each observation.
AVERAGE	=AVERAGE(D2:D1001)	Calculates mean absolute prediction error.

---

VBA (Optional)

Automate calculation of prediction error measures across validation datasets and generate comparative predictive performance reports.

---

Sources

• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation.
• Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed.
• Harrell FE. Regression Modeling Strategies. 2nd ed.
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning.
• James G, Witten D, Hastie T, Tibshirani R. An Introduction to Statistical Learning.
• ISPOR-SMDM Modeling Good Research Practices Task Force. Model Validation. Medical Decision Making. 2012.