Overfitting

Overfitting in Health Economic Prediction Models

Overfitting happens when a statistical model learns the chance quirks of the sample used to build it along with the real pattern, so its accuracy on that sample overstates how well it will predict for other patients. In health economics the models at risk are the regressions and risk equations that feed decision models. An overfitted equation gives predictions that are too extreme, which widens apparent differences between patient groups and distorts the costs and QALYs a cost-effectiveness model attaches to them. The usual defences are enough data for the parameters considered and shrinkage of the estimated effects, checked by internal validation.

Why apparent fit flatters a model

Every added parameter lets a model follow its development data more closely. Hastie, Tibshirani and Friedman show that training error keeps falling as complexity rises, typically reaching zero, and that a model with zero training error is overfit and will typically predict poorly. The gap between training error and in-sample error, the error for new outcomes at the same predictor values, is what they call optimism. The split of prediction error into bias, variance and noise is on the bias-variance tradeoff page.

For a linear fit under squared error loss, their textbook gives the expected optimism directly.

$$E_y(\text{Err}_{\text{in}}) = E_y(\overline{\text{err}}) + 2 \cdot \frac{d}{N} \, \sigma_\varepsilon^2$$

where $\text{Err}_{\text{in}}$ is the in-sample prediction error, $\overline{\text{err}}$ is the training error, $d$ is the number of inputs or basis functions fitted, $N$ is the number of observations, $\sigma_\varepsilon^2$ is the variance of the random error and $E_y$ is the expectation over the outcome values with the predictors held fixed.

Optimism rises with the number of parameters and falls as the sample grows. For an illustrative cost regression on 400 patients, 9 parameters give an expected optimism of 2 × 9 / 400 = 0.045 times the error variance, while 86 parameters give 2 × 86 / 400 = 0.43 times, nearly ten times as much. Babyak describes overfitting as asking too much of the available data, with complexity measured by the degrees of freedom spent at any stage of the analysis, counting interaction and non-linear terms.

Where overfitting enters health economic analysis

Overfitting reaches cost-effectiveness results through regression inputs. A risk score or logistic regression can supply the transition probability for each subgroup in a Markov model, and a multiple regression of patient costs can supply the cost of each state.

A mapping function that predicts EQ-5D utility from a clinical score is a regression of the same kind. The NICE Decision Support Unit's Technical Support Document 22 (TSD 22, June 2023, updated September 2026) notes that reserving data for out-of-sample checks reduces the sample for estimation and increases parameter uncertainty, and that a purely random split has very little power to detect systematic errors caused by misspecification. The ISPOR good practices task force on mapping judged it premature to recommend empirical validation for every mapping study, partly because many studies have one dataset too small to split.

For survival extrapolation, NICE DSU Technical Support Document 21 (TSD 21) notes that flexible parametric models built on restricted cubic splines generally fit extremely well within the observed data when enough knots are used, without this making their extrapolation reliable, and that many spline parameters may be impractical when events are few.

Parameters spent against the information available

The risk depends less on the number of patients than on the information the outcome carries, which for a binary or time-to-event outcome sits mainly in the number of events. In a simulation based on a cardiac trial of 673 patients with 252 deaths, Peduzzi and colleagues found no major problems with 10 or more events per variable, while below 10 the coefficients were biased in both directions, confidence intervals lost their proper coverage and paradoxical associations became more common.

The resulting rule of 10 events per variable is now treated with caution. Van Smeden and colleagues concluded that the evidence supporting such rules for logistic regression is weak, and that the problems also depend on factors such as total sample size. Riley and colleagues note that the rule has generated much debate, that Harrell generally recommends at least 15, that some situations need 20 or up to 50, and that any blanket rule is too simplistic. They count parameters, so a three-category predictor or a non-linear continuous term uses two or more, and they count every candidate considered before any selection.

Measuring overfitting with a shrinkage factor

A global shrinkage factor summarises overfitting in one number. Riley and colleagues describe multiplying every maximum likelihood coefficient by a factor $S$, estimated by bootstrapping or a closed form, and re-estimating the intercept so that the average predicted risk still matches the observed risk. The closed form is the heuristic shrinkage factor of Van Houwelingen and Le Cessie.

$$S_{VH} = 1 - \frac{p}{LR} = 1 + \frac{p}{n \ln\left(1 - R^2_{CS,app}\right)}$$

where $S_{VH}$ is the expected shrinkage factor, $p$ is the number of candidate predictor parameters, $LR$ is the likelihood ratio chi-squared statistic against a model with no predictors, $n$ is the number of patients and $R^2_{CS,app}$ is the apparent Cox-Snell $R^2$, which equals one minus $\exp(-LR/n)$.

A value of 1 means no expected overfitting. Riley and colleagues propose that a new model for a binary or time-to-event outcome should target a shrinkage factor of at least 0.9 and a difference of no more than 0.05 between apparent and adjusted Nagelkerke $R^2$, alongside precise estimation of overall risk. Rearranging gives the minimum sample size for the shrinkage criterion.

$$n = \frac{p}{(S_{VH} - 1) \ln\left(1 - R^2_{CS,adj} / S_{VH}\right)}$$

where $R^2_{CS,adj}$ is the anticipated Cox-Snell $R^2$ corrected for optimism, approximately $S_{VH}$ times the apparent value.

Penalised regression builds shrinkage into estimation. Hastie, Tibshirani and Friedman define the lasso as the solution of a penalised least squares problem.

$$\hat{\beta}^{\text{lasso}} = \arg\min_{\beta} \left\{ \frac{1}{2} \sum_{i=1}^{N} \Big( y_i - \beta_0 - \sum_{j=1}^{p} x_{ij}\beta_j \Big)^2 + \lambda \sum_{j=1}^{p} |\beta_j| \right\}$$

where $y_i$ is the outcome for patient $i$, $x_{ij}$ is the value of predictor $j$, $\beta_0$ is the intercept, $\beta_j$ are the coefficients and $\lambda$ is a complexity parameter that sets how strongly they are pulled towards zero. Ridge regression replaces $|\beta_j|$ with $\beta_j^2$. The lasso can set some coefficients exactly to zero, and the penalty is chosen to minimise an estimate of expected prediction error, typically by cross-validation.

Detecting overfitting through internal validation

Internal validation estimates how much apparent performance will survive in new patients from the same population. Steyerberg and colleagues compared methods on a logistic model with eight predictors of 30-day mortality after myocardial infarction, using samples from the GUSTO-I trial. Split-sample analyses gave overly pessimistic estimates with large variability, cross-validation on 10% of the sample had low bias and low variability but did not suit every performance measure, and bootstrapping gave stable estimates with low bias, so they recommended the bootstrap. Fold mechanics are on the k-fold cross-validation page, and wider checks under model validation.

Worked example with a hospital admission risk equation

All numbers in this example are illustrative. A logistic equation for the probability of an emergency admission within a year is to set transition probabilities for risk subgroups in a cost-effectiveness model. The data hold 1,200 patients, 180 of them admitted (15%), and 30 candidate predictor parameters.

1. Events per parameter. 180 events divided by 30 parameters gives 6, below even the contested rule of 10.

2. Expected shrinkage. The fitted model has a likelihood ratio statistic of 150, so $S_{VH}$ is 1 − 30/150 = 0.80. Since 150/1,200 is 0.125 and exp(−0.125) is 0.8825, the apparent Cox-Snell $R^2$ is 1 − 0.8825 = 0.1175. The coefficients need shrinking by about 20%, and the adjusted $R^2$ is 0.80 × 0.1175 = 0.094.

3. Data needed for the 0.9 target. The ratio 0.094/0.9 is 0.1044 and ln(1 − 0.1044) is −0.1103. The denominator is the product of −0.1 and −0.1103, which is 0.01103, so the minimum is 30 divided by 0.01103, or about 2,720 patients, with 408 events at 15% and 13.6 events per parameter. With 1,200 patients the criterion allows about 1,200 × 0.01103 = 13 parameters.

4. Effect on predictions. For simplicity the intercept is held at the log odds of the average risk, ln(0.15/0.85) = −1.735. A high-risk patient with a predictor score 2.0 above average has a risk of 1/(1 + exp(−0.265)) = 56.6% from the unshrunk equation. With coefficients multiplied by 0.80 the score is 1.6 and the risk is 1/(1 + exp(0.135)) = 46.6%. A low-risk patient 1.0 below average moves from 6.1% to 7.3% (score −0.8).

5. Effect on costs. At an illustrative GBP 3,500 per admission, expected admission costs for the high-risk patient are GBP 1,981 unshrunk and GBP 1,631 shrunk, and for the low-risk patient GBP 214 and GBP 256. The gap between them is GBP 1,767 unshrunk and GBP 1,375 shrunk, so the overfitted equation makes it about 29% larger.

6. Interpretation. An intervention targeted at high-risk patients would look better value under the unshrunk equation, because the risk it prevents is exaggerated. The analyst can cut the prespecified candidates to about 13 parameters before fitting, find more data or apply shrinkage, and report the internal validation either way.

Boundaries with nearby ideas

Underfitting, the opposite failure, leaves out real structure and predicts poorly even on the development data. The Akaike information criterion and the Bayesian information criterion estimate optimism and add it to the training error, and the choice between candidates is treated under model selection. Overfitting also differs from misspecification, where the model has the wrong form, and from poor transportability, where a well-fitted model meets a population with a different case mix.

Common misreadings

Most errors come from trusting performance measured on the data that built the model. In a decision model each one passes the optimism on to subgroup estimates.

1. Reading a high apparent R-squared as accuracy. Training error falls with every parameter added.
2. Counting only the predictors in the final model. Every candidate parameter counts, and Babyak shows that automated variable selection and pretesting of candidates carry a considerable risk of spurious findings.
3. Treating 10 events per variable as a safe harbour. The evidence behind the rule is weak, and the data needed depend on the anticipated $R^2$ and the outcome rate.
4. Assuming penalisation rescues a small sample. Riley and colleagues found penalised models more unreliable when overfitting may be large, because tuning parameters are then estimated with large uncertainty.
5. Equating good within-trial fit of a survival curve with sound extrapolation. TSD 21 says close fit inside follow-up does not make extrapolation reliable.

Sources

• Babyak MA. What you see may not be what you get: a brief, nontechnical introduction to overfitting in regression-type models. Psychosomatic Medicine. 2004;66(3):411-421. https://doi.org/10.1097/01.psy.0000127692.23278.a9
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. 2nd ed. New York: Springer; 2009. Sections 3.4 and 7.2 to 7.5. https://hastie.su.domains/ElemStatLearn/
• Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR. A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology. 1996;49(12):1373-1379. https://doi.org/10.1016/s0895-4356(96)00236-3
• Riley RD, Snell KIE, Ensor J, Burke DL, Harrell FE, Moons KGM, Collins GS. Minimum sample size for developing a multivariable prediction model: Part II, binary and time-to-event outcomes. Statistics in Medicine. 2019;38(7):1276-1296. https://pmc.ncbi.nlm.nih.gov/articles/PMC6519266/
• Riley RD, Snell KIE, Martin GP, Whittle R, Archer L, Sperrin M, Collins GS. Penalization and shrinkage methods produced unreliable clinical prediction models especially when sample size was small. Journal of Clinical Epidemiology. 2021;132:88-96. https://doi.org/10.1016/j.jclinepi.2020.12.005
• Rutherford MJ, Lambert PC, Sweeting MJ, Pennington B, Crowther MJ, Abrams KR, Latimer NR. NICE DSU Technical Support Document 21: Flexible methods for survival analysis. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2020. Section 3.2. https://doi.org/10.15131/shef.data.33849118
• Steyerberg EW, Harrell FE, Borsboom GJ, Eijkemans MJ, Vergouwe Y, Habbema JD. Internal validation of predictive models: efficiency of some procedures for logistic regression analysis. Journal of Clinical Epidemiology. 2001;54(8):774-781. https://doi.org/10.1016/s0895-4356(01)00341-9
• van Smeden M, de Groot JA, Moons KG, Collins GS, Altman DG, Eijkemans MJ, Reitsma JB. No rationale for 1 variable per 10 events criterion for binary logistic regression analysis. BMC Medical Research Methodology. 2016;16:163. https://doi.org/10.1186/s12874-016-0267-3
• Wailoo A, Hernandez Alava M, Pudney S. NICE DSU Technical Support Document 22: Mapping to estimate health state utilities. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; June 2023, updated September 2026. Section 4.6. https://doi.org/10.15131/shef.data.33849340.v1
• Wailoo AJ, Hernandez-Alava M, Manca A, Mejia A, Ray J, Crawford B, Botteman M, Busschbach J. Mapping to estimate health-state utility from non-preference-based outcome measures: an ISPOR Good Practices for Outcomes Research Task Force report. Value in Health. 2017;20(1):18-27. https://doi.org/10.1016/j.jval.2016.11.006