Cross-Validation

Cross-Validation for Prediction Models in Health Economics

Cross-validation tests a prediction model on observations that were left out when the model was fitted, then rotates which observations are left out so that every observation is tested once. The held-out error is a more honest guide to performance on new patients than the error on the fitting data, which is usually too flattering. Health economists use it to choose between cost regressions, to build risk adjustment formulas, to compare survival models before extrapolation and to report the internal validation of risk equations that feed decision models. The ISPOR-SMDM modelling good practice report uses the same word for a different check, the comparison of one decision model's results with those of other models, so a reader meeting the term in a model validation report needs to know which sense is meant.

Why fit to the development data overstates performance

A model judged on the same data used to build it shows its apparent performance. Collins and colleagues, in a BMJ series on evaluating prediction models, describe overfitting as a model fitting unimportant idiosyncrasies in the development data to the point that it performs poorly in new data. Varma and Simon note that testing a classifier on the samples used to train it gives a re-substitution estimate that is falsely low, usually zero, in small samples.

The problem is familiar in health econometrics. Mihaylova and colleagues, reviewing statistical methods for healthcare resources and costs, warn that more complex approaches to mean costs could run a risk of misfitting or overfitting the data. They also report that two-part models beat single-equation models on split-sample mean squared forecast error, an out-of-sample comparison of the kind that cross-validation generalises.

How K-fold cross-validation estimates prediction error

Hastie, Tibshirani and Friedman call cross-validation probably the simplest and most widely used method for estimating prediction error. In K-fold cross-validation the data are split into $K$ roughly equal parts; for each part, the model is fitted to the other $K-1$ parts and its prediction error is computed on the part held out.

$$\mathrm{CV}(\hat f) = \frac{1}{N} \sum_{i=1}^{N} L\big(y_i, \hat f^{-\kappa(i)}(x_i)\big)$$

where $N$ is the number of observations, $y_i$ and $x_i$ are the outcome and predictors for observation $i$, $\kappa(i)$ is the fold to which observation $i$ belongs, $\hat f^{-k}$ is the model fitted with fold $k$ removed and $L$ is the loss function, such as absolute error, squared error or the Brier score. Because every observation contributes one loss, each fold is weighted by its size.

$$\mathrm{CV} = \sum_{k=1}^{K} \frac{n_k}{N} \, \bar{L}_k$$

where $n_k$ is the number of observations in fold $k$ and $\bar{L}_k$ is the mean loss in that fold. A simple average of the fold means gives the same answer only when the folds are equal in size.

The textbook gives 5 or 10 as typical choices of $K$. Setting $K$ equal to $N$ gives leave-one-out cross-validation, which is approximately unbiased for the expected prediction error but can have high variance because the training sets are so similar. With five or ten folds each training set is smaller than the full sample, so the method overestimates prediction error if performance still improves steeply with sample size; the authors recommend five- or tenfold cross-validation as a good compromise. Collins and colleagues add that repeated K-fold cross-validation, averaged over several random splits, will generally improve on a single run.

What a cross-validated error describes

Cross-validation assesses a model-building procedure, not one set of fitted coefficients. Varma and Simon interpret the estimate as the error of the model that the training algorithm would produce if applied to the entire dataset, and the textbook refits the chosen model to all the data once tuning is finished.

Hastie and colleagues show that cross-validation estimates well only the expected prediction error averaged over possible training sets, not the error of the particular model fitted to the data in hand. Bates, Hastie and Tibshirani prove this for ordinary least squares regression and report that naive confidence intervals from cross-validation can have coverage far below the stated level. Fold errors are correlated because each observation is used for both training and testing, so the usual variance estimate is too small.

Choices that decide whether the estimate can be trusted

Analyst choices about the procedure can make the estimate badly optimistic. TRIPOD+AI item 12c asks authors to specify the type of model, all model-building steps including any hyperparameter tuning, and the method for internal validation.

Every learned step stays inside the fold

Hastie and colleagues describe a simulation with 50 samples and 5,000 predictors unrelated to the outcome, in which predictors were screened on all the data before cross-validation. The average cross-validation error was 3% against a true error rate of 50%, because the selected predictors had already seen the left-out samples. Cross-validation must therefore cover the entire sequence of modelling steps, with an exception only for unsupervised screening that ignores the outcome.

Tuning and model choice need their own protection

Choosing a tuning parameter or a model because it has the lowest cross-validated error, then reporting that minimum as the expected error, is optimistic. Varma and Simon found that the minimum was not an unbiased estimate of the true error of the optimised classifier and that the difference could exceed 20% more than one-fifth of the time. A nested procedure, with an inner loop for tuning and an outer loop for evaluation, gave an estimate very close to the true error.

The unit of splitting matches the intended use

Random splits ask how the model performs for new people like those in the sample. Where patients contribute repeated records, all of a patient's records belong in one fold, otherwise the model is credited for recognising people it has already seen. Steyerberg and Harrell describe internal-external cross-validation, in which each study or centre is left out in turn and the model built on the rest is tested on it, and Collins and colleagues use it to explore how performance varies across centres. Collins and colleagues advise against a single hold-out of a later period at the development stage, because it discards data and leaves either period unrepresentative.

Uses in health economic analysis

Cross-validation appears wherever an economic analysis rests on predictions rather than on estimated treatment effects alone. Each case uses held-out error to choose or report a model that feeds a cost or outcome estimate.

• Cost regression. Analysts choosing between ordinary least squares, a generalized linear model, two-part models and more flexible methods can compare out-of-sample error rather than in-sample fit.
• Risk adjustment for plan payment. Rose compared regression, penalised regression, decision trees, neural networks and an ensemble super learner for predicting total annual expenditure, using cross-validated R squared on 2011 and 2012 MarketScan data, and found that a simplified formula kept much of the efficiency of a traditional larger one.
• Survival models before extrapolation. Jackson's survextrap package compares flexible Bayesian survival models by leave-one-out cross-validation summarised as the LOOIC, where lower means better predictive ability. In his cetuximab example three models had LOOIC values of 1156, 1157 and 1160 on the trial data, yet differed more when extrapolating.
• Risk equations inside decision models. A risk equation that sets transition probabilities in a cost-effectiveness model needs reported internal validation, which TRIPOD+AI defines as evaluating a model in its development population, for example by cross-validation or bootstrapping.

The textbook notes that Stone showed in 1977 that the Akaike information criterion and leave-one-out cross-validation are asymptotically equivalent, which links information criteria to cross-validation in model selection.

Worked example comparing two cost models across five folds

The numbers in this example are illustrative. An analyst predicting annual hospital cost for 500 patients compares Model A, a generalised linear model with a few main effects, with Model B, which adds many interactions. Patients are grouped by general practice and practices are kept whole, giving five folds of unequal size and a mean absolute error in GBP per patient for each fold.

1. Weight each fold by its size.

$$\mathrm{MAE}_A = \frac{90 \times 1020 + 110 \times 980 + 100 \times 1050 + 95 \times 1000 + 105 \times 960}{500} = \frac{500400}{500} = 1000.8$$

where $\mathrm{MAE}_A$ is Model A's cross-validated mean absolute error in GBP per patient, the multipliers are the fold sizes and 500 is the total number of patients. The products are 91,800, 107,800, 105,000, 95,000 and 100,800. A simple average of the fold values gives 5,010 / 5 = 1,002, slightly higher because the two largest folds, 2 and 5, have the lowest errors.

2. Repeat for Model B. The products are 99,000, 116,600, 114,000, 102,600 and 107,100, which sum to 539,300, so the cross-validated MAE is 539,300 / 500 = GBP 1,078.60 per patient.

3. Compare with the apparent error. Suppose that, fitted and tested on all 500 patients, Model A shows an MAE of GBP 950 and Model B GBP 820. Model B looks better by GBP 130 per patient on the development data but is worse by 1,078.60 − 1,000.80 = GBP 77.80 per patient on held-out data. The optimism, held-out minus apparent error, is GBP 50.80 for Model A and 1,078.60 − 820 = GBP 258.60 for Model B.

4. Interpretation. The extra interactions fit the development data more closely but predict new practices less well, the signature of overfitting. Model A would be refitted to all 500 patients and carried forward. With only two candidates the selection optimism is small; screening dozens of specifications would call for nested cross-validation. MAE measures individual-level error, so a cohort model that uses the mean predicted cost also needs a check that predicted and observed mean costs agree in the held-out folds.

How cross-validation differs from related checks

Cross-validation is one of several ways of judging a model, and the names overlap. The main neighbours differ in what is held out and what question is answered.

• Bootstrap validation. Bootstrapping refits the procedure in resamples drawn with replacement and subtracts the average optimism from the apparent performance. Steyerberg and Harrell prefer it for prediction models.
• Split-sample validation. Steyerberg and Harrell state that split-sample approaches only work when not needed, and Collins and colleagues generally advise against them.
• External validation. Testing in a different population, place or period asks a different question, and Collins and colleagues expect performance to vary across them.
• Cross-validation of decision models. The ISPOR-SMDM Task Force-7 report by Eddy and colleagues lists cross validity among five types of model validation, meaning the comparison of different models that address the same problem, also called external consistency or model corroboration. Its value depends on how independent the models' methods and data are.

What a cross-validated score cannot show

A good cross-validated score shows that a modelling procedure predicts well for people like those in the sample. Collins and colleagues stress that a model can never be proven to be always valid, so the label "validated" says little without the design behind it.

• Extrapolation is untested. Held-out observations lie within the observed follow-up, so cross-validation cannot check the tail of a survival curve or a cost trajectory beyond the data.
• Causal effects are untested. A low predictive error says nothing about whether a coefficient is an unbiased treatment or policy effect.
• "Validation data" is ambiguous. TRIPOD+AI notes that in machine learning the term can mean data used for tuning or data used to evaluate performance, and calls the latter evaluation data.

Sources

• Bates S, Hastie T, Tibshirani R. Cross-validation: what does it estimate and how well does it do it? Journal of the American Statistical Association. 2024;119(546):1434-1445. doi:10.1080/01621459.2023.2197686
• Collins GS, Dhiman P, Ma J, et al. Evaluation of clinical prediction models (part 1): from development to external validation. BMJ. 2024;384:e074819. doi:10.1136/bmj-2023-074819
• Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ. 2024;385:e078378. doi:10.1136/bmj-2023-078378
• Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health. 2012;15(6):843-850. doi:10.1016/j.jval.2012.04.012
• Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning. 2nd ed. New York: Springer; 2009. Section 7.10.
• Jackson CH. survextrap: a package for flexible and transparent survival extrapolation. BMC Medical Research Methodology. 2023;23:282. doi:10.1186/s12874-023-02094-1
• Mihaylova B, Briggs A, O'Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Economics. 2011;20(8):897-916. doi:10.1002/hec.1653
• Rose S. A machine learning framework for plan payment risk adjustment. Health Services Research. 2016;51(6):2358-2374. doi:10.1111/1475-6773.12464
• Steyerberg EW, Harrell FE. Prediction models need appropriate internal, internal-external, and external validation. Journal of Clinical Epidemiology. 2016;69:245-247. doi:10.1016/j.jclinepi.2015.04.005
• Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics. 2006;7:91. doi:10.1186/1471-2105-7-91