K-Fold Cross-Validation

K-Fold Cross-Validation: Concept Architecture

Test predictions on data withheld from fitting

K-fold cross-validation is an internal validation procedure that divides available observations into $K$ disjoint parts, or folds. It fits the full model-building process on $K-1$ parts and evaluates predictions on the remaining part, repeating until every part has served once as validation data. Its main use is to estimate how a specified prediction procedure may perform on new observations drawn under conditions comparable to those in the study data. It can also support model and tuning choices, provided those choices are assessed without reusing the same validation information as a final unbiased performance test.

Cross-validation applies to a procedure, not merely a fixed list of fitted coefficients. Every operation learned from data—imputation, normalization, feature selection, calibration, and hyperparameter fitting when part of the procedure—must be repeated within the training data of each validation fold. A good internal result is not evidence by itself that the model will work at a different hospital, in a later year, or after a change in care pathways.

Partition the right unit

Let a dataset of $n$ eligible units be partitioned into nonoverlapping folds $F_1,\ldots,F_K$ whose union is the full set. For fold $k$, fit the prespecified pipeline on all units outside $F_k$ and obtain predictions for units in $F_k$ only. If $L(y_i,\widehat{f}_{-k}(x_i))$ is a per-unit loss, an observation-weighted out-of-fold loss is:

$$\widehat{R}_{\mathrm{OOF}}=\frac{1}{n}\sum_{k=1}^{K}\sum_{i\in F_k} L\left(y_i,\widehat{f}_{-k}(x_i)\right).$$

Here $\widehat{f}_{-k}$ denotes the model and preprocessing fitted without fold $k$. For equal fold sizes, this is the mean of the $K$ fold-average losses. For unequal fold sizes, an unweighted mean of fold losses gives each fold equal influence, whereas pooling per-unit losses gives each unit equal influence; specify which target is intended. Metrics such as area under the ROC curve or calibration slope do not decompose into per-person losses in this way, so calculate and describe them using a suitable fold or out-of-fold prediction procedure rather than inserting them into the additive formula.

Choose the unit of independence and the target use case before splitting. Several visits from one patient generally belong in the same fold if the target is prediction for a new patient. Multiple patients within a hospital can share site-specific information; leaving out entire hospitals is more relevant for predicting at a new hospital. If the target is future patients, a chronological split or rolling-origin validation may be more credible than randomly mixing past and future. Plain random K-fold validation assumes the held-out units are sufficiently exchangeable with the training population for the intended generalization question.

Work a five-fold example

Suppose 100 independent patients are divided into five folds of 20. On each iteration a classification pipeline trains on 80 patients and predicts an outcome for the 20 withheld patients. Imagine that the numbers of incorrect classifications across folds are 2, 3, 4, 1, and 5. Each patient receives one out-of-fold prediction, and the illustrative error estimate is:

$$\widehat{R}_{\mathrm{OOF}}=\frac{2+3+4+1+5}{100}=0.15.$$

The corresponding accuracy is 85% under the chosen classification cutoff, but this alone says little about probability calibration, sensitivity for a rare outcome, or clinical usefulness. A procedure that predicts “no event” for nearly everyone can have high accuracy with poor detection. If the eventual model is to be deployed, refit the selected, fixed procedure to all 100 patients after validation; do not report its in-sample accuracy as the cross-validated estimate. The cross-validation estimate largely reflects fits trained on 80, rather than 100, patients and can be variable in a small dataset.

The example's folds are equal-sized. If fold sizes vary or outcomes are rare, report fold sizes and event counts; stratification may improve the stability of class proportions in a random split, but it does not solve patient-level dependence, temporal leakage, or lack of external representativeness.

Keep the entire training pipeline inside each fold

Splitting after using all patients to choose features or calculate imputation means allows information from a supposedly held-out fold to affect the fitted model. Likewise, estimating standardization parameters on the full dataset, balancing classes before splitting, selecting variables by their full-sample association with the outcome, or fitting a target encoder globally leaks validation information. Define the split first, then fit every learned step only on that iteration's training units and apply the fitted transformations to its validation units. Fixed, externally specified transformations that use no study outcomes or data-derived parameters can be applied consistently without fitting.

Patient-level splitting must also respect the prediction time. A variable recorded after the outcome or a code generated by future treatment cannot become a valid baseline predictor merely by putting records in different folds. Duplicates and related admissions across training and validation sets can inflate performance. In a censored survival prediction task, define the outcome horizon, censoring handling, and validation metric so that a patient with unknown status at the horizon is not treated as an observed event-free case.

Separate tuning from performance estimation

If many model classes, feature sets, or tuning values are tried and the one with the best $K$-fold score is reported using that same score, selection exploits noise in validation results and tends to make the winner look too good. Nested cross-validation uses an inner procedure for all model choices within each outer training set, then scores the selected pipeline on an untouched outer fold. The outer-fold results estimate the performance of the whole selection strategy, subject to sample size and the chosen splitting scheme. A genuinely separate test set or later external dataset can also evaluate a finalized procedure.

With nested validation, avoid global feature selection or tuning before the outer split; that would contaminate every outer assessment. After completing evaluation, choose the final modeling procedure using the prespecified development strategy and fit it on the eligible development data. The exact coefficients of that final fit differ from those in the folds; the validation estimate concerns the method of producing a model. Repeated K-fold splitting can reveal sensitivity to the arbitrary partition, but repeated fold scores are correlated because their training sets overlap and should not be treated as independent trial replications.

Choose folds that match the deployment question

Grouped or temporal schemes can have fewer effective independent validation units and much wider uncertainty than ordinary patient-level folds. Stratification preserves outcome proportions approximately but is not a replacement for grouping. A leave-one-hospital-out evaluation addresses a harder and different question than random patient-level K-fold within all hospitals. State whether there are enough groups to make the intended assessment meaningful.

Read the performance estimate carefully

Select measures that reflect use: calibration and a proper probability score for risk predictions, discrimination when ranking matters, and consequences of decisions at relevant thresholds. For survival outcomes, use time-specific discrimination and calibration with appropriate censoring methods. Report confidence or uncertainty intervals where defensible, including variation across sites or periods, rather than only a convenient mean score. Performance may differ across important subgroups even when the overall average is satisfactory.

The choice of $K$ trades training-set size against the size of each validation fold and computational burden; five or ten folds are common conventions rather than universally optimal constants. A small $K$ trains each candidate on less data; a large $K$ can leave tiny and highly correlated validation assessments. The sampling behavior depends on the estimator, outcome prevalence, dataset size, and splitting strategy. Neither a large $K$ nor repeated resampling repairs outcome leakage, biased sampling, or a badly defined prediction target.

Link validation to health economic decisions

If a risk model selects patients for treatment or feeds transition probabilities into an economic model, prediction errors affect who is treated, resource use, events avoided, costs, and QALYs. Evaluate calibration and clinically relevant decision thresholds in the population where the model would be used, then carry prediction and model uncertainty into the economic analysis. A cross-validated accuracy or area under the curve is not itself an incremental cost-effectiveness result. External validation, updating, and impact evaluation may still be needed before relying on a model for policy.

Report the unit of splitting, $K$, fold sizes and event counts, randomization seed or temporal cutoffs, grouping and stratification, the entire within-fold pipeline, all tuning and nested-selection steps, metrics, handling of missingness and censoring, uncertainty, and target setting. This allows another analyst to reproduce the validation and judge whether the folds answer the intended generalization question.

Sources and further reading

• scikit-learn User Guide: Cross-validation—evaluating estimator performance. Primary documentation on K-fold, stratified, grouped, and time-aware validation.
• scikit-learn User Guide: Common pitfalls and recommended practices. Primary documentation on preprocessing leakage and pipeline construction.
• Varma S, Simon R. Bias in error estimation when using cross-validation for model selection. BMC Bioinformatics (2006). Original simulation study of selection bias and nested validation.
• Efthimiou O, et al. Developing clinical prediction models: A step-by-step guide. BMJ (2024). Discusses validation and interpretation for clinical prediction.