Model Selection

Model Selection: Concept Architecture

Model selection is a decision about which mathematical representation is fit for a specified question, data set, and use. This page separates prediction, causal explanation, and health economic extrapolation because each demands different evidence of adequacy. It then shows how candidate models are compared, how selection itself can bias apparent performance, and what should be reported when the choice affects a decision.

Begin with the question the model must answer

A predictive model estimates future or unseen outcomes, an explanatory or causal model estimates a specified relationship or effect, and an economic decision model projects costs and health consequences of alternatives. A model that predicts well need not identify a causal effect, and a model with interpretable coefficients need not forecast accurately. Define the target population, outcome, time horizon, estimand, intended deployment, and acceptable errors before choosing candidates or metrics.

For a cost-effectiveness model, the candidate structure might differ in health states, competing events, treatment waning, or how progression is represented. Within a fixed structure, analysts may choose alternative distributions, covariates, or parameter estimates for survival. These are related levels of selection, but a comparison of two survival distributions cannot establish that the health-state structure is correct. Candidate models should represent plausible mechanisms and data-generating features, not merely a search across formulas until a favorable result appears.

Compare the same evidence under a stated criterion

For statistical models fitted by maximum likelihood to the same outcome observations under comparable likelihood conventions, Akaike's information criterion (AIC) and Bayesian information criterion (BIC) summarize fit with a complexity penalty. Let $\hat\ell$ be the maximized log-likelihood, $k$ the number of estimated free parameters, and $n$ the relevant number of independent observations. Lower values are preferred within the specified comparison set.

$$\mathrm{AIC}=2k-2\hat\ell,\qquad \mathrm{BIC}=k\log n-2\hat\ell.$$

These criteria are not probabilities that a model is true and do not prove that the lowest-scoring candidate will predict best outside the fitted data. Their penalties differ, so they can select different candidates. Compare criteria only when the underlying likelihoods, outcome definition, sample, censoring treatment, and any omitted likelihood constants permit a legitimate comparison; do not compare a likelihood-based AIC with an unrelated error score as if they were the same quantity.

For illustration, suppose $n=100$ and two candidate models use the same observations and compatible full log-likelihoods. Model A has $k=2$ and $\hat\ell=-120$; model B has $k=4$ and $\hat\ell=-116$. The calculations are:

Here AIC favors B while BIC favors A. For A, $\mathrm{AIC}=2(2)-2(-120)=244$ and $\mathrm{BIC}=2\log(100)-2(-120)\approx249.21$. For B, $\mathrm{AIC}=2(4)-2(-116)=240$ and $\mathrm{BIC}=4\log(100)-2(-116)\approx250.42$. The disagreement does not license choosing whichever criterion supports a preferred conclusion after seeing the result; state the selection rationale and investigate material consequences.

Choose a method that matches the purpose

For prediction, evaluate a preselected performance measure on data not used to fit or choose the model. Depending on the outcome, relevant measures may include absolute or squared prediction error, calibration, discrimination, and subgroup performance. A clinically meaningful forecast may require reliable uncertainty and decision consequences as well as a good aggregate score.

For causal inference, select variables and functional forms using a defensible causal question and knowledge of confounding, selection, and timing. Automated screening of covariates by individual p-values is not a substitute for a causal design; adjusting for an inappropriate post-treatment variable can introduce bias. In some settings a small set of prespecified alternative specifications is more informative than a single selected coefficient. Define the causal estimand and assumptions before judging which model is appropriate.

For survival extrapolation in health technology assessment (HTA), compare observed fit, the shape of hazards, long-term clinical and biological plausibility, external data, and the effects on projected costs and QALYs. AIC and BIC characterize fit to the observed data under assumptions but cannot directly verify decades of unobserved survival. A choice that changes mean survival or cost-effectiveness materially calls for transparent alternative scenarios and expert assessment.

Keep the selection process outside the test data

Trying many candidate models and then reporting performance on the same data used to select among them tends to produce optimistic performance estimates. A training set fits candidate parameters; an inner validation process chooses hyperparameters, features, or specifications; an untouched test set estimates performance for the resulting selection procedure. Cross-validation can help use limited data, but feature selection, scaling, imputation, and tuning must be repeated inside each training fold rather than computed once from the complete dataset.

Nested cross-validation uses inner folds for selection and outer held-out folds to estimate the performance of that selection procedure. It does not automatically provide a final model for deployment: after evaluation, refit the prespecified procedure on the appropriate development data, and distinguish that refit from an independently evaluated test. Separate participants or sites across folds when observations are correlated, and use a time-respecting split when the real task predicts future outcomes. External validation in a new setting tests transportability more directly than reuse of the same development sample, though it still depends on outcome comparability.

When the choice is a health economic model structure rather than a machine-learning tuning parameter, no random split alone resolves clinical plausibility, missing mechanisms, or long-run assumptions. Document a priori decision criteria, validate against external evidence when available, and retain plausible alternatives as structural uncertainty scenarios. A retrospective fit ranking is one piece of evidence, not an automatic release gate.

Make uncertainty in the choice visible

Estimated parameters are uncertain within a chosen model, while structural uncertainty concerns which model form or pathway representation is appropriate. Conventional confidence intervals calculated after selecting a model often treat that selection as fixed and can understate the total uncertainty. A probabilistic sensitivity analysis around parameters of a single chosen structure does not by itself account for alternative structures or extrapolation rules.

Report how costs, QALYs, and decisions change across defensible candidate models, including scenarios that produce less favorable results. Model averaging may be appropriate when its candidate set, weights, and target are justified; averaging cannot rescue implausible candidates or missing pathways. If external evidence conflicts with the statistically best in-sample fit, explain the conflict and the resulting expert judgment rather than concealing either. Distinguish uncertainty due to sampling, parameters, structure, heterogeneity, and future practice.

A practical selection record

A reproducible selection record connects the problem to the final choice. It allows a reviewer to see which alternatives were considered, why one was chosen, and whether the decision changes if assumptions change. Each step below should be completed before declaring a candidate fit for use.

1. Specify the target. Record outcome, population, estimand, decision horizon, and intended use before model comparison.
2. Set admissible candidates. Describe plausible structures, distributions, variables, constraints, and why any candidate was excluded.
3. Define comparison rules. Choose metrics and validation design that match the task; state how missing data, censoring, clustering, and competing events are handled.
4. Fit and check. Examine estimation stability, convergence, residuals or calibration, subgroup behavior, and numerical or structural diagnostics.
5. Validate beyond fitted data. Assess held-out or external performance where possible and clinical plausibility over the full decision horizon.
6. Quantify decision impact. Recalculate key outcomes under plausible alternatives and report unresolved structural uncertainty.
7. Preserve provenance. Record data version, code, specifications, selection rationale, expert input, and reviewer decisions so the analysis can be reproduced.

Common mistakes to avoid

The following mistakes can make a selected model appear stronger than the available evidence warrants. They arise because a comparison metric measures only a particular feature under a particular data-generating and validation setup. State the boundaries of the chosen model when reporting results.

• One criterion as proof: A lower AIC or BIC is a relative score among comparable candidates, not validation of the candidate set or its extrapolation.
• Post-selection test reuse: Reporting the best score from repeated tuning on a test set makes that set part of model development.
• Wrong target: Selecting by MAE when the policy needs an unbiased mean cost or a clinically weighted error can optimize the wrong prediction.
• Confusing prediction and causation: A predictive variable need not be a permissible confounder adjustment in a causal analysis.
• Unexamined extrapolation: Similar within-trial survival fits can produce materially different long-term means and QALYs.
• Hidden structural choice: Treating selected transition pathways as fixed while varying only parameter distributions understates model uncertainty.

Sources and further study

These sources distinguish validation of predictive models from selecting structures and extrapolations for economic evaluation. Their specific settings matter: a cross-validation rule for prediction is not a substitute for clinical and external validation of a lifetime survival model.

1. NICE Decision Support Unit. Technical Support Document 14: Survival analysis for economic evaluations. Primary methods guidance for systematic survival model selection using observed and external validity.
2. NICE. Economic evaluation in the technology appraisal manual. Official guidance on economic model structure, validation, and extrapolation plausibility.
3. Collins GS, et al. TRIPOD Statement. Original reporting guidance for clinical prediction model development and validation.
4. R Core Team. AIC documentation in the stats package. Primary software reference for AIC and BIC formulas and comparison conventions.
5. scikit-learn developers. Cross-validation and model selection. Primary software guidance on validation, tuning, and data leakage.