Topic

Model validation and calibration

Validation tests whether a model's structure, inputs and results are credible enough to inform a decision, for example by comparing predictions with observed data. Calibration adjusts uncertain inputs until model outputs match known targets. Statistical criteria such as the Akaike and Bayesian information criteria help choose between competing models.

Concepts in this topic

• Akaike Information CriterionAkaike information criterion (AIC) is a statistic ranking models fitted to the same data, such as survival curves in HTA, by fit penalised per parameter.
• Bayesian Information CriterionBayesian information criterion (BIC) ranks models fitted to the same data, such as HTA survival curves, by fit with a penalty that grows with sample size.
• CalibrationCalibration is the process of estimating or adjusting model parameters so that selected model outputs agree with relevant observed targets under an explicit measure of fit.
• Calibration AssessmentAn evaluation of how closely a calibrated model's predicted outputs match the observed data used as calibration targets, typically using formal fit statistics.
• Credibility AssessmentAn overall evaluation of whether a model's structure, evidence base, and validation results provide enough confidence to inform a real decision.
• Cross-ValidationCross-validation is a resampling procedure that repeatedly trains a predictive model on one part of a dataset and evaluates it on held-out observations to estimate performance under a specified data-splitting scheme.
• DFBETAA regression diagnostic measuring how much a coefficient would change if a particular observation were removed, identifying disproportionately influential data points.
• Goodness of FitGoodness of fit describes how adequately a statistical model or probability distribution represents the observed data for its intended purpose.
• Goodness of Fit TestA goodness of fit test assesses whether a specified discrepancy between observed data and a proposed probability model is unusually large under that model’s sampling assumptions.
• Likelihood Ratio TestA statistical test comparing the fit of two nested models by their ratio of likelihoods, used to judge whether added parameters improve fit.
• Model DiagnosticsA set of statistical and technical checks assessing a model's performance, such as poor calibration or unstable results under minor input changes.
• Model SelectionThe process of choosing a model from plausible alternatives using criteria suited to the intended question, data, predictive or explanatory performance, and decision context.
• Model ValidationModel validation is the structured assessment of whether a model is sufficiently accurate, credible, and fit for its stated decision purpose.
• Predictive AccuracyThe degree to which a model's forecasts or classifications correspond to actual observed outcomes when tested against real data.
• Residual AnalysisA statistical technique examining the differences between a model's predicted values and observed data, checking for systematic patterns left unexplained.
• Statistical FitA measure of how well a specified statistical model's predictions align with observed data, judging whether the specification adequately fits the pattern.
• Wald TestA statistical test assessing whether an estimated parameter differs significantly from a specified value, based on the ratio of estimate to standard error.