Model Validation

What model validation establishes

It examines the conceptual structure, data, implementation, behaviour, and outputs rather than treating agreement with one observed result as proof that the whole model is valid.

Validation supports confidence in a model but does not certify it as universally true. A model may be valid enough for one population, time horizon, or decision while remaining unsuitable for another.

Validation is different from verification and calibration

Verification asks whether the model was implemented as intended, whereas validation asks whether the intended model represents the relevant real-world system adequately for its purpose. Calibration estimates uncertain parameters so that selected model outputs agree with calibration targets; it can contribute to validity but cannot replace independent validation.

A model can be technically correct but conceptually inappropriate, and a calibrated model can fit its targets while predicting new data poorly. The three activities should therefore be documented separately, even when they occur within one development workflow.

Validation begins with the decision problem

Validity is always judged against an intended use. The validation plan should therefore begin with the population, interventions, comparators, perspective, outcomes, time horizon, setting, and decisions the model is expected to support.

The required strength of validation should be proportionate to decision risk. A model informing national reimbursement, irreversible investment, or high-cost population policy generally warrants more extensive independent and external evaluation than an exploratory teaching model.

The main validation domains

No single test covers every way a model can be wrong. A defensible assessment combines complementary domains and records both the work performed and its findings.

The domains overlap but are not interchangeable. Strong internal fit cannot establish external validity, and agreement with another model is weak evidence if both models share the same structural error or data source.

Face validity and expert review

Face validity asks whether the model's assumptions, structure, inputs, and outputs appear clinically and economically plausible to people with relevant expertise. It is most useful when reviewers receive specific materials and questions rather than being asked only whether the model “looks reasonable.”

Expert review should document who reviewed the model, their relevant expertise, which components they examined, what concerns they raised, and how each concern was resolved or retained as a limitation. Agreement among experts strengthens confidence but does not replace empirical or technical validation.

Internal validation and diagnostic testing

Internal validation checks the model against its own specification, calibration targets, and accounting relationships. These tests are essential for detecting implementation errors and unexpected behaviour, but they mainly show that the model works as designed rather than that its design is correct.

Useful internal tests include:

• Conservation tests confirm that cohort proportions, state occupancy, or patient counts reconcile where the model requires them to do so.
• Boundary tests confirm that probabilities remain between zero and one and that impossible costs, utilities, or event counts are rejected.
• Extreme-value tests confirm that setting a risk to zero, a cost to zero, or treatment effects to equality produces the expected limiting result.
• Monotonicity tests confirm that outputs move in the expected direction when a parameter has a known directional relationship.
• Trace tests follow selected patients or cohorts through every calculation and reconcile intermediate values independently.
• Replication tests compare results with a separately implemented calculation or an analytical special case.

A successful internal check should record the expected result, actual result, tolerance, status, and model version. A generic statement that the model was “tested” is not enough to reproduce or appraise the validation.

External and predictive validation

External validation compares model outputs with observations that were not used to construct or calibrate the model. The comparison is most informative when the external population, treatment patterns, outcome definitions, follow-up, and calendar period are sufficiently aligned with the model's intended setting.

Predictive validation is a particularly strong form of external validation because the predictions are specified before outcomes are observed or are tested in a genuinely held-out period or dataset. Failure to match an external source may reflect model error, data error, or a real difference in case mix or care, so discrepancies require investigation rather than automatic rejection.

Quantifying agreement

Graphical comparisons should be accompanied by numerical measures that match the output and decision context. Metrics do not make the validity judgement automatically, but they reveal the direction, size, and pattern of discrepancies.

For observed values $O_i$, predicted values $P_i$, and $m$ comparison points, the mean error is

$$ME=\frac{1}{m}\sum_{i=1}^{m}(P_i-O_i).$$

The mean absolute error is

$$MAE=\frac{1}{m}\sum_{i=1}^{m}|P_i-O_i|.$$

The root mean squared error is

$$RMSE=\sqrt{\frac{1}{m}\sum_{i=1}^{m}(P_i-O_i)^2}.$$

Mean error identifies systematic overprediction or underprediction, while MAE describes typical absolute disagreement and RMSE gives greater weight to larger errors. Calibration plots, observed-to-expected ratios, interval coverage, and decision-specific loss measures may be more informative for some outputs.

Worked validation example

Consider a cost-effectiveness model that predicts overall survival at one, three, and five years. An independent registry provides external observations for a sufficiently comparable population.

The mean absolute error is

$$MAE=\frac{0.02+0.03+0.02}{3}=0.0233.$$

The root mean squared error is

$$RMSE=\sqrt{\frac{0.02^2+0.03^2+0.02^2}{3}}=0.0238.$$

The model overpredicts survival at all three points, so the issue is systematic rather than a mixture of cancelling errors. Whether an absolute difference of two to three percentage points is acceptable depends on sampling uncertainty, source comparability, the model's purpose, and the effect on incremental costs, quality-adjusted life-years, and the decision.

Calibration without circular validation

Calibration can improve agreement with observed targets by estimating parameters that are not directly available. The calibration objective should be prespecified, and the fitted parameters, bounds, weights, convergence, identifiability, and residual lack of fit should be reported.

A weighted least-squares objective is one common choice:

$$S(\theta)=\sum_{i=1}^{m}w_i\left[P_i(\theta)-O_i\right]^2,$$

where $\theta$ contains calibrated parameters and $w_i$ controls the contribution of each target. Evaluating the calibrated model against the same targets measures goodness of fit, not independent external validity.

To reduce circularity, analysts can reserve data for validation, use temporal or geographic holdouts, compare outcomes not included in the objective, and test the model in an independent population. Extensive calibration against many targets can overfit the available evidence and reduce transportability.

Decision validity

A discrepancy matters most when it could alter the decision supported by the model. Decision validation therefore traces validation findings through incremental costs, incremental effects, net benefit, budget impact, or another decision-relevant output.

At threshold $\lambda$, incremental net monetary benefit is

$$INMB=\lambda\Delta E-\Delta C.$$

If a plausible correction for an external-validation discrepancy changes the sign of $INMB$, the model's recommendation is not robust to that finding. A model may reproduce clinical outcomes closely yet remain decision-invalid if small errors occur in influential outcomes, costs, or long-term extrapolations.

Validation under uncertainty

Validation does not require exact agreement because both model outputs and empirical comparators are uncertain. Comparisons should preserve confidence intervals, sampling error, parameter uncertainty, and structural differences rather than treating observed estimates as error-free truth.

Probabilistic sensitivity analysis characterises uncertainty in model inputs, but it does not validate the model structure or implementation. Validation scenarios should instead represent plausible alternative structures, external-data corrections, or discrepancy adjustments and show their effect on decision outputs.

A practical validation workflow

Validation should be planned during model development rather than added after results are known. A version-controlled plan creates traceability from each test to its evidence, result, response, and remaining limitation.

1. Define the decision purpose, intended users, setting, scope, and consequences of error.
2. Specify the validation domains and criteria before inspecting final model results.
3. Map each important assumption, input, calculation, and output to an appropriate test.
4. Verify the implementation with unit, trace, boundary, extreme-value, and replication tests.
5. Review the conceptual model and input evidence with appropriately qualified experts.
6. Compare internal outputs with calibration targets and explain material lack of fit.
7. Compare outputs with independent data and reconcile differences in population and setting.
8. Compare benchmark scenarios with other credible models where genuine comparability exists.
9. Test whether unresolved discrepancies could change the decision conclusion.
10. Record findings, corrective actions, residual limitations, reviewers, dates, and model version.

The validation plan should evolve when the model, evidence, population, or intended use changes. Reusing an earlier validation report without assessing those changes can create false reassurance.

Recording validation evidence

A validation record should show what was tested and what happened, including failed or inconclusive tests. Transparent reporting allows a reviewer to distinguish missing validation from validation that was performed but revealed limitations.

Validation status should not be reduced to a single unqualified “passed” label. The report should explain coverage, material findings, unperformed tests, and the consequences of remaining uncertainty.

Common validation failures

Weak validation often results from asking only whether the model produces plausible headline outputs. Strong validation actively seeks errors and tests the model under conditions likely to expose them.

• Reusing calibration targets as the only external validation evidence creates a circular assessment.
• Comparing only aggregate outputs can conceal compensating errors in subgroups, states, or time periods.
• Selecting favourable comparators after results are known introduces confirmation bias.
• Treating expert agreement as empirical proof overstates face validity.
• Treating successful probabilistic sensitivity analysis as validation confuses uncertainty analysis with credibility assessment.
• Ignoring failed tests or undocumented model changes prevents an auditable assessment.
• Assuming that a model validated in one jurisdiction is automatically valid in another ignores transportability.

These failures do not always invalidate the entire model, but they reduce confidence and should be linked to the claims or decisions they affect. Material unresolved failures require correction, additional analysis, or explicit decision-maker judgement.

When validation is complete enough

Model validation is proportionate rather than absolute. The work is sufficient when the important failure risks have been examined with appropriate methods, material discrepancies have been resolved or disclosed, and the remaining uncertainty is acceptable for the intended decision.

Completion should be judged by coverage and consequence, not by the number of checklist items marked. Independent review, external data, and prospective testing strengthen confidence, but feasibility constraints should be reported honestly rather than converted into unsupported claims of validity.

How validation should be reported

The model report should describe the validation plan, tests, evidence, numerical results, discrepancies, corrective actions, and remaining limitations. It should also identify who performed each assessment, whether reviewers were independent, and which exact model version was examined.

Tools such as AdViSHE can support systematic reporting of validation efforts, while technical verification checklists can make implementation testing more reproducible. A checklist organises evidence; it does not determine by itself whether the model is valid for a particular decision.