Model Diagnostics

Model Diagnostics in Health Economic Evaluation

Model diagnostics are the checks an analyst runs to find out whether a fitted model, or a decision model built from fitted inputs, is behaving as its assumptions say it should. In health economics they are applied in two places, to the statistical models that supply costs, utilities and survival curves, and to the health economic model that combines those inputs into an ICER or net benefit. A model can produce a plausible headline result while hiding a misfitting cost regression or a coding fault, and diagnostics find such problems before a committee relies on the result. They form one part of model validation, which judges whether the whole model is fit for its decision purpose.

Two places where diagnostics are run

Statistical diagnostics ask whether an estimated equation fits its data and meets its assumptions. Behavioural checks ask whether the programmed decision model does what its specification says. The table pairs common checks with what they expose.

Residual and influence checks for cost and utility regressions

Regression diagnostics start from the residual, the gap between an observed value and the model's prediction. Mihaylova and colleagues, reviewing methods for health care cost data, describe costs as non-negative, often substantially skewed and heavy tailed, and often with a mass at zero for non-users, so residual behaviour shows whether the chosen estimator suits the data.

$$r_i = \frac{e_i}{s\sqrt{1 - h_{ii}}} \qquad D_i = \frac{r_i^{2}}{p} \cdot \frac{h_{ii}}{1 - h_{ii}}$$

where $e_i$ is the raw residual for patient $i$ (observed minus fitted value), $s$ is the residual standard error, $h_{ii}$ is the hat value of patient $i$ (the diagonal element of the hat matrix), $p$ is the number of estimated coefficients, $r_i$ is the internally standardised residual and $D_i$ is Cook's distance.

A large standardised residual marks an outcome far from its prediction, a large $h_{ii}$ marks a patient with unusual covariate values, and Cook's distance combines the two into a measure of how much the fitted model moves when the patient is left out. Generalised linear models for costs use adapted versions of the standardised residual and Cook's distance.

Manning and Mullahy compared ordinary least squares on log costs with generalised linear models, found that no single alternative was best under all the conditions examined, and proposed an algorithm for choosing among estimators. Mihaylova and colleagues recommend checking sensitivity to the choice of transformation and distribution, and they broadly agree with an earlier conclusion that many diagnostics used to choose among cost models have limitations of their own.

Fit diagnostics for survival curves used in extrapolation

The extrapolated part of a survival curve generally has a large effect on estimated mean survival, so the choice among fitted curves weighs heavily in appraisal. NICE Decision Support Unit Technical Support Document 14 (Latimer, June 2011, last updated March 2013) describes visual comparison of a fitted parametric survival model with the Kaplan-Meier curve, log-cumulative hazard plots and information criteria. It notes that residual tests such as Cox-Snell, martingale and Schoenfeld residuals may also be useful.

The standard log-cumulative hazard plot graphs the log of minus the log of the survivor function against log time. TSD 14 states that it tests the suitability of the Weibull and exponential distributions and also shows whether the proportional hazards assumption is reasonable.

$$\mathrm{AIC} = 2k - 2\ln L \qquad \mathrm{BIC} = k\ln n - 2\ln L$$

where $k$ is the number of estimated parameters, $\ln L$ is the maximised log-likelihood of the fitted model, $n$ is the number of patients and lower values indicate a better trade-off between fit and complexity. The Akaike information criterion and the Bayesian information criterion can compare models that are not nested. TSD 14 contrasts them with minus twice the log-likelihood, which it says suits only nested models and was used erroneously in past NICE appraisals, and notes that the BIC penalises extra parameters more heavily.

TSD 14 is explicit about the limit of these checks. Visual inspection, log-cumulative hazard plots and AIC or BIC are based only on relative fit to the observed data, so they address the internal validity of the fitted models and say nothing about the period after trial follow-up. The NICE manual (PMG36, section 4.6.20) asks that the external validity of an extrapolation be assessed against clinical and biological plausibility and external data, and that the limitations of statistical fit measures be documented when they are used.

Diagnostics for mapping models that predict utilities

A mapping function predicts EQ-5D utilities from a clinical measure, so its diagnostics judge predicted group means and the predicted distribution of scores. NICE DSU Technical Support Document 22 (Wailoo, Hernandez Alava and Pudney, June 2023, updated September 2026) says that summary fit measures provide limited information. It describes mean absolute error and root mean squared error as relatively insensitive given the narrow range of utility data, calls mean error not useful in this context, and notes that AIC and BIC may conflict and cannot compare models with different forms of the dependent variable.

TSD 22 recommends plots of predicted EQ-5D means against observed group means, with groups defined by a clinical conditioning variable, and plots of the cumulative proportion of observed values against data generated from the model. It also states that the model should not generate predictions outside the feasible range of EQ-5D.

Behaviour checks on the decision model

Diagnostics on the decision model look for implementation errors and for outputs that move in unexpected ways. The ISPOR-SMDM task force report on transparency and validation (Eddy and colleagues, 2012) calls this verification, or internal validity, and lists techniques that include double programming, comparison with hand calculations, sensitivity analysis, extreme value analysis and trace analysis of individual events. It adds that verification should involve sensitivity analysis of all parameters to see whether the direction and magnitude of outputs behave as expected.

The TECH-VER checklist (Büyükkaramikli and colleagues, 2019) orders verification tests in a hierarchy. Black-box tests check results against prior expectations, including at extreme parameter values, and, to save time, white-box tests or replication become necessary when an earlier test fails. The NICE manual (PMG36, section 4.6.4) asks submissions to detail the quality assurance methods used in developing the model and to provide the methods and results of model validation.

Worked example separating a coding fault from genuine sensitivity

All numbers in this example are illustrative. A decision model compares a new drug with usual care. The response probability is 0.40 with usual care and 0.45 with the drug, each responder gains 0.20 QALYs a year for 5 years (undiscounted for simplicity), the drug costs GBP 1,500 per patient, and each responder avoids GBP 2,000 of other care.

$$\Delta E = (p_1 - p_0) \times g \times T \qquad \Delta C = c_d - (p_1 - p_0) \times a$$

where $p_1$ and $p_0$ are the response probabilities with the drug and usual care, $g$ is the annual QALY gain per responder, $T$ is the number of years, $c_d$ is the drug cost per patient, $a$ is the care cost avoided per responder, and $\Delta E$ and $\Delta C$ are the incremental QALYs and costs per patient.

1. Base case. The incremental QALYs are 0.05 × 0.20 × 5 = 0.05. The incremental cost is 1,500 minus 0.05 × 2,000, or GBP 1,400. The ICER is 1,400 / 0.05 = GBP 28,000 per QALY gained.

2. Extreme value check. Setting the drug's response probability equal to usual care at 0.40 must give zero incremental QALYs and an incremental cost of exactly GBP 1,500, the drug cost with no savings. Setting the responder QALY gain to zero must give zero incremental QALYs and an incremental cost of GBP 1,400. A non-zero QALY difference in either test would point to code that bypasses an input.

3. Monotonicity check. Raising the drug cost by GBP 100 to GBP 1,600 should raise the ICER by 100 / 0.05 = GBP 2,000, to GBP 30,000. A model whose ICER fell when the price rose would fail this test.

4. Small input change. Moving the drug's response probability by 0.01, a relative change of about 2.2%, gives 0.04 or 0.06 incremental QALYs and incremental costs of GBP 1,420 or GBP 1,380. The ICER becomes 1,420 / 0.04 = GBP 35,500 or 1,380 / 0.06 = GBP 23,000, a change of plus 26.8% or minus 17.9%.

5. Net monetary benefit. At GBP 25,000 per QALY the net monetary benefit is 25,000 × 0.05 minus 1,400, or minus GBP 150, in the base case, minus GBP 420 at 0.44 and plus GBP 120 at 0.46.

6. Switching values. Net monetary benefit is zero at a response probability of 0.40 + 1,500 / 27,000, about 0.456, at GBP 25,000 per QALY, and 0.40 + 1,500 / 37,000, about 0.441, at GBP 35,000, both within 0.01 of the base case.

7. Interpretation. The model passes steps 2 and 3 and its step 4 outputs match the hand calculation, so the swing in step 4 does not point to a coding fault. The ICER responds disproportionately because the QALY difference sits in its denominator. The finding is that the decision is genuinely sensitive to the response probability, which the NICE manual (PMG36, section 4.7.22) suggests presenting as a switching value, with its distance from the best estimate, for thresholds such as GBP 25,000 and GBP 35,000 per QALY gained.

Reading diagnostic results for a decision

A clean set of diagnostics supports a narrow claim. Eddy and colleagues state that verification does not evaluate the accuracy of the model's structure or predictions, so a well-coded model with a poorly chosen structure can still mislead. Where several specifications pass the checks equally well, model averaging or a set of scenarios can carry the choice into the sensitivity analysis.

Where model diagnostics sit among related checks

Neighbouring checks overlap with diagnostics, which look for misfit or malfunction in one component. Goodness of fit is one diagnostic question, how closely a fitted model reproduces its data. Calibration adjusts unknown parameters until outputs match targets, and Eddy and colleagues ask that its reporting include the target, goodness-of-fit metric, search algorithm, acceptance criteria and stopping rule.

Cross-validation estimates performance on data not used for fitting, which guards against overfitting that within-sample diagnostics cannot see. Validation combines this evidence with face validity and external comparisons.

Misreadings and limits of model diagnostics

Diagnostic statistics are easy to compute and easy to over-read. The first two errors below are named in NICE DSU guidance.

1. Treating the best AIC as the best extrapolation. TSD 14 states that fit statistics address internal validity only.
2. Relying on a single summary statistic. TSD 22 warns that mean absolute error and root mean squared error can conceal systematic patterns in prediction errors.
3. Deleting influential patients to improve fit. A genuine high-cost patient belongs to the population whose mean cost is being estimated, so showing results with and without that patient keeps the choice visible.
4. Reading instability as error. A ratio such as the ICER can swing widely after a small input change in a correct model, while a failed extreme value test points to a fault.

Sources

• Büyükkaramikli NC, Rutten-van Mölken MPMH, Severens JL, Al M. TECH-VER: a verification checklist to reduce errors in models and improve their credibility. PharmacoEconomics. 2019;37(11):1391-1408. https://doi.org/10.1007/s40273-019-00844-y
• Eddy DM, Hollingworth W, Caro JJ, Tsevat J, McDonald KM, Wong JB. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health. 2012;15(6):843-850. https://doi.org/10.1016/j.jval.2012.04.012
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Decision Support Unit; June 2011, last updated March 2013. https://www.sheffield.ac.uk/nice-dsu/tsds/survival-analysis
• Manning WG, Mullahy J. Estimating log models: to transform or not to transform? Journal of Health Economics. 2001;20(4):461-494. https://doi.org/10.1016/S0167-6296(01)00086-8
• Mihaylova B, Briggs A, O'Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Economics. 2011;20(8):897-916. https://doi.org/10.1002/hec.1653
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 4.6.4, 4.6.20 and 4.7.22. https://www.nice.org.uk/process/pmg36
• Wailoo A, Hernandez Alava M, Pudney S. NICE DSU Technical Support Document 22: Mapping to estimate health state utilities. Decision Support Unit; June 2023, updated September 2026. https://doi.org/10.15131/shef.data.33849340.v1