Concept Architecture
Concept
Theoretically, Model Diagnostics comprise the collection of statistical and analytical procedures used to evaluate whether a fitted model satisfies its underlying assumptions, adequately represents the observed data and produces reliable parameter estimates. Model diagnostics identify potential problems including poor fit, influential observations, multicollinearity, heteroscedasticity, non-linearity and violations of distributional assumptions. In health economics, model diagnostics support the validation of statistical models before their results are incorporated into economic evaluations or decision-analytic models.
Mathematically, model diagnostics are based on diagnostic statistics that quantify deviations between model assumptions and observed data. The specific mathematical framework depends on the modelling approach and may include residual analysis, influence diagnostics, goodness-of-fit measures, likelihood-based statistics and information criteria. No single diagnostic is sufficient; multiple complementary measures are interpreted together to evaluate model adequacy.
In practice, analysts perform model diagnostics after model estimation by examining residual plots, leverage statistics, influence measures, goodness-of-fit statistics and assumption tests. Health economists routinely apply these procedures when developing regression models, survival models, utility mapping algorithms and risk prediction equations to ensure robust parameter estimation before model outputs are used in cost-effectiveness analyses.
Purpose
Used to evaluate whether statistical models satisfy their assumptions, identify potential model deficiencies and support the development of reliable models for health economic evaluation.
Mathematical Formulae
Primary Formula
There is no universally recognised canonical mathematical formula.
Supporting Formulae
Examples of recognised diagnostic measures include:
Coefficient of determination:
R� = 1 ? [�(y? ? ??)�] / [�(y? ? ?)�]
Standardised residual:
r? = e? / [�?�(1 ? h??)]
Cook's Distance:
D? = �(?? ? ?????)� / (p ? MSE)
Related Mathematical Methods
- Residual analysis
- Goodness-of-fit assessment
- Likelihood ratio testing
- Cook's Distance
- DFBETA
- Leverage analysis
- Variance Inflation Factor
- Akaike Information Criterion
- Bayesian Information Criterion
- Cross-validation
Example
A health economist develops a regression model predicting annual healthcare costs using age, disease severity and comorbidity.
Model diagnostics identify:
- R� = 0.84
- No systematic residual pattern
- Maximum Cook's Distance = 0.18
- All variance inflation factors below 3
- Residuals approximately normally distributed
Collectively, the diagnostic results indicate that the model adequately satisfies the principal assumptions and is appropriate for estimating cost parameters used within a cost-effectiveness model.
Excel Implementation
| Function | Example Formula | Health Economics Application |
|---|---|---|
| RSQ | =RSQ(B2:B501,C2:C501) | Calculates model goodness of fit. |
| SUMXMY2 | =SUMXMY2(B2:B501,C2:C501) | Calculates residual variation. |
| ABS | =ABS(B2-C2) | Calculates absolute residuals for diagnostic review. |
| AVERAGE | =AVERAGE(D2:D501) | Summarises residual behaviour across observations. |
| CHISQ.TEST | =CHISQ.TEST(B2:B6,C2:C6) | Performs selected goodness-of-fit assessments where appropriate. |
VBA (Optional)
Automate generation of a comprehensive diagnostic report including residual analyses, influence statistics, goodness-of-fit measures and assumption checks.
Sources
- Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed.
- Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation.
- Harrell FE. Regression Modeling Strategies. 2nd ed.
- Cook RD, Weisberg S. Residuals and Influence in Regression.
- Hastie T, Tibshirani R, Friedman J. The Elements of Statistical Learning.
- ISPOR Good Practice Reports on statistical modelling and model validation.
Related Concepts (2)
Library
Publications
1
Model Transparency and Validation: A Report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7 — Eddy, Hollingworth, Caro, Tsevat, McDonald & Wong, Task Force Report 7 ed., 2012 (Value in Health / Medical Decision Making)
Best-practice guidance on model transparency and validation, defining face, internal, external and predictive validation and setting out how models should be documented for scrutiny.
Journal ArticleView source →
Frequently Asked Questions (6)
What are model diagnostics?
A set of statistical and technical checks assessing a model's performance, such as poor calibration or unstable results under minor input changes.
Source: Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Value in Health. 2012;15(6):843-850. doi:10.1016/j.jval.2012.04.012.
What does instability in a model's results suggest?
If small, reasonable changes to an input produce large swings in a model's results, the model is unstable, and its output cannot be relied on as a firm basis for a decision. Diagnostics that probe this, by perturbing inputs and watching the response, can expose such fragility, which may stem from a coding error, an ill-conditioned calculation, or a structure too sensitive to be trusted. Stable, predictable responses to small changes are a sign of a sound model. Instability is a warning to investigate. Eddy and colleagues (2012) describe these checks.
Source: Eddy et al. 2012
What do model diagnostics check?
Model diagnostics check aspects of a model's performance and behaviour, such as whether its predictions are well calibrated to observed data, whether its results are stable rather than changing erratically under small input changes, whether it behaves sensibly at extreme values, and whether its outputs are internally consistent. By examining these, diagnostics reveal problems like poor fit, instability, or errors that the model's main results might not expose, so they provide a fuller picture of whether the model is functioning correctly.
Source: Briggs, Claxton & Sculpher 2006
Why are model diagnostics important?
Model diagnostics are important because a model can produce plausible-looking results while suffering from problems, such as poor calibration or instability, that undermine their reliability, and diagnostics reveal these issues. Checking a model's behaviour beyond its headline outputs helps detect weaknesses that would otherwise go unnoticed, so that they can be addressed. Diagnostics thus contribute to establishing whether a model performs well and can be trusted, forming part of the checks that support confidence in its results.
Source: Eddy et al. 2012
In model diagnostics, what does instability under minor input changes indicate?
Instability, where a model's results change disproportionately under small changes in inputs, indicates that the model may be poorly conditioned, contain errors, or rest on a fragile structure, so its outputs are unreliable and sensitive to noise. Such behaviour suggests the results cannot be trusted as robust, since minor variation should not produce large swings unless the model is genuinely sensitive to those inputs. Detecting instability through diagnostics prompts investigation of its cause and, where appropriate, revision of the model.
Source: Briggs, Claxton & Sculpher 2006
How do model diagnostics relate to validation?
Model diagnostics relate to validation as technical and statistical checks that contribute to assessing whether a model performs adequately. While validation broadly establishes that a model is built correctly and represents reality, diagnostics examine specific aspects of its behaviour, such as calibration and stability, that reveal problems. They form part of the evidence used in validation and credibility assessment, so diagnostics support the overall judgement of whether a model is sound by probing its performance in ways its main outputs do not.
Source: Eddy et al. 2012
Trust Record
Verified by Dr Darrin Baines
British health economist
Professional identity: darrinbaines.org
Verification date: 15 Oct 2025
Content version: 1.0.0
Canonical Identity
- Persistent URI
- https://healtheconomics.wiki/concept/model-diagnostics
- Term code
- HE-EM-MV-043
Stable URI · Machine-readable · Resolvable · CC BY 4.0