Credibility Assessment

How credibility connects a model to a real decision

A credibility assessment judges whether a model is sufficiently trustworthy and relevant to inform a specified decision. It integrates the model's conceptual basis, technical implementation, evidence, validation, uncertainty, transparency, and independence of review. This page explains how those elements are examined together and why credibility is always conditional on the decision context rather than a permanent property of the model.

Credibility is not the same as correctness

No decision model is a complete or perfectly correct representation of reality. A credible model is one whose simplifications, evidence, calculations, and limitations are understood well enough for its intended use. The assessment therefore asks whether the model is fit for purpose, not whether it is true in every setting.

Concept	Central question	Why it is distinct
Credibility assessment	Is there enough justified confidence to use the model for this decision?	It integrates multiple forms of evidence and judgment.
Verification	Was the model implemented as intended?	Correct code can still implement a poor conceptual model.
Validation	Does the model behave consistently with relevant evidence and expectations?	Agreement supports credibility but does not prove universal accuracy.
Calibration	Were uncertain parameters adjusted to reproduce selected targets?	Fit to calibration targets is not independent validation.
Transparency	Can users understand and scrutinise the model and evidence?	Transparency enables assessment but does not guarantee quality.

The intended use sets the standard

The same model may be credible for exploratory analysis but inadequate for reimbursement, clinical guidance, or long-term policy. Stakes, reversibility, population size, uncertainty, and the consequences of error determine how much evidence and scrutiny are needed. The assessment should begin by defining the decision before reviewing technical detail.

The decision specification should identify:

• The decision maker and question the model is intended to inform.
• The target population, subgroups, intervention, comparators, and setting.
• The perspective, outcome measures, time horizon, and discounting rules.
• The jurisdiction and decision date.
• The expected use of results, including whether the model ranks options, estimates value, forecasts budgets, or explores scenarios.
• The consequences of a false-positive, false-negative, or miscalibrated decision.

A clear conceptual model comes before code

Conceptual validity concerns whether the model includes the important disease processes, treatment pathways, outcomes, dependencies, and feedback relevant to the decision. Clinical and methodological experts should be able to understand why the chosen structure represents the problem. Important omissions or implausible causal assumptions cannot be repaired by technically flawless programming.

Reviewers should examine:

• Whether the model type fits the clinical and decision problem.
• Whether health states, events, pathways, and interactions reflect current understanding.
• Whether comparators represent relevant practice and future changes.
• Whether the time horizon captures material benefits, harms, and costs.
• Whether structural assumptions are supported, testable, and explored.
• Whether heterogeneity, treatment waning, recurrence, competing risks, and adverse events are represented when material.

Evidence must be relevant as well as high quality

Credibility depends on how well each input source supports the parameter and decision context for which it is used. A rigorous study can still be poorly matched to the model's population, comparator, setting, outcome, or time horizon. Evidence selection should be systematic, documented, and resistant to choosing values because they produce a preferred result.

For each important input, assess:

• Internal validity and risk of bias in the source evidence.
• Applicability to the model population, intervention, comparator, and setting.
• Consistency across sources and reasons for heterogeneity.
• Recency and whether practice or prices have changed.
• Transformation from observed evidence to the parameter used in the model.
• Extrapolation beyond observed follow-up.
• Correlation with other inputs and uncertainty in the estimate.

Verification checks whether the model was built correctly

Verification tests whether equations, code, data handling, and outputs implement the documented design. It includes checking individual components and the integrated model. A model that runs without an error message has not necessarily been verified.

Useful verification methods include:

• Independent review of equations, code, formulas, and data transformations.
• Unit tests for calculations with known answers.
• Boundary and extreme-value tests.
• Conservation checks for populations, probabilities, flows, and costs.
• Replication of key results using an independent calculation.
• Comparison of programmed logic with the technical specification.
• Automated regression tests that flag unintended changes after revisions.

For mutually exclusive transition probabilities from state (i), a basic conservation check is:

$$ \sum_{j=1}^{J} p_{ij}(t) = 1 $$

Passing this check confirms internal consistency for that row; it does not establish that the transition probabilities are clinically realistic.

Validation asks whether model behaviour is plausible

Validation compares model structure, behaviour, and outputs with evidence or informed expectations. Different validation types answer different questions and should not be collapsed into a single statement that the model is validated. Evidence should include the result of the test, discrepancies found, action taken, and any unresolved limitation.

Validation type	What is compared	Main limitation
Face validity	Structure, assumptions, inputs, and outputs against expert expectations	Expert agreement can be incomplete or biased.
Internal validation	Model outputs against data or targets used in development	Agreement may reflect calibration rather than prediction.
Cross-validation	Predictions for withheld parts of the development data	Withheld data may still share the same setting and biases.
External validation	Model outputs against independent data	Differences can reflect transportability as well as model error.
Predictive validation	Forecasts against outcomes observed later	It requires time, stable definitions, and genuinely prospective separation.
Comparative validation	Results against another model addressing the same problem	Shared assumptions can create agreement without accuracy.

Calibration fit must be interpreted carefully

Calibration estimates uncertain parameters so that selected model outputs reproduce observed targets. Good fit supports the calibrated portion of the model, but overfitting can weaken prediction in other populations, outcomes, or periods. The calibration targets, parameters, objective function, search method, convergence, and alternative solutions should be reported.

A weighted sum of squared errors can be used as an objective function:

$$ SSE_w = \sum_{k=1}^{K} w_k\left(y_k-\hat{y}_k(\boldsymbol{\theta})\right)^2 $$

where (y_k) is observed target (k), (\hat{y}_k(\boldsymbol{\theta})) is the modelled value under parameters (\boldsymbol{\theta}), and (w_k) reflects the target's precision or importance. A low value shows fit to the selected targets, not proof that unobserved predictions are correct.

Prediction error needs scale and context

Validation statistics should retain the units and decision relevance of the outcome. Average error can conceal large subgroup or time-specific discrepancies, while percentage errors can behave poorly near zero. Graphical comparisons, calibration plots, residual patterns, and uncertainty intervals should accompany headline statistics.

For (n) observed values, the root mean squared error is:

$$ RMSE = \sqrt{\frac{1}{n}\sum_{i=1}^{n}\left(y_i-\hat{y}_i\right)^2} $$

RMSE penalises larger errors and is expressed in the outcome's units. Whether a value is acceptable depends on the decision, comparator differences, natural variability, and consequences of error.

Uncertainty must be propagated to the decision

Credibility requires more than showing that input values are uncertain. The model should demonstrate how parameter, structural, methodological, and heterogeneity uncertainty affect the decision. A stable mean result can still be untrustworthy if plausible structural alternatives reverse the conclusion.

• Deterministic sensitivity analysis identifies influential inputs and assumptions.
• Probabilistic sensitivity analysis propagates joint parameter uncertainty.
• Scenario analysis tests alternative structures, extrapolations, perspectives, and implementation choices.
• Subgroup analysis examines heterogeneity relevant to decisions and equity.
• Threshold analysis identifies the value at which the preferred option changes.
• Value-of-information analysis estimates the consequence of remaining uncertainty and the potential value of research.

Decision robustness is not model validity

A result is robust when the preferred decision remains unchanged across a justified range of assumptions. Robustness can increase confidence in the decision even when individual parameters are uncertain. It cannot rescue a model with a fundamentally inappropriate structure, biased evidence, or programming error.

For a cost-effectiveness model, incremental net monetary benefit is:

$$ INMB = \lambda\Delta E - \Delta C $$

where (\lambda) is the decision maker's value per unit of health, (\Delta E) is incremental health, and (\Delta C) is incremental cost. Credibility assessment examines whether the sign and magnitude of INMB remain defensible across uncertainties and whether the model producing those quantities is fit for purpose.

Transparency makes scrutiny possible

Reviewers need enough information to understand, reproduce, and challenge the model. Documentation should connect the decision problem to the conceptual model, equations, data, assumptions, code, results, and limitations. Confidential inputs can sometimes be protected, but they should not prevent meaningful validation or hide assumptions that determine the conclusion.

A transparent package should include:

• A plain-language description and technical specification.
• A diagram of model states, events, pathways, and data flows.
• Parameter tables with definitions, values, distributions, sources, and transformations.
• Search and selection methods for key evidence.
• Calibration targets, methods, diagnostics, and fitted parameters.
• Verification and validation plans, tests, results, failures, and corrections.
• Executable files or sufficient access for independent review.
• Version history, change log, conflicts of interest, and funding.

Independent review reduces avoidable bias

Model developers possess essential knowledge but can also overlook familiar assumptions or implementation errors. Independent clinical, methodological, economic, statistical, and technical review adds different forms of challenge. Independence should be described rather than assumed from an organisation's name.

The review record should state who reviewed each component, their expertise, access to model files and data, financial or intellectual conflicts, findings, developer responses, and unresolved disagreements. Recalculation or code inspection provides stronger assurance than review of a summary report alone.

Reproducibility and version control protect credibility

A credible result must be traceable to the exact model, inputs, software environment, and run settings that produced it. Uncontrolled spreadsheet edits, undocumented overrides, broken links, and stale external data can make a reviewed model different from the one used for the decision. Version control and reproducible execution reduce this risk.

The release record should preserve:

• Model version, date, commit or file identifier, and responsible owner.
• Input datasets and their extraction dates.
• Software, packages, settings, random seeds, and calculation mode.
• Approved overrides and scenario selections.
• Quality-assurance results and known issues.
• Output files linked to the exact execution record.

Credibility can vary across outputs

One part of a model may be well supported while another relies on speculative extrapolation. For example, short-term event rates may validate well while lifetime survival, quality of life, or resource use remains uncertain. The assessment should therefore rate or describe confidence by material output and assumption rather than issuing an undifferentiated label.

This distinction is especially important when different outputs serve different decisions. A model may support short-term budget planning but not lifetime cost-effectiveness, or population averages but not subgroup allocation.

A structured assessment should not become a mechanical score

Checklists improve completeness and comparability, but a total score can obscure fatal flaws and give equal weight to issues with very different consequences. A missing external validation dataset may be unavoidable, while an incorrect comparator or coding error can invalidate the decision. The final judgment should identify critical issues, material limitations, and their effect on use.

A useful classification separates:

• Critical issues that make the model unsuitable for the intended decision.
• Major issues that could change the conclusion and require correction or explicit scenario analysis.
• Moderate issues that reduce confidence but are unlikely to reverse the decision alone.
• Minor issues that should be documented or corrected without materially affecting results.

Proportionality links scrutiny to decision risk

The depth of assessment should reflect the model's influence and the harm that could follow an incorrect conclusion. A high-stakes national coverage decision usually requires more independent validation than an early exploratory analysis. Proportionality does not permit basic verification, transparency, or conflict disclosure to be skipped.

Factors increasing the required scrutiny include large affected populations, irreversible decisions, substantial expenditure, narrow decision margins, major structural uncertainty, long extrapolation, limited direct evidence, vulnerable groups, and strong sponsor incentives.

Worked credibility judgment

Suppose a lifetime model of a new cancer treatment accurately reproduces trial survival for three years, passes independent code checks, and provides transparent parameter sources. Beyond three years, survival is extrapolated from immature data, alternative parametric curves produce different cost-effectiveness conclusions, and no external long-term dataset is available. The model may be credible for describing trial-period outcomes but only conditionally credible for a lifetime reimbursement decision.

The assessment should not average the strengths and weaknesses into a reassuring score. It should identify long-term extrapolation as a major decision-critical uncertainty, present alternative curves, quantify the effect on INMB, and consider whether additional follow-up or conditional coverage could reduce the risk of error.

Common mistakes

Credibility is often overstated when one successful test is treated as proof of the whole model. It can also be understated when unavoidable uncertainty is mistaken for poor modelling despite transparent and decision-robust analysis. The assessment should distinguish evidence gaps from defects and relate both to the decision.

• Expert face validity does not replace empirical validation.
• Fit to calibration targets is not independent validation.
• Reproducing published results does not verify undocumented code pathways.
• A probabilistic analysis does not capture structural uncertainty automatically.
• Agreement between two models is weak evidence when they share data and assumptions.
• Transparent documentation does not make biased evidence credible.
• A robust decision does not prove that the model is structurally valid.
• A checklist total should not outweigh a single critical flaw.
• Credibility in one jurisdiction, population, or time period does not guarantee transportability to another.

Reporting the final assessment

The report should make the chain from evidence to judgment visible. It should identify what was examined, what was not accessible, which tests passed or failed, how developers responded, and how each unresolved issue affects the decision. Conditional language is preferable to a blanket statement that a model is valid.

• State the intended use and required credibility standard.
• Summarise conceptual, evidence, verification, validation, uncertainty, transparency, and independence findings separately.
• Link every major claim to a documented test, source, or review record.
• Classify unresolved issues by materiality and decision consequence.
• State which outputs and uses are supported, conditionally supported, or unsupported.
• Recommend corrections, additional analyses, evidence collection, or restrictions on use.
• Record reviewer names, conflicts, access, date, model version, and reassessment triggers.

The decision standard

A model is credible when the total evidence provides justified confidence that it is fit to inform its intended decision, with limitations made explicit and uncertainty carried through to the result. Credibility is stronger when the conceptual model is appropriate, implementation is verified, outputs withstand relevant validation, evidence is applicable, and independent scrutiny is possible. The conclusion remains conditional, version-specific, and open to revision when the model, evidence, or decision context changes.