Analysis Error

Analysis Error in Economic Models and Trial Analyses

An analysis error is a mistake made somewhere between the evidence and the reported result, such as a wrong formula in a cost-effectiveness model, an input entered in the wrong units, a miscoded treatment variable or a figure copied incorrectly into a submission. Unlike uncertainty, an error has a right answer that the analysis failed to reach. In health economics the consequences are concrete. Tappenden and Chilcott note that a single error or inappropriate judgement within a model may lead to inappropriate decisions and an inefficient allocation of healthcare resources. A single misplaced cost can nearly double an ICER, as the illustrative case further down shows.

Error, bias and uncertainty are different problems

An analysis error is a departure from the analysis that was intended or appropriate. Sampling variation and parameter uncertainty remain even when every step is done correctly, and bias is a systematic gap between the expected estimate and the target, which an error can cause but which can also arise from design or selection.

Tappenden and Chilcott divide threats to the credibility of health economic models into unequivocal errors, violations and matters of judgement, distinguished by whether criteria for judging correctness exist and how forcefully they can be applied. A disagreement between two defensible choices is a matter of judgement, better explored as structural uncertainty than labelled an error. In a qualitative study by Chilcott and colleagues of 12 health technology assessment (HTA) modellers, there was no common language for modelling errors, and interviewees tended to exclude matters of judgement and focus on slips and lapses, although slips and lapses made up less than 20% of the discussion of error types.

Where errors enter an economic evaluation

Interviewees in the same study gave examples of every major error type identified in the literature. These were errors in describing the decision problem, in model structure, in the use of evidence, in implementing the model, in operating it, and in presenting and understanding its results. Implementation errors are the most mechanical. In a health economic model they include the following examples.

• a one-off cost, such as treatment initiation, applied in every cycle;
• an adverse event costed twice, once within a state cost and again as a separate event cost;
• discounting at the wrong rate or from the wrong time point;
• a relative treatment effect applied to the wrong baseline risk;
• people in the dead state still accruing costs or quality-adjusted life years (QALYs);
• an ICER calculated against the wrong comparator in a fully incremental analysis, or dominated options left in the ranking;
• standard deviations used in place of standard errors when sampling parameters for a probabilistic sensitivity analysis.

Unit and time-scale errors

A correct formula applied to inconsistent units gives a wrong answer. Rates and probabilities are a common trap. Under a constant rate, the probability of an event within a cycle is given below.

$$p = 1 - e^{-r \Delta t}$$

where $p$ is the cycle transition probability, $r$ is the event rate per year and $\Delta t$ is the cycle length in years. With an illustrative rate of 0.12 per year and monthly cycles, the correct probability is $p = 1 - e^{-0.01} = 0.00995$. Setting $\Delta t = 1$ by mistake gives $p = 1 - e^{-0.12} = 0.1131$ per month, more than 11 times too high; over twelve such cycles about 76% of the cohort would have the event, against 11.3% in a year under the correct conversion. In a NICE appraisal of pomalidomide, the Evidence Review Group found that the company had converted an annual resource use estimate into a weekly probability instead of dividing it by 52, after the company's figure differed from the group's own rough calculation. Mixing price years or currencies without inflating and converting costs to a common base is a similar trap.

How often errors occur

Panko reports that people are 95% to 98% accurate when they create formulas. Errors compound across a workbook. If each of $N$ distinct formulas is correct with probability $a$, independently, the chance of at least one error is given below.

$$P(\text{at least one error}) = 1 - a^{N} = 1 - 0.98^{100} = 0.867$$

where $a = 0.98$ is formula-level accuracy and $N = 100$ root formulas. Detection is weaker than construction. Individual inspectors find 50% to 60% of seeded errors, and team inspection can raise this to about 80%.

Hill and colleagues reviewed 326 submissions to the Australian Pharmaceutical Benefits Scheme received between 1994 and 1997. Of these, 218 (67%) had significant problems; among the 249 problems identified, 154 (62%) concerned uncertainty about comparative clinical efficacy, 71 (28.5%) related to modelling, serious calculation errors were found on 9 occasions, and 159 (64%) were judged avoidable. Reporting of checks remains thin, and a 2025 review of 78 published early breast cancer models found that validation of the computerised model was reported in fewer than 4% of studies.

Errors in the statistical analysis of trial data

Trial-based economic evaluations add statistical pitfalls. Barber and Thompson reviewed 45 randomised trials published in 1995 with patient-level costs and found that only 9 (20%) reported adequate measures of variability, 25 (56%) gave a test or measure of precision for the cost comparison, and 16 (36%) drew conclusions justified by the results presented. Leurent and colleagues reviewed 52 trial-based cost-effectiveness analyses published in the NIHR HTA journal from 2013 to 2015 and found that complete-case analysis remained the most common way of handling missing data, contrary to recommendations favouring multiple imputation. Only two studies (4%) assessed sensitivity to departures from the missing-at-random assumption comprehensively, and half did no missing-data sensitivity analysis.

Some errors are logical. A treatment coefficient changes sign when the reference category changes, so reporting it against the wrong reference reverses the conclusion. Testing many outcomes or subgroups inflates false positives. With 20 independent tests of true null hypotheses at the 5% level, the chance of at least one type I error is $P = 1 - 0.95^{20} = 0.64$, which is why post hoc subgroup analysis findings need cautious interpretation.

Quality assurance and validation

Quality assurance relies on checks whose expected answers are known before the model is run. The ISPOR-SMDM task force report by Eddy and colleagues lists verification, or internal validity, as checking the accuracy of coding, alongside face, cross, external and predictive validity. Interviewees in the Chilcott study used verification to mean that the computer model correctly implements the intended model, and validation to mean that the model is fit for purpose; model validation has its own page.

• AdViSHE, developed through a Delphi panel of 47 experts, has 13 items covering the conceptual model, input data, the implemented software and model outcomes. Developers report the validation done and its results, so users can judge whether more is needed.
• TECH-VER structures technical verification into five domains, which are input calculations, event-state (patient flow) calculations, result calculations, uncertainty analysis calculations and overall checks. Tests run in order, starting with black-box tests against prior expectations, then white-box review line by line or cell by cell, then replication or parallel programming if anomalies remain.
• Black-box checks from TECH-VER include state membership that does not sum to the cohort size, total life years exceeding total QALYs, undiscounted results exceeding discounted results, and ICERs in a fully incremental analysis that never decrease.

The NICE manual (PMG36) asks companies to detail the quality assurance methods used in developing the model, to provide the methods and results of model validation, and to present results in a disaggregated format. Companies submitting a fully executable model must give NICE full access to the programming code and ensure that the model and the written submission match. The External Assessment Group critically evaluates the submission and may amend the company's model in exploratory analyses. PMG36 also asks for enough simulations to minimise Monte Carlo error in a probabilistic sensitivity analysis.

Worked example of one cost charged in every cycle (illustrative)

A five-year cohort model with annual cycles compares a new treatment with standard care, discounting at an illustrative 3.5% a year and assuming everyone survives the five years. Correctly built, the discounted results are an incremental cost of GBP 6,000 and 0.30 incremental QALYs, so the ICER is GBP 20,000 per QALY. A one-off initiation cost of GBP 1,500 for the new treatment, due only at the start of cycle 0, is placed in the per-cycle cost row and charged again at the start of cycles 1 to 4 as well, which adds the discounted error shown below.

$$\Delta C_{\text{err}} = 1500 \times \left(\frac{1}{1.035} + \frac{1}{1.035^{2}} + \frac{1}{1.035^{3}} + \frac{1}{1.035^{4}}\right) = 1500 \times 3.6731 = 5509.6$$

where $\Delta C_{\text{err}}$ is the discounted cost added in error and each term $\frac{1}{1.035^{t}}$ is the discount factor for a cost paid at the start of cycle $t$. The erroneous incremental cost is 6,000 + 5,509.6 = 11,509.6, and the erroneous ICER is 11,509.6 / 0.30 = 38,365, or GBP 38,365 per QALY.

At an illustrative threshold of GBP 30,000 per QALY, incremental net monetary benefit falls from 30,000 × 0.30 − 6,000 = 3,000 to 9,000 − 11,509.6 = −2,509.6 GBP. One misplaced cell turns a cost-effective treatment into one that appears not to be. A disaggregated cost table would expose it, because the undiscounted initiation cost per person would read GBP 7,500 against an input of GBP 1,500; this is the kind of mismatch with prior expectations that TECH-VER's black-box tests look for.

Correcting an error and judging its effect

Finding an error calls for a controlled correction rather than a quiet patch, which means reproducing it, tracing every affected output, correcting and independently re-checking, and recording whether conclusions change. Sensitivity analysis is no substitute for verification, because a one-way sensitivity analysis varies inputs through the same logic, so it reveals an implementation error only when a result responds unexpectedly.

Several common misunderstandings follow from this.

• A plausible result is not evidence of a correct model; offsetting errors can produce one.
• Rerunning the same code tests reproducibility; independent replication tests the logic.
• Deleting extreme values without finding their cause can hide a unit error.
• Choosing between defensible alternatives is a matter of judgement to be tested in scenario analysis, and is not an error.

Sources

• Abraham K, Corro Ramos I, Braal CL, et al. Under-reporting of validation efforts for health economic models persists despite the availability of validation tools: a systematic review. PharmacoEconomics. 2025;43(8):849-858. https://doi.org/10.1007/s40273-025-01491-2
• Barber JA, Thompson SG. Analysis and interpretation of cost data in randomised controlled trials: review of published studies. BMJ. 1998;317(7167):1195-1200. https://doi.org/10.1136/bmj.317.7167.1195
• Büyükkaramikli NC, Rutten-van Mölken MPMH, Severens JL, Al M. TECH-VER: a verification checklist to reduce errors in models and improve their credibility. PharmacoEconomics. 2019;37(11):1391-1408. https://doi.org/10.1007/s40273-019-00844-y
• Chilcott J, Tappenden P, Rawdin A, et al. Avoiding and identifying errors in health technology assessment models: qualitative study and methodological review. Health Technology Assessment. 2010;14(25). https://doi.org/10.3310/hta14250
• Eddy DM, Hollingworth W, Caro JJ, et al. Model transparency and validation: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-7. Medical Decision Making. 2012;32(5):733-743. https://doi.org/10.1177/0272989X12454579
• Gidwani R, Russell LB. Estimating transition probabilities from published evidence: a tutorial for decision modelers. PharmacoEconomics. 2020;38(11):1153-1164. https://doi.org/10.1007/s40273-020-00937-z
• Hill SR, Mitchell AS, Henry DA. Problems with the interpretation of pharmacoeconomic analyses: a review of submissions to the Australian Pharmaceutical Benefits Scheme. JAMA. 2000;283(16):2116-2121. https://doi.org/10.1001/jama.283.16.2116
• Leurent B, Gomes M, Carpenter JR. Missing data in trial-based cost-effectiveness analysis: an incomplete journey. Health Economics. 2018;27(6):1024-1040. https://doi.org/10.1002/hec.3654
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. https://www.nice.org.uk/process/pmg36
• Panko RR. Thinking is bad: implications of human error research for spreadsheet research and practice. Proceedings of the European Spreadsheet Risks Interest Group (EuSpRIG). 2007:69-80. https://arxiv.org/abs/0801.3114
• Tappenden P, Chilcott JB. Avoiding and identifying errors and other threats to the credibility of health economic models. PharmacoEconomics. 2014;32(10):967-979. https://doi.org/10.1007/s40273-014-0186-2
• Vemer P, Corro Ramos I, van Voorn GAK, Al MJ, Feenstra TL. AdViSHE: a validation-assessment tool of health-economic models for decision makers and model users. PharmacoEconomics. 2016;34(4):349-361. https://doi.org/10.1007/s40273-015-0327-2