Uncertainty

Uncertainty in Health Economic Evaluation

In health economic evaluation, uncertainty means that the costs and health outcomes a model predicts for each option cannot be known exactly, so a decision based on them may turn out to be the wrong one. It has several distinct sources, from chance events in individual patients to biased evidence and contested model structures, and each calls for a different kind of analysis. The second report of the Health Economics Methods Advisory (HEMA), convened by the Institute for Clinical and Economic Review, Canada's Drug Agency and NICE, recommends keeping uncertainty analysis focused on decision uncertainty, its drivers and its consequences. The analysis runs within the health economic model, and the NICE technology appraisal manual (PMG36) sets out how submissions should explore it.

Why uncertainty matters for a resource allocation decision

The payoffs in a cost-effectiveness analysis, usually net monetary benefit or net health benefit, rest on imperfect and incomplete evidence. The ISPOR value of information task force calls the resulting risk of a wrong decision decision uncertainty, and notes that with a fixed budget suboptimal decisions result in health loss. The HEMA report separates two errors, funding an intervention whose benefit proves smaller than the health its costs displace, and refusing one that would have increased population health, and asks for consequences to be weighed in both directions.

A risk-neutral decision maker still chooses the option with the highest expected net benefit, so uncertainty matters through the other actions open to the decision maker. The ISPOR-SMDM task force on parameter estimation argues that a decision maker who must decide now and cannot commission research, delay or review should rely on expected values. Bodies that can delay, review or mandate research also need the value of further research and the cost of reversing a decision that proves suboptimal. The HEMA report adds that resting an appraisal on one most plausible set of assumptions effectively sets the likelihood that any other set is correct to zero.

The HEMA report doubts the case for making decision makers risk averse, which would penalise evidential uncertainty and lower the acceptable price, finding it hard to see how a risk aversion factor could be conceptualised and quantified. It treats risk aversion in patients or the public about health as a separate matter and does not propose adjusting the analysis for them.

Six concepts in the uncertainty typology

The ISPOR-SMDM report distinguishes stochastic (first-order) uncertainty from parameter (second-order) uncertainty and from heterogeneity, and treats the structure of the model as a further level of uncertainty. It ties each to part of a regression model. The table adds methodological uncertainty, discussed in the report's text, and decision uncertainty, a consequence of the others.

Parameter uncertainty arises because the probabilities, costs and utilities in a model are estimated from limited and sometimes conflicting data. Structural uncertainty concerns assumptions rarely expressed as numbers, such as the duration of treatment effects. The task force observes that it is frequently ignored although it may affect results more than parameter uncertainty.

The task force notes that methodological uncertainty can be as important as parameter uncertainty and favours a reference case. The value of information task force excludes methodological uncertainty and heterogeneity, since research cannot reduce the first and the second does not reflect uncertainty about the truth.

Classifying uncertainty by its source in the evidence

The HEMA report classifies uncertainty by its source, namely sampling uncertainty, bias, limited relevance to the decision setting, extrapolation beyond observed data, evidence gaps that force assumptions, and choices about model structure. Sampling uncertainty affects precision while bias affects validity, so a precise estimate can still be a poor basis for a decision. The report recommends correcting coding errors and avoidable methodological shortcomings rather than carrying them forward as uncertainty through conservative assumptions or general caution, and affordability and budget impact are not uncertainty about cost-effectiveness.

The report recommends documenting important uncertainties before methods or decision options are chosen, for example in an uncertainty assessment table. For each one the table notes its source, what it affects, its likely direction and size, how it is represented and whether further research could reduce it, which helps avoid counting one evidence limitation twice.

Uncertainty, variability and heterogeneity

The word variability is used both for stochastic uncertainty and for differences in parameter values between patients or subgroups, which the ISPOR-SMDM report identifies as a source of confusion. It compares the difference between stochastic and parameter uncertainty with that between a standard deviation, which describes how individual observations vary, and a standard error, which describes the precision of an estimated quantity. More data shrink the standard error of a mean but leave the spread between patients in place, so further research can reduce parameter uncertainty but cannot remove variability.

Heterogeneity is the part of between-patient variability that patient characteristics such as age and sex can explain, which matters for identifying subgroups that need separate cost-effectiveness analyses. The HEMA report treats restricting a decision to a subgroup as exploiting heterogeneity rather than analysing uncertainty, and asks reports to say whether observed variation reflects heterogeneity, uncertainty or both. Stochastic uncertainty matters mainly in patient-level simulation such as microsimulation, where it must be eliminated (it is sometimes called Monte Carlo error) before parameter uncertainty can be assessed.

Tools for analysing uncertainty

The sensitivity analysis page compares the tools in detail. The HEMA report says the probabilistic approach can more accurately be called uncertainty analysis, since it reflects the combined uncertainty of all parameters, and gives deterministic sensitivity analysis the narrower role of understanding model behaviour and finding influential inputs, since it gives no overall measure of decision uncertainty. Scenario analysis handles structural and data-source uncertainty. Probabilistic sensitivity analysis samples (preferably) all uncertain inputs together, and its results appear on the cost-effectiveness plane and the cost-effectiveness acceptability curve. Value of information analysis then values reducing decision uncertainty, and the EVPI equals the expected cost of uncertainty in the current decision.

What the NICE manual (PMG36) expects

Section 4.7 of PMG36 (last updated 13 October 2026) asks submissions for an overall assessment of the relative effect of different types of uncertainty, highlighting those unlikely to be reduced by further evidence or expert input. The model should quantify decision uncertainty, defined as the probability that a different decision would be reached if the true cost effectiveness of each technology could be known before deciding.

Structural uncertainty is explored through separate analyses of plausible scenarios (4.7.4), and uncertainty about reference-case methods is presented separately (4.7.7). The committee's preferred estimate should come from a probabilistic analysis when possible unless the model is linear (4.7.12). Results should include acceptability curves with the acceptability frontier and the probability that the treatment is cost effective at maximum acceptable ICERs of GBP 25,000 to GBP 35,000 per QALY gained, with the error probability (4.7.15).

The committee takes account of the likelihood of decision error and its consequences for patients and the NHS (6.2.32), and is normally more cautious when less certain about the evidence (6.2.33). When highly uncertain evidence leads to a high degree of decision uncertainty, it may consider managed access (medicines only), data collection or research (6.2.30). Managed access requires significant resolvable uncertainty, a plausible potential to be cost effective at the agreed price and data that could feasibly be collected within a maximum of 5 years (6.4.6).

Reporting recommendations in the HEMA report

The HEMA report recommends reporting expected incremental net health benefit alongside the ICER at individual and population level, and using PSA to combine as many sources of uncertainty as possible while stating which remain outside it. It recommends showing acceptability curves or the acceptability frontier with at least the population EVPI. The report proposes three consistent layers of reporting, a technical report, a committee decision brief and a plain-language summary, and recommends looking beyond yes or no to price, access while evidence is collected, or access only in research.

Worked example with one spread of results and three meanings

All numbers are illustrative. A new treatment adds GBP 10,000 per patient to costs, the threshold is GBP 25,000 per QALY, and the incremental QALY gain takes one of three values, 0.2, 0.5 or 0.8, each with probability one third. Incremental net monetary benefit is defined as follows.

$$\text{INMB} = \lambda \, \Delta E - \Delta C$$

where $\lambda$ is the threshold (GBP per QALY), $\Delta E$ is the incremental QALY gain and $\Delta C$ is the incremental cost (GBP).

1. Net benefit for each value. With a gain of 0.2, INMB = 25,000 × 0.2 − 10,000 = −5,000. With 0.5 it is 12,500 − 10,000 = 2,500, and with 0.8 it is 20,000 − 10,000 = 10,000 (all GBP per patient).

2. Expected result. Expected INMB = (−5,000 + 2,500 + 10,000) / 3 = 2,500, so the new treatment is preferred; at the mean gain of 0.5 the ICER is 10,000 / 0.5 = GBP 20,000 per QALY. The switching value of the gain is 10,000 / 25,000 = 0.4 QALYs.

3. Read as parameter uncertainty. If the three values are possible true population means, the probability that the treatment is cost-effective is 2/3 and the error probability is 1/3. With perfect information the better option would be chosen in each case, giving (0 + 2,500 + 10,000) / 3 = 4,166.67, so EVPI per patient = 4,166.67 − 2,500 = 1,666.67, the same as the expected loss of 1/3 × 5,000. Research is worth considering only if the population EVPI exceeds its cost.

4. Read as heterogeneity. If the values belong to three equal-sized subgroups that can be identified in advance, there is no decision uncertainty. The subgroup ICERs are GBP 50,000, 20,000 and 12,500 per QALY, and treating only the last two raises average net benefit from 2,500 to 4,166.67 per patient. The same 1,666.67 is gained now, through a subgroup recommendation rather than research.

5. Read as stochastic uncertainty. If the mean gain is known to be 0.5 and the spread reflects chance differences between identical patients, nobody can be picked out in advance. Decision uncertainty and EVPI are both zero, and adopting the treatment yields the expected 2,500 per patient.

6. Interpretation. The same spread of outcomes points to research, stratification or no further action depending on its source, so an analysis should name the kind of uncertainty it represents.

Common misunderstandings

Several errors recur in models and in their appraisal. Most come from mixing up the kinds of uncertainty described above.

• Treating uncertainty as automatic rejection. A wide interval leaves the highest expected net benefit unchanged; NICE weighs it through caution, managed access or research (6.2.30, 6.2.33).
• Assuming a probabilistic analysis captures all uncertainty. Its results are conditional on the model structure, the reference-case methods and the uncertainties included.
• Leaving out poorly informed parameters. Excluding a parameter for lack of data, the value of information task force notes, is equivalent to assuming it is known with certainty.

Sources

• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722-732. https://doi.org/10.1177/0272989X12458348
• Fenwick E, Steuten L, Knies S, Ghabri S, Basu A, Murray JF, Koffijberg H, Strong M, Sanders Schmidler GD, Rothery C. Value of information analysis for research decisions: an introduction. Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force. Value in Health. 2020;23(2):139-150. https://doi.org/10.1016/j.jval.2020.01.001
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.7, 6.2 and 6.4. https://www.nice.org.uk/process/pmg36
• Steuten L, Thavorn K, Cipriano LE, Guertin JR, McQueen B, Schmidler G, Sullivan S, Wailoo A, Sculpher M. Methods to Analyze and Present Uncertainty in Economic Evaluation to Support HTA Decisions. HEMA Report 2. Health Economics Methods Advisory; 2026. Summaries and sections 1.2, 1.5, 2.2, 2.4, 3.1, 3.3 and 3.5. https://hemamethods.org/wp-content/uploads/2026/10/HEMA-Report-2_For-Publication_100726.pdf