Structural Uncertainty

Structural Uncertainty in Health Economic Models

Structural uncertainty is doubt about the shape of a model itself, such as which health states it contains, how people move between them and how survival is projected beyond the trial, as opposed to doubt about the numbers fed into a settled model. Two defensible structures can give different costs, QALYs and decisions from the same evidence, which probabilistic sensitivity analysis run within one structure cannot show. The ISPOR-SMDM task force on parameter estimation and uncertainty observes that structural uncertainty is frequently ignored, although it may have a much greater impact on results than parameter uncertainty. The NICE technology appraisal manual (PMG36) asks for structural assumptions to be documented and their effect explored, and HEMA Report 2 from the Health Economics Methods Advisory recommends bringing more of this uncertainty into the probabilistic analysis instead of leaving it to scenarios alone.

Where structural uncertainty arises in a model

HEMA Report 2 describes uncertainty about model structure as concerning how the decision problem is represented, including health states and transitions, functional forms, care pathways, extrapolation methods and relationships between model components. The NICE manual gives two examples in section 4.7.3, how different states of health are categorised and how pathways of care are represented. A review by Bojke and colleagues grouped the sources into four themes, namely the inclusion of relevant comparators, the inclusion of relevant events, alternative statistical estimation methods, and clinical uncertainty.

Section 4.7.6 of the NICE manual names uncertainty about the right assumption for extrapolation beyond trial follow-up as a case for scenario analysis, and section 4.6.25 asks for the clinical plausibility of the hazard functions behind parametric, spline-based and cure model extrapolations to be assessed routinely.

Not every assumption is a source of uncertainty. HEMA Report 2 separates assumptions forced by limited evidence, such as the duration of a treatment effect or the handling of treatment switching and subsequent therapies, from assumptions that only simplify, such as using a general-population life table or average costs. The first kind becomes an important source of uncertainty when several plausible alternatives exist and the choice could materially affect the results. The second kind should be documented and justified but does not always need to be treated as uncertain. Where such a simplification hides decision-relevant heterogeneity, the report suggests modelling subgroups separately, as in a subgroup analysis.

How structural uncertainty differs from parameter uncertainty

The ISPOR-SMDM task force uses an analogy with regression. Parameter uncertainty corresponds to uncertainty in the estimated coefficients, while structural uncertainty corresponds to the form of the regression model itself and covers the assumptions inherent in the decision model. The task force concludes that any representation of uncertainty in a decision model is conditional on its structural assumptions.

The line between the two is partly a modelling choice. HEMA Report 2 notes that the waning of a treatment effect may be represented by a waning-rate parameter, and the choice between alternative survival curves by weights assigned to each. The task force adds that turning a structural question into a parameter is trivial for nested structures, where a constant hazard is replaced by a more flexible function, but much harder for non-nested structures that could require the model to be rebuilt. HEMA Report 2 asks analysts to state clearly what is uncertain, whether it has been parameterised and, if not, how it has been characterised.

Methods for characterising structural uncertainty

Four broad approaches are in use. They differ mainly in whether they attach any probability to the competing structures.

Scenario analysis across alternative structures

The NICE manual (section 4.7.4) asks for the effect of structural uncertainty to be explored using separate analyses of a representative range of plausible scenarios that are consistent with the evidence, with demonstrably implausible scenarios useful only to show that a conclusion is insensitive to a source of uncertainty. HEMA Report 2 calls scenario analysis the most common approach and treats it as a form of deterministic sensitivity analysis, since it shows whether the decision changes but not how likely each structure is to be correct.

HEMA Report 2 also asks for scenarios to be chosen before the results are seen and recommends that each be run probabilistically with at least the expected value of perfect information.

Parameterising the structural choice

Jackson and colleagues set out a framework in which the model is expanded to include parameters that encompass the possible structural choices. Where data exist, the uncertainty comes from imprecise estimates of those parameters; where none exist, the parameters can often be informed by eliciting expert beliefs as probability distributions. Bojke and colleagues concluded that only parameterising the uncertainty directly in the model can inform a decision to conduct further research to resolve it. The ISPOR-SMDM task force recommends testing structural uncertainties identified while building a model, with consideration given to parameterising them.

Model averaging

Model averaging combines several structures using weights that express how far each is supported. In the approach of Jackson, Thompson and Sharples, the weights come from each model's adequacy judged against data, using the Akaike information criterion or the Bayesian information criterion, and probabilistic draws from each model are pooled in proportion to its weight. Where data cannot judge between structures, the weights can come from prior beliefs instead. The NICE manual (section 4.7.5) notes that structural uncertainty may be incorporated in this way, with the alternatives, weights and any expert advice transparently documented.

The weighting rule is itself a judgement. In the authors' study of endovascular aneurysm repair, the probability of cost-effectiveness at GBP 20,000 per QALY was 0.011 in the base case, 0.069 when all eight combinations of model assumptions were averaged with AIC weights, and 0.35 with BIC weights. HEMA Report 2 prefers model averaging where the alternatives can be enumerated and weighted, because it returns a single distribution instead of separate results that still have to be combined by judgement.

Model discrepancy and comparison of independent models

Strong, Oakley and Chilcott define model discrepancy as the difference between the model evaluated at its true inputs and the true costs and health effects. Their method decomposes a cost-effectiveness model into subfunctions, considers potential error at each, and uses a variance-based sensitivity analysis to locate the sources of discrepancy that most need refinement. HEMA Report 2 also describes running independently built models against the same question, as in the CISNET cancer models, to show whether defensible structures reach the same policy conclusion.

Worked example with two survival extrapolations

The numbers are illustrative. A new cancer treatment is compared with standard care in a partitioned survival model, and two extrapolations fit the trial data about equally well. Structure A uses a cure model in which part of the cohort has long-term survival; structure B uses a parametric curve with no plateau. The decision is assessed with net health benefit at GBP 25,000 per QALY, the lower of the two values the NICE manual uses for net health benefit.

$$NHB_s = \Delta Q_s - \frac{\Delta C_s}{k}$$

where $NHB_s$ is the incremental net health benefit per patient under structure $s$ in QALYs, $\Delta Q_s$ is the incremental QALYs, $\Delta C_s$ is the incremental cost in GBP and $k$ is GBP 25,000 per QALY.

Under structure A the cost converts to 30,000/25,000 = 1.20 QALYs of health forgone, so the net health benefit is $NHB_A = 1.40 - 1.20 = 0.20$ QALYs. Under structure B it is $NHB_B = 0.70 - 1.04 = -0.34$, since 26,000/25,000 = 1.04. Structure B's ICER also lies above the higher NICE value of GBP 35,000 per QALY. A scenario analysis therefore shows that the decision reverses with the structure but gives no basis for choosing between them.

If structure A is given weight $w$ and structure B the remaining weight, the averaged result is a weighted mean.

$$\overline{NHB} = w(0.20) + (1 - w)(-0.34) = 0.54w - 0.34$$

where $\overline{NHB}$ is the model-averaged net health benefit and $w$ is the weight on structure A. With equal weights the result is $\overline{NHB} = 0.54 \times 0.5 - 0.34 = -0.07$, so the treatment would be rejected. With a weight of 0.7 it is $\overline{NHB} = 0.54 \times 0.7 - 0.34 = 0.038$, so it would be adopted. The decision switches at a weight of 0.34/0.54 = 0.63, so the recommendation rests on a judgement about how credible the cure assumption is, and equal weights are no more neutral than any other choice.

The example also prices the structural question. With a weight of 0.7 and adoption, the expected net health benefit is 0.038 QALYs per patient. If the true structure were known first, the treatment would be adopted only when structure A holds, giving 0.7 × 0.20 + 0.3 × 0 = 0.14. The difference of 0.102 QALYs per patient (0.14 less 0.038), equal to 0.3 × 0.34, is the expected loss from structural uncertainty alone, with parameter uncertainty ignored for simplicity. Summed over the eligible population, it sets an upper limit on the value of research able to discriminate between the two curves, such as longer trial follow-up.

Interpretation and decision use in HTA

The NICE manual asks for an overall assessment describing the relative effect of different types of uncertainty, such as parameter and structural, on the cost-effectiveness estimates (section 4.7.1). It requires the model type and structure to be justified for each new decision problem, stating that prior use in published models or earlier submissions is not enough (section 4.6.3).

HEMA Report 2 notes that decision makers in practice often weigh a few selected, frequently structural, uncertainties more heavily than the integrated probabilistic analysis. Full integration of every characterised uncertainty, including the choice of survival model and the waning of effectiveness, into one probabilistic analysis is the ideal, but integration is usually partial. For that common case the report recommends stating which sources of uncertainty were left out, because what is missing bears on which option is indicated. Structural uncertainty matters to a committee when it becomes decision uncertainty, and comparing the decision across structures is the direct test of that.

Errors in handling structural uncertainty

The same mistakes recur when structural uncertainty is handled informally. Most treat one structure, or one summary, as if it settled the question.

• Treating the PSA as complete. A probabilistic analysis on one structure can look reassuring when another credible structure points the other way.
• Choosing the best-fitting model and stopping there. Jackson, Thompson and Sharples note that basing a decision purely on the best-fitting model implies certainty that this model, and no other, is reasonable.
• Reading a scenario range as a probability interval. The spread between structural scenarios has no probability attached, and extreme scenarios can make uncertainty look larger or smaller than the evidence supports.
• Averaging away a reversal. A single averaged result can hide a decision that flips between structures, so the structure-specific results remain informative alongside the average.
• Assuming the tested structures are all the structures. Agreement across the alternatives examined depends on having identified the important ones, which is why NICE section 4.6.3 asks for a transparent conceptual model and details of expert involvement in choosing the structure.

Sources

• Bojke L, Claxton K, Sculpher M, Palmer S. Characterizing structural uncertainty in decision analytic models: a review and application of methods. Value in Health. 2009;12(5):739-749. https://doi.org/10.1111/j.1524-4733.2008.00502.x
• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722-732. https://doi.org/10.1177/0272989X12458348
• Jackson CH, Thompson SG, Sharples LD. Accounting for uncertainty in health economic decision models by using model averaging. Journal of the Royal Statistical Society Series A. 2009;172(2):383-404. https://doi.org/10.1111/j.1467-985X.2008.00573.x
• Jackson CH, Bojke L, Thompson SG, Claxton K, Sharples LD. A framework for addressing structural uncertainty in decision models. Medical Decision Making. 2011;31(4):662-674. https://doi.org/10.1177/0272989X11406986
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.2.16, 4.6.3, 4.6.25, 4.7.1 and 4.7.3 to 4.7.6. https://www.nice.org.uk/process/pmg36
• Steuten L, Thavorn K, Cipriano LE, Guertin JR, McQueen B, Schmidler G, Sullivan S, Wailoo A, Sculpher M. Methods to Analyze and Present Uncertainty in Economic Evaluation to Support HTA Decisions. HEMA Report 2. Health Economics Methods Advisory; 2026. Sections 3.4 and 4.3 to 4.7. https://hemamethods.org/wp-content/uploads/2026/10/HEMA-Report-2_For-Publication_100726.pdf
• Strong M, Oakley JE, Chilcott J. Managing structural uncertainty in health economic decision models: a discrepancy approach. Journal of the Royal Statistical Society Series C. 2012;61(1):25-45. https://doi.org/10.1111/j.1467-9876.2011.01014.x