Model Averaging

Model Averaging in Health Economic Models

Model averaging gives each plausible candidate model a weight and pools their predictions, so doubt about which model is right reaches the costs, QALYs and decision uncertainty of an economic evaluation. A typical use in health technology assessment is survival extrapolation, where several parametric curves fit the trial data about equally well yet imply very different lifetime survival. Basing the decision on the single best-fitting curve implies certainty that this curve, and no other, is reasonable, so the probabilistic results look more precise than the evidence allows. Model averaging is one of several ways of handling structural uncertainty, and the NICE technology appraisal manual (PMG36) names it as a way of bringing that uncertainty inside a probabilistic model.

Averaging distributions across candidate models

Hoeting and colleagues set out Bayesian model averaging in its standard form. The quantity of interest, such as mean survival or incremental net benefit, has a posterior distribution under each candidate model, and the averaged distribution is the mixture of these, each weighted by the posterior probability of its model.

$$p(\Delta \mid D) = \sum_{k=1}^{K} p(\Delta \mid M_k, D)\, p(M_k \mid D)$$

where $\Delta$ is the quantity of interest, $D$ is the observed data, $M_1$ to $M_K$ are the candidate models, $p(\Delta \mid M_k, D)$ is the posterior distribution of $\Delta$ under model $k$ and $p(M_k \mid D)$ is the posterior probability of model $k$.

$$p(M_k \mid D) = \frac{p(D \mid M_k)\, p(M_k)}{\sum_{l=1}^{K} p(D \mid M_l)\, p(M_l)}$$

where $p(D \mid M_k)$ is the integrated likelihood of model $k$, that is its likelihood averaged over the prior distribution of its own parameters, and $p(M_k)$ is the prior probability of model $k$. Hoeting and colleagues stress that all these probabilities are conditional on the set of models considered, and regard equal prior probabilities as a reasonable neutral choice when little is known.

What is averaged is a model output, never the parameters, since a Weibull shape and a log-normal scale have no common meaning. An ICER is then formed from the averaged incremental costs and QALYs, not by averaging model-specific ratios.

Mean and variance of the averaged result

The averaged mean is the weighted mean of the model-specific means. Hoeting and colleagues give the variance in closed form, and it shows where the extra uncertainty comes from.

$$\mathrm{Var}(\Delta \mid D) = \sum_{k=1}^{K} \left[ \mathrm{Var}(\Delta \mid D, M_k) + \hat{\Delta}_k^{2} \right] p(M_k \mid D) - E(\Delta \mid D)^{2}$$

where $\hat{\Delta}_k$ is the posterior mean of $\Delta$ under model $k$, $\mathrm{Var}(\Delta \mid D, M_k)$ is its variance under that model and $E(\Delta \mid D)$ is the averaged mean, equal to the sum of each $\hat{\Delta}_k$ times its model probability. Rearranged, the variance is the weighted mean of the within-model variances plus the weighted spread of the model means around the averaged mean, and the second part is zero only when the models agree. Jackson, Thompson and Sharples note that averaging models with different predictions generally gives greater uncertainty than the model-specific results.

Where the weights come from

The method stands or falls on its weights, which come from three broad sources. They differ in what they reward and in whether they claim that one candidate is the true model.

Information criterion weights

Information criteria give weights directly from fitted likelihoods. Jackson, Thompson and Sharples write the adequacy of model $k$ as the exponential of minus half its criterion, so only differences between models matter.

$$w_k = \frac{\exp(-\delta_k / 2)}{\sum_{r=1}^{K} \exp(-\delta_r / 2)}$$

where $w_k$ is the weight of model $k$, $\delta_k$ is its Akaike information criterion or Bayesian information criterion value minus the smallest value in the set, and $K$ is the number of candidate models. With AIC the results are known as Akaike weights. BIC approximates minus twice the logarithm of the integrated likelihood, so BIC weights approximate the posterior model probabilities above with equal prior probabilities.

The two criteria can disagree sharply. In the authors' abdominal aortic aneurysm model, a treatment effect on cardiovascular deaths had a hazard ratio of 3.06 (95% interval 1.12 to 8.36), p = 0.029, from 1,016 trial patients. AIC averaging gave the model with that effect a probability of 0.801 and BIC averaging only 0.256. The authors argue that a predictive measure such as AIC suits health economic models, built to predict a complex reality, and that BIC suits settings where a simple true model is believed to generate the data.

Predictive and judgement-based weights

In a later fully Bayesian analysis of implantable cardioverter defibrillators, Jackson, Sharples and Thompson derived weights from the pseudo-marginal likelihood and the deviance information criterion, without treating them as probabilities that any model is true. Stacking, also described in their 2009 paper, chooses the weights to minimise cross-validatory squared error, as in cross-validation. Where a choice cannot be judged against data, such as which of two sources of 30-day mortality represents the target population, the model probabilities can come purely from elicited expert beliefs.

Running model averaging inside a probabilistic sensitivity analysis

Averaging fits naturally into probabilistic sensitivity analysis. Jackson, Thompson and Sharples built a sample of 5,000 draws by merging draws from each model in proportion to its weight. Sampling a model indicator at each iteration, with the weights as probabilities, targets the same mixture. Either way the output is one set of draws, so the cost-effectiveness acceptability curve and the expected value of perfect information are computed exactly as for a single model.

HEMA Report 2 observes that a PSA rarely includes the choice of survival curve unless model averaging is used. Within each draw, the chosen model should apply to every arm it governs, so incremental results are not formed from mismatched curves.

Worked example with four survival curves

All numbers in this example are illustrative. Four parametric curves are fitted to overall survival in the treatment arm of a trial with 400 patients, and each gives a mean survival in life years with a within-model standard deviation from its own probabilistic analysis.

1. Criteria. AIC and BIC are computed as on their own pages, with the natural logarithm of 400, which is 5.991, as the BIC penalty per parameter.

2. AIC weights. The numerators are 0.6065, 1, 0.2231 and 0.6065, which sum to 2.4362. Dividing each by the sum gives weights of 0.249, 0.410, 0.092 and 0.249.

3. BIC weights. For the three two-parameter curves the penalties cancel, so their BIC differences equal their AIC differences. The generalised gamma pays 5.991 per parameter instead of 2, its difference rises to 4.99 and its numerator falls to 0.0824. The numerators now sum to 1.9121, giving weights of 0.317, 0.523, 0.117 and 0.043.

4. Averaged mean. With AIC weights the mean is 0.7719 + 1.599 + 0.4048 + 0.996 = 3.7717 life years, the four terms being each weight times its model mean. With BIC weights it is 0.9827 + 2.0397 + 0.5148 + 0.172 = 3.7092 life years.

5. Averaged variance under AIC weights. The within-model part is 0.0156 + 0.0502 + 0.0186 + 0.0398 = 0.1242, each term a weight times a squared SD. The between-model part uses the squared distance of each model mean from 3.7717, giving 0.1123 + 0.0067 + 0.0363 + 0.0130 = 0.1683. The total is 0.1242 + 0.1683 = 0.2925, an SD of 0.54 life years.

6. Draws for the PSA. Of 5,000 draws, the AIC weights allocate 1245 + 2050 + 460 + 1245 = 5000 to the four curves in table order.

7. Interpretation. The best-fitting log-logistic curve alone gives 3.9 life years with an SD of 0.35. The averaged mean is lower, at 3.77, and its SD of 0.54 is about half as large again, with 58% of the variance (0.1683 out of 0.2925) coming from disagreement between curves. The choice of criterion moves the mean only slightly but cuts the generalised gamma weight from 0.249 to 0.043. None of these weights says anything about survival after the trial ends, so a curve whose long-term hazard clinicians judge implausible should be removed or down-weighted on that evidence.

Choosing the candidate set and checking plausibility

Every weight is relative to the set of candidates, so the set does as much work as the weighting rule. Jackson, Thompson and Sharples cite Draper's advice to choose well-supported models with different predictive consequences, since averaging models with very similar inferences adds nothing.

Fit-based weights have a specific blind spot in survival work. NICE DSU Technical Support Document 14 explains that AIC and BIC are based only on the relative fit of models to the observed data and address internal validity, not the validity of the extrapolated tail that drives the mean. PMG36 (section 4.6.25) asks for the clinical plausibility of the hazard functions behind alternative extrapolations to be assessed routinely. Negrín, Nam and Briggs compared two hip prostheses using 8 years of registry data. The probability that the Spectron prosthesis was cost effective at a threshold of GBP 20,000 was 100% with the best-fitting model and 93% after Bayesian model averaging, but 0% with 16 years of follow-up. Averaging tempered the certainty without reaching the later answer, and the authors noted the value of regularly updating models.

Decision use and reporting in HTA

The NICE manual (PMG36, section 4.7.5) states that structural uncertainty may be incorporated within a probabilistic model, for example by model averaging, and that the alternative assumptions and the probabilities used to weight them should be transparently documented, including details of any expert advice. HEMA Report 2 recommends greater use of model averaging, wherever feasible, for uncertainty now handled by scenarios alone.

A usable submission reports the candidate set with reasons for exclusions, a weighting rule fixed before the results were seen, the weights, and the model-specific results next to the average. The last matters because an average can hide a decision that reverses between curves, and decision uncertainty is easier to judge when a committee can see which curve drives it.

Model averaging beside neighbouring methods

Several methods sit close to model averaging, and the differences lie in what the weights mean.

• Model selection. Model selection is the special case in which one model receives a weight of 1 and all others 0.
• Scenario analysis. Scenario analysis runs the same alternatives without weights, so they show whether the decision changes but not how likely each structure is.
• Finite mixture models. A finite mixture model is one model of a heterogeneous population, whose weights are shares of patients, not support for competing models.
• Machine learning ensembles. Hoeting and colleagues place stacking, boosting and bagging in a non-Bayesian literature focused on point prediction.

Common errors with model averaging

Most errors come from treating the weights as more than they are. Those below recur in survival extrapolation.

• Averaging ICERs or parameters. Only outputs on a common scale are averaged, and the ratio is formed afterwards.
• Reading a weight as the probability that a curve extrapolates correctly. Fit-based weights describe the observed period only.
• Choosing AIC or BIC after seeing the results. The rule changes the weights and should be fixed in advance with its rationale.

Sources

• Hoeting JA, Madigan D, Raftery AE, Volinsky CT. Bayesian model averaging: a tutorial. Statistical Science. 1999;14(4):382-401. https://doi.org/10.1214/ss/1009212519
• Jackson CH, Thompson SG, Sharples LD. Accounting for uncertainty in health economic decision models by using model averaging. Journal of the Royal Statistical Society Series A. 2009;172(2):383-404. https://doi.org/10.1111/j.1467-985X.2008.00573.x
• Jackson CH, Sharples LD, Thompson SG. Structural and parameter uncertainty in Bayesian cost-effectiveness models. Journal of the Royal Statistical Society Series C. 2010;59(2):233-253. https://doi.org/10.1111/j.1467-9876.2009.00684.x
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2011, last updated March 2013. Sections 3.3 to 3.5. https://doi.org/10.15131/shef.data.33745843
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; published 31 January 2022, last updated 13 October 2026. Sections 4.6.25 and 4.7.5. https://www.nice.org.uk/process/pmg36
• Negrín MA, Nam J, Briggs AH. Bayesian solutions for handling uncertainty in survival extrapolation. Medical Decision Making. 2017;37(4):367-376. https://doi.org/10.1177/0272989X16650669
• Steuten L, Thavorn K, Cipriano LE, Guertin JR, McQueen B, Schmidler G, Sullivan S, Wailoo A, Sculpher M. Methods to Analyze and Present Uncertainty in Economic Evaluation to Support HTA Decisions. HEMA Report 2. Health Economics Methods Advisory; 2026. Sections 3.4 and 4.5. https://hemamethods.org/wp-content/uploads/2026/10/HEMA-Report-2_For-Publication_100726.pdf