Logarithmic Pool

Logarithmic Pool for Combining Expert Distributions in Health Economic Models

When several experts each give a probability distribution for a model input that no study has measured, the logarithmic pool multiplies their densities, each raised to a weight, and rescales the product so that it integrates to one. Values that any expert judges very unlikely are pushed down in the result, so the pooled distribution gathers where the experts agree. That choice changes the parameter uncertainty passed to probabilistic sensitivity analysis and to value of information calculations, so in health technology assessment the choice of pooling rule can change the inputs that drive the decision. The method is also called the logarithmic opinion pool, the geometric pool or the log-linear pool, and its main alternative is the arithmetic linear pool.

Pooling as one step in a structured elicitation

The NICE technology appraisal manual (PMG36, section 3.3.21) allows expert elicitation in the absence of empirical evidence from trials, non-randomised studies or registries, prefers structured methods and asks that they adhere to existing protocols such as the Medical Research Council protocol. Section 4.7.10 notes that formal elicitation methods are available when data are lacking to inform a parameter's mean and distribution.

NICE Decision Support Unit Technical Support Document 26, on expert elicitation for long-term survival, separates behavioural aggregation, where experts discuss and agree one set of judgements, from mathematical aggregation. The mathematical route fits a distribution to each expert's judgements and takes a pointwise average, typically the arithmetic mean (the linear pool) or the geometric mean (the logarithmic pool), with experts allowed different weights. Bojke and colleagues, developing the reference protocol for structured expert elicitation in health care decision making, report that linear pooling was the most common approach in their review of elicitation in that field, that the log pool is the main alternative, and that harmonic averaging has also been used. The Delphi method and consensus building are behavioural alternatives, and calibration of experts covers how performance scores can set the weights.

The pooled density and its normalising constant

The logarithmic pool works on the experts' fitted densities, not on their raw quantile judgements, so each expert's judgements are first turned into a distribution. For $n$ experts who each provide a density for the same parameter, the pool is defined as follows.

$$\pi(\theta) = \frac{\prod_{i=1}^{n} f_i(\theta)^{w_i}}{\int \prod_{i=1}^{n} f_i(u)^{w_i}\, du}, \qquad w_i \ge 0, \qquad \sum_{i=1}^{n} w_i = 1$$

where $\theta$ is the uncertain parameter, $f_i(\theta)$ is the density fitted to expert $i$, $w_i$ is that expert's weight, $u$ is the variable of integration and $\pi(\theta)$ is the pooled density. Taking logarithms shows where the name comes from.

$$\ln \pi(\theta) = \sum_{i=1}^{n} w_i \ln f_i(\theta) - \ln c$$

where $c$ is the integral in the denominator above. The log of the pooled density is a weighted average of the experts' log densities, shifted by a constant. Carvalho and colleagues, reviewing the theory, report a proof by Genest and colleagues that for weights in the open simplex the constant exists for any finite set of proper densities, so the pool is always a proper distribution. In a spreadsheet, EXP(SUMPRODUCT(Weights, LogDensities)) gives the unnormalised pool at each grid point, which is then divided by the column sum times the grid width.

Closed forms for beta and normal experts

When experts' views are fitted to distributions of the same family, the pool often stays in that family. Carvalho and colleagues show that pooling beta densities gives a beta distribution whose parameters are the weighted means of the experts' parameters.

$$\alpha = \sum_{i=1}^{n} w_i \alpha_i, \qquad \beta = \sum_{i=1}^{n} w_i \beta_i, \qquad E(\theta) = \frac{\sum_{i} w_i n_i m_i}{\sum_{i} w_i n_i}$$

where $\alpha_i$ and $\beta_i$ are the parameters of expert $i$'s beta distribution, $\alpha$ and $\beta$ are those of the pool, $E(\theta)$ is the pooled mean, $m_i$ is expert $i$'s mean and $n_i = \alpha_i + \beta_i$. The pooled mean is therefore a weighted average of the experts' means in which more confident experts, with larger $n_i$, count for more.

For normal experts, as on a log hazard ratio scale, collecting the squared terms in the exponent gives a normal pool.

$$\tau = \sum_{i=1}^{n} \frac{w_i}{\sigma_i^2}, \qquad \mu = \frac{1}{\tau} \sum_{i=1}^{n} \frac{w_i \mu_i}{\sigma_i^2}$$

where $\mu_i$ and $\sigma_i^2$ are expert $i$'s mean and variance, $\tau$ is the pooled precision and $\mu$ is the pooled mean. If every expert reports the same variance, the pooled variance equals it whatever the distance between their means, so two sharply divided experts give a pool as tight as either of them.

Properties that set the geometric average apart

Dietrich and List, in a review of probabilistic opinion pooling, show that geometric pooling preserves unanimity and is externally Bayesian, an axiom proposed by Madansky under which updating on new information gives the same result before or after aggregation. The proof given by Carvalho and colleagues uses the weights summing to one, so the likelihood appears exactly once in the pooled posterior. Dietrich and List add that geometric pooling is not the only rule with these two properties and becomes unique only once further technical axioms are imposed. Carvalho and colleagues also report relative propensity consistency, under which the pool keeps any ordering of two outcomes that every expert shares, and prove that pooling log-concave densities gives a log-concave result, which supports unimodality.

The same multiplication brings costs. The support of the pool is the intersection of the experts' supports, so a value any expert rules out is ruled out. Williams, Wilson and Wilson, comparing aggregation methods in a clinical trial elicitation, note that when two experts' distributions do not overlap at all the pool has no density, and that with little overlap it can put most of its mass between the experts, on values none of them thought likely. Carvalho and colleagues record West's objections, including that the log pool violates coherence under marginalisation, and Genest and Zidek's reply that these rest on a restrictive assumption about group utility. Bojke and colleagues add that no formula can satisfy every potentially desirable criterion at once.

Choosing and interpreting the weights

Equal weights are the common default. Bojke and colleagues list alternatives based on seed questions, on the expert's background or on agreement with later data, and they cite methods by Rufo and colleagues and others that set weights from the elicited judgements alone. Carvalho and colleagues review weights chosen to maximise the entropy of the pool or to minimise the Kullback-Leibler divergence between pool and experts, and they propose a hierarchical prior on the weights so that the data can update them.

The constraint that weights sum to one carries a judgement about information. Dietrich and List argue that geometric pooling is justified when experts' opinions rest on the same information, while multiplicative pooling, which multiplies the opinions instead of averaging them, suits experts whose views rest on private information. A panel of clinicians who read the same trials sits closer to the first case. Dropping the constraint treats each expert as independent evidence and narrows the pool, as step 5 of the example shows.

Worked example with three clinicians on a response probability

All numbers in this example are illustrative. A cost-effectiveness model needs the probability $p$ of response at 12 months to a new therapy. Three clinicians' quantile judgements are fitted, for example by the method of moments, to Beta(4, 16) for expert A with mean 0.20, Beta(6, 9) for expert B with mean 0.40 and Beta(9, 21) for expert C with mean 0.30. The experts are weighted equally.

1. Pool the parameters. The pool is Beta((4 + 6 + 9)/3, (16 + 9 + 21)/3) = Beta(6.333, 15.333).

2. Pooled mean. The mean is 6.333/21.667 = 0.292, exactly 19/65, below the simple average of 0.300. The experts' implied weights are $n_i$ divided by 65, giving 20/65 = 0.308 for A, 15/65 = 0.231 for B and 30/65 = 0.462 for C, and (20 × 0.20 + 15 × 0.40 + 30 × 0.30)/65 = 19/65 confirms the mean.

3. Spread. The pooled variance is (6.333 × 15.333)/(21.667² × 22.667) = 0.00913, a standard deviation of 0.096 with a 95% central interval of 0.126 to 0.495. The equally weighted linear pool has the same experts but a standard deviation of 0.128 and an interval of 0.086 to 0.584, because it adds the spread of the experts' means.

4. A tail probability. The probability that $p$ exceeds 0.40 is 0.023 for A, 0.486 for B and 0.119 for C. The linear pool averages these to 0.209, while the logarithmic pool gives 0.136.

5. Weights that do not sum to one. The plain product of the three densities is Beta(17, 44), with mean 0.279 and standard deviation 0.057, narrower than any single expert. This treats the three opinions as independent evidence.

6. Data arrive. An early access scheme records 5 responders among 20 patients. Updating the pool gives Beta(6.333 + 5, 15.333 + 15) = Beta(11.333, 30.333). Updating each expert first gives Beta(9, 31), Beta(11, 24) and Beta(14, 36), and pooling those gives Beta(34/3, 91/3), the same distribution with mean 0.272. For the linear pool the two orders give means of 0.273 and 0.269.

7. Decision consequence. Incremental net monetary benefit per patient is taken to be 40,000$p$ minus 10,000 in GBP. Expected net benefit is 40,000 × 19/65 minus 10,000 = GBP 1,692 under the logarithmic pool and 40,000 × 0.300 minus 10,000 = GBP 2,000 under the linear pool. The probability that net benefit is negative, P($p$ < 0.25), is 0.354 against 0.379. Per patient EVPI, the expected value of the larger of net benefit and zero minus the larger of expected net benefit and zero, found by numerical integration, is GBP 792 against GBP 1,138.

8. Interpretation. Adoption is favoured under both rules, yet the logarithmic pool reports about 30% less value from removing uncertainty about $p$, because it discounts expert B's higher values rather than carrying the disagreement forward. Which figure is defensible depends on whether the disagreement is itself uncertainty the decision maker should see.

Consequences for probabilistic analysis and research priorities

The NICE manual (PMG36, section 4.7.10) asks that distributions for probabilistic analysis represent the available evidence, be justified and not be chosen arbitrarily, with separate probabilistic analyses where alternative plausible distributions exist. For an elicited input the pooling rule is such a choice, so running the analysis under both pools is a direct way to show its effect.

Bojke and colleagues, drawing on O'Hagan and colleagues, explain that log pooling discounts values found implausible by at least one expert and concentrates on areas of agreement, while a linear pool encompasses all values any expert finds plausible. They argue that the broader linear pool is preferable in health care decision making because it acknowledges uncertainty from between-expert variation, which can motivate research, and their protocol recommends linear pooling with equal weights. They also list pooling methods other than linear pooling among areas needing research. TSD 26 notes that updating a linear pool with data gives different answers depending on order, the difficulty that external Bayesianity removes, but warns that experts who have seen the data have in effect given a posterior, so a further update counts the data twice. The case for the logarithmic pool is therefore strongest for panels that share an evidence base, where the pooled distribution is used as a prior distribution and later updated to a posterior distribution with data the experts have not seen.

Common misreadings of the logarithmic pool

Most errors come from reading the narrower pool as better evidence. Five recur in elicited inputs.

1. Treating concentration as consensus. A tight pool can sit between two experts who disagree sharply, so the experts' own distributions should be reported alongside it.
2. Ignoring zero tails. When one expert's distribution is bounded by an elicited minimum and maximum, those bounds become hard limits for the whole pool.
3. Letting the weights sum to more than one. The pool then behaves like independent data and understates parameter uncertainty.
4. Reading the pooled mean as the average of means. With beta experts the mean is weighted by each expert's confidence as well as the chosen weight.
5. Assuming external Bayesianity settles the choice. It concerns the order of updating, and says nothing about whether disagreement should widen the distribution.

Sources

• Bojke L, Soares M, Claxton K, Colson A, Fox A, Jackson C, Jankovic D, Morton A, Sharples L, Taylor A. Developing a reference protocol for structured expert elicitation in health-care decision-making: a mixed-methods study. Health Technology Assessment. 2021;25(37):1-124. https://doi.org/10.3310/hta25370
• Carvalho LMF, Villela DAM, Coelho FC, Bastos LS. Combining probability distributions: extending the logarithmic pooling approach. arXiv:1502.04206v2; 2020. Published as Bayesian inference for the weights in logarithmic pooling. Bayesian Analysis. 2023;18(1):223-251. https://arxiv.org/abs/1502.04206
• Dietrich F, List C. Probabilistic opinion pooling. MPRA Paper No. 54806; March 2014. https://mpra.ub.uni-muenchen.de/54806/
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 3.3.21 and 4.7.10. https://www.nice.org.uk/process/pmg36
• Oakley JE, Ren S, Forsyth JE, Gosling JP, Wilson K, Latimer N, Rutherford MJ, Uttley L, Fotheringham J. NICE DSU Technical Support Document 26: Expert elicitation for long-term survival outcomes. Decision Support Unit, ScHARR, University of Sheffield; March 2025. Sections 2.5 and 3.6. https://www.sheffield.ac.uk/nice-dsu/tsds/expert-elicitation-tsd
• Williams CJ, Wilson KJ, Wilson N. A comparison of prior elicitation aggregation using the classical method and SHELF. Journal of the Royal Statistical Society Series A. 2021;184(3):920-940. https://doi.org/10.1111/rssa.12691