Confidence Profile Method

Confidence Profile Method in Bayesian Evidence Synthesis

The Confidence Profile Method turns each piece of evidence on a health technology, whether a prospective, case-control or cross-sectional study, into a likelihood function, adjusts that likelihood for the biases the analyst believes the study carries, and combines everything through Bayes' formula into a joint posterior distribution for the parameters of interest. David Eddy described it in Operations Research in 1989, after confidence profile analyses had appeared in 1987 and 1988, and with Hasselblad and Shachter published fuller accounts in 1990 and a book in 1992. For health technology assessment it matters because a decision model needs a joint probability distribution for its parameters that reflects all the evidence and its flaws, and the method was an early, explicit way to produce that. Its treatment of bias and of indirect evidence linked through a structural model anticipated multi-parameter evidence synthesis and the Bayesian methods that feed probabilistic sensitivity analysis.

Why assessors needed a new synthesis method

Mulrow, Langhorne and Grimshaw called evidence direct when it relates an intervention to the principal health outcome, and indirect when two or more bodies of evidence are needed to make that link. Formal synthesis by meta-analysis had concentrated almost exclusively on randomised studies, as Prevost and colleagues observed, and differences in rigour were typically handled by a threshold for inclusion, which Turner and colleagues note can discard valuable evidence.

Eddy's 1989 paper in Operations Research aimed at parameters such as the effect of health technologies on health outcomes. The 1990 papers with Hasselblad and Shachter describe its use where evidence involves combinations of biases, uncertainty about biases, indirect evidence and technology families, and a book followed in 1992.

A RAND review of spinal manipulation (1992) used it to report, for uncomplicated acute low-back pain, a difference in the probability of recovery at 3 weeks of 0.17 with 95% probability limits of 0.07 to 0.28. In 1988 Eddy and colleagues paired confidence profiles with the CAN*TROL model to estimate the health and economic consequences of annual mammography for women aged 40 to 49, an early case of evidence synthesis used to estimate economic consequences.

Likelihoods, priors and the joint posterior

The 1990 technology assessment paper lists the elements as prior distributions, likelihood functions, and models for specific experimental designs that include adjustment for biases. Each study contributes a likelihood, the probability of its observed results as a function of possible parameter values, and the posterior follows from Bayes theorem.

$$p(\boldsymbol{\theta} \mid D_1, \ldots, D_k) \propto p(\boldsymbol{\theta}) \prod_{j=1}^{k} L_j(\boldsymbol{\theta})$$

where $\boldsymbol{\theta}$ is the vector of model parameters, $p(\boldsymbol{\theta})$ is the prior distribution, $D_j$ is the data from piece of evidence $j$, $L_j(\boldsymbol{\theta})$ is its likelihood function and $k$ is the number of pieces of evidence.

Likelihood functions were derived for prospective studies, case-control and cross-sectional studies. The output is a joint posterior distribution, from which the marginal distribution of any parameter can be calculated, with a covariance matrix for all parameters.

Bias models and adjusted likelihoods

Eddy adjusted each piece of evidence for threats to internal or external validity by modelling the bias and deriving an adjusted likelihood, covering crossover in controlled trials, error in measuring outcomes, patient selection, differences in the technologies studied and differences in length of follow-up. As a general illustration rather than Eddy's own formulation, an adjusted likelihood for an additive bias can be written as follows.

$$L_j^{\text{adj}}(\theta) = \int L_j(\theta + \beta_j) \, p(\beta_j) \, d\beta_j$$

where $\theta$ is the target parameter, $\beta_j$ is the bias in study $j$, $p(\beta_j)$ is the distribution the analyst assigns to that bias and $L_j^{\text{adj}}(\theta)$ is the adjusted likelihood. A bias with a non-zero mean moves the study's estimate and uncertainty about it flattens the likelihood, so the study counts for less.

Indirect evidence through a structural model

Evidence often bears on a function of several parameters. Ades (2003) reanalysed a data set from the confidence profile literature on thrombolysis after acute myocardial infarction, with data on reperfusion, on survival in reperfused and non-reperfused patients, and on overall survival without reperfusion state. If treatment is assumed to change survival only through reperfusion, the law of total probability links the three.

$$S_t = r_t \, s_R + (1 - r_t) \, s_N$$

where $S_t$ is overall survival under treatment $t$, $r_t$ is the probability of reperfusion under treatment $t$, and $s_R$ and $s_N$ are survival in reperfused and non-reperfused patients. Trials of overall survival inform $S_t$ directly and the other studies inform its components, so all three update the same parameters.

Worked example with a trial and a biased registry

All numbers in this example are illustrative. An assessment needs the log odds ratio for death within one year. A randomised trial gives -0.30 (standard error 0.15) and a registry study -0.50 (standard error 0.10), and the assessor judges that such registries exaggerate benefit by 0.10 on the log odds scale on average, with a standard deviation of 0.15 for that bias. The prior is vague and all distributions are normal.

1. Adjust the registry likelihood. Removing the expected bias moves the registry estimate from -0.50 to -0.40. The bias uncertainty adds to the sampling variance, giving 0.0100 plus 0.0225, or 0.0325, a standard error of 0.180.

2. Combine by precision. The trial has precision 1 / 0.0225 = 44.44 and the adjusted registry 1 / 0.0325 = 30.77.

$$\mu_{post} = \frac{44.44 \times (-0.30) + 30.77 \times (-0.40)}{44.44 + 30.77} = \frac{-25.64}{75.21} = -0.341$$

where $\mu_{post}$ is the posterior mean log odds ratio. The posterior standard deviation is the square root of 1 / 75.21, or 0.115, so the posterior odds ratio is 0.711 with a 95% credible interval of 0.567 to 0.891. Without adjustment the odds ratio would be 0.645 (0.548 to 0.759).

3. Feed the model. A comparator one-year death probability of 0.20 gives odds of 0.25, treated odds of 0.25 times 0.711, or 0.178, and the treated probability 0.178 / 1.178 = 0.151, about 49 deaths avoided per 1,000. At face value the registry gives about 61 per 1,000.

4. Interpret. The bias model pulls the estimate towards the trial and widens it, so a Markov model fed the unadjusted figure would overstate the health gain and understate decision uncertainty.

From joint posterior to probabilistic cost-effectiveness analysis

Ades, Claxton and Sculpher (2006) argued that many evidence structures induce correlations, often because studies inform functions of several parameters, and that Monte Carlo sampling from the joint posterior handles this correctly. The NICE Decision Support Unit tutorials note that expected net benefit equals net benefit at the expected parameter values only when net benefit is linear in every parameter and the parameters are uncorrelated, which is rare.

Spiegelhalter and Best (2003) weighted sources of evidence according to their assumed quality in a generalised synthesis and carried the joint posterior to cost-effectiveness acceptability curves using Markov chain Monte Carlo. Cooper and colleagues (2004) fitted synthesis, model inputs and the decision model in one Bayesian analysis, an approach the tutorials refer to as comprehensive decision analysis.

The NICE manual (PMG36) asks for parameter distributions that represent the available evidence (section 4.7.10), for correlation between parameters to be reflected in the probabilistic analysis (section 4.7.19), and for bias adjustment to be considered when there are concerns about the quality or size of studies in a network meta-analysis (section 3.4.21).

Assumptions and choices that drive the result

The bias distributions are judgements, and the authors of a 1989 confidence profile analysis of angioplasty wrote that their results were as strong as the assumptions in them were valid. Turner and colleagues (2009) elicited expert opinion to build bias distributions for each study in a reanalysis of a NICE appraisal of anti-D prophylaxis. Adjustment moved the combined estimate about 10% further from the null and almost tripled its variance.

The structural model is a second choice. When Ades refitted the thrombolysis data, the model from the confidence profile literature, which assumed no differences between studies in baseline effects, fitted poorly, and cross-validation showed the overall survival evidence was inconsistent with the rest. Under random study effects the evidence appeared consistent, but results were sensitive to the prior on the between-study variance.

Confidence Profile Method compared with later synthesis methods

Ades and Sutton (2006) grouped the method with cross-design synthesis, hierarchical models and generalised evidence synthesis as approaches that combine different study designs or evidence on multiple parameters. Ades defined multi-parameter evidence synthesis as a generalisation of meta-analysis in which several parameters are estimated jointly, and refitted confidence profile data within it. Bayesian meta-analysis usually adds a hierarchical model for between-study variation, which Prevost and colleagues (2000) extended with a level for study type.

The NICE manual requires network meta-analysis to preserve randomisation within trials (section 3.4.20), whereas the confidence profile method was designed to bring non-randomised designs and their biases into the same model.

Limitations and common misreadings

The method asks more of the analyst than a standard meta-analysis. Several common errors come from how its output is read.

1. Reading the posterior as objective. It depends on the priors and bias distributions, so plausible alternatives need testing.
2. Treating adjustment as removal of bias. A bias model replaces an unknown bias with a stated belief about it.
3. Skipping consistency checks. Evidence on linked parameters can conflict and should be tested.
4. Confusing the name with a confidence interval. The output is a posterior distribution, reported by RAND as probability limits.

Sources

• Eddy DM. The confidence profile method: a Bayesian method for assessing health technologies. Operations Research. 1989;37(2):210-228. https://doi.org/10.1287/opre.37.2.210
• Eddy DM, Hasselblad V, Shachter R. A Bayesian method for synthesizing evidence: the Confidence Profile Method. International Journal of Technology Assessment in Health Care. 1990;6(1):31-55. https://doi.org/10.1017/s0266462300008928
• Eddy DM, Hasselblad V, Shachter R. An introduction to a Bayesian method for meta-analysis: the confidence profile method. Medical Decision Making. 1990;10(1):15-23. https://doi.org/10.1177/0272989X9001000104
• Eddy DM, Hasselblad V, Shachter R. Meta-analysis by the Confidence Profile Method: the Statistical Synthesis of Evidence. San Diego: Academic Press; 1992.
• Critchfield GC, Eddy DM. A confidence profile analysis of the effectiveness of disulfiram in the treatment of chronic alcoholism. Medical Care. 1987;25(12 Suppl):S66-S75. https://doi.org/10.1097/00005650-198712001-00004
• Eddy DM, Hasselblad V, McGivney W, Hendee W. The value of mammography screening in women under age 50 years. JAMA. 1988;259(10):1512-1519. https://pubmed.ncbi.nlm.nih.gov/3123720/
• Adar R, Critchfield GC, Eddy DM. A confidence profile analysis of the results of femoropopliteal percutaneous transluminal angioplasty in the treatment of lower-extremity ischemia. Journal of Vascular Surgery. 1989;10(1):57-67. https://doi.org/10.1067/mva.1989.vs0100057
• Shekelle PG, Adams AH, Chassin MR, et al. Spinal manipulation for low-back pain. Annals of Internal Medicine. 1992;117(7):590-598. https://doi.org/10.7326/0003-4819-117-7-590
• Mulrow C, Langhorne P, Grimshaw J. Integrating heterogeneous pieces of evidence in systematic reviews. Annals of Internal Medicine. 1997;127(11):989-995. https://doi.org/10.7326/0003-4819-127-11-199712010-00008
• Prevost TC, Abrams KR, Jones DR. Hierarchical models in generalized synthesis of evidence: an example based on studies of breast cancer screening. Statistics in Medicine. 2000;19(24):3359-3376. https://pubmed.ncbi.nlm.nih.gov/11122501/
• Ades AE. A chain of evidence with mixed comparisons: models for multi-parameter synthesis and consistency of evidence. Statistics in Medicine. 2003;22(19):2995-3016. https://doi.org/10.1002/sim.1566
• Spiegelhalter DJ, Best NG. Bayesian approaches to multiple sources of evidence and uncertainty in complex cost-effectiveness modelling. Statistics in Medicine. 2003;22(23):3687-3709. https://doi.org/10.1002/sim.1586
• Cooper NJ, Sutton AJ, Abrams KR, et al. Comprehensive decision analytical modelling in economic evaluation: a Bayesian approach. Health Economics. 2004;13(3):203-226. https://doi.org/10.1002/hec.804
• Ades AE, Sutton AJ. Multiparameter evidence synthesis in epidemiology and medical decision-making: current approaches. Journal of the Royal Statistical Society Series A. 2006;169(1):5-35. https://doi.org/10.1111/j.1467-985X.2005.00377.x
• Ades AE, Claxton K, Sculpher M. Evidence synthesis, parameter correlation and probabilistic sensitivity analysis. Health Economics. 2006;15(4):373-381. https://doi.org/10.1002/hec.1068
• Turner RM, Spiegelhalter DJ, Smith GCS, Thompson SG. Bias modelling in evidence synthesis. Journal of the Royal Statistical Society Series A. 2009;172(1):21-47. https://doi.org/10.1111/j.1467-985X.2008.00547.x
• Dias S, Sutton AJ, Welton NJ, Ades AE. Evidence synthesis for decision making 6: embedding evidence synthesis in probabilistic cost-effectiveness analysis. Medical Decision Making. 2013;33(5):671-678. https://pmc.ncbi.nlm.nih.gov/articles/PMC3704202/
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 2022, last updated 13 October 2026. Sections 3.4.20, 3.4.21, 4.7.10 and 4.7.19. https://www.nice.org.uk/process/pmg36