Confidence Interval

Confidence Interval Coverage, Construction and Use in Economic Evaluation

A confidence interval puts lower and upper limits around an estimate, such as an incremental cost or a hazard ratio, to show how precisely the data pin it down. At the usual 95% level, the method that produces the limits captures the true value in 95% of repeated samples when its assumptions hold, which is a statement about the method rather than about one reported pair of numbers. In health economics confidence intervals surround differences in mean cost and QALYs from trials, the treatment effects and costs that feed decision models, and incremental net benefit at a chosen threshold. An interval is built from an estimate and its standard error, and its width depends on choices such as the scale, the handling of skewed cost data and the treatment of ratio statistics. Separate intervals for costs and QALYs can mislead, as an illustrative trial example shows, and published intervals are also used to parameterise models.

What the 95% describes

Greenland and colleagues, in their guide to misinterpretations, note that Neyman proposed confidence intervals because of a coverage property, namely that if 95% intervals are calculated repeatedly in valid applications, 95% of them on average will contain the true effect size. Neyman stressed that this coverage probability belongs to a long sequence of intervals computed from valid models, not to any single interval. A reported interval either contains the true value or does not; the 95% refers only to how often intervals computed from very many studies would contain it if all the assumptions used were correct.

The same authors give a second reading that is often more useful in practice. The effect sizes whose test gives P greater than 0.05 typically form the 95% interval, so they are more compatible with the data than values outside it, again only if the statistical model is correct. An interval that can be read as having a 95% probability of containing the true value needs a prior distribution as well, and is then a credible interval from a Bayesian analysis.

Building an interval from an estimate and its standard error

Most intervals in economic evaluation follow one pattern, the point estimate plus and minus a critical value times its standard error. The critical value sets the confidence level, and the standard error carries the information about precision.

$$\hat{\theta} \pm z_{1-\alpha/2} \times SE(\hat{\theta})$$

where $\hat{\theta}$ is the estimate (for example a difference in mean cost), $SE(\hat{\theta})$ is its standard error, $\alpha$ is one minus the confidence level and $z_{1-\alpha/2}$ is the standard normal critical value, 1.96 for a 95% interval. The Cochrane Handbook (chapter 6) notes that a large-sample 95% interval is therefore 3.92 standard errors wide, against 3.29 for a 90% interval and 5.15 for a 99% interval. With small samples the critical value comes from the t distribution, and the handbook gives 2.0639 for a 95% interval from 25 observations.

Because a standard error usually falls with the square root of the sample size, halving the width of an interval needs about four times as many patients. Scale matters for ratio measures. The handbook states that ratio measures such as risk ratios, odds ratios and hazard ratios are analysed on the natural log scale, so the interval is built for the log ratio and the limits are exponentiated, which gives an interval that is asymmetric around the ratio.

Choices that change the interval for costs and cost-effectiveness

Cost data are usually right-skewed, which raises the question of which mean to put the interval around. Thompson and Barber argue that the arithmetic mean is the informative measure for cost data in pragmatic trials, because only it gives the total cost of treating all patients that budgets need. A t test on log-transformed costs compares geometric means instead, so its interval answers a different question. They add that a t-based interval for the difference in arithmetic means will be reliable if the skewness is not too extreme or the samples are moderately large, and offer the non-parametric bootstrap as an alternative that makes no assumption about the shape of the distribution. In the menorrhagia trial they use as an example, with 78 and 70 patients, the 95% interval for the GBP 320 difference in mean cost was GBP 204 to GBP 437 by the t-based method and GBP 192 to GBP 426 by the bootstrap.

The ICER is harder. Briggs, Mooney and Wonderling describe the variance of a ratio estimator as intractable, and in their simulation study methods that assumed a normal sampling distribution performed poorly, as did simply combining the separate intervals on costs and effects. Briggs and Gray's Health Technology Assessment monograph recommended the Fieller method or bootstrapping for an interval around a cost-effectiveness ratio. An interval around incremental net benefit avoids the ratio altogether. Hoch and colleagues show that in net benefit regression the 95% interval for the treatment coefficient is the 95% interval for incremental net benefit.

Worked example with two significant differences and an uncertain net benefit

All numbers here are illustrative. A trial-based cost-utility analysis estimates an incremental cost of GBP 1,200 (standard error GBP 400) and an incremental effect of 0.060 QALYs (standard error 0.025). The correlation between the two estimates is 0.3, so their covariance is 0.3 × 400 × 0.025 = 3.0.

Step 1. Separate intervals. Since 1.96 × 400 = 784, the cost interval is 1,200 ± 784, or GBP 416 to GBP 1,984. Since 1.96 × 0.025 = 0.049, the QALY interval is 0.060 ± 0.049, or 0.011 to 0.109. Both exclude zero, and the ICER is 1,200 / 0.060 = GBP 20,000 per QALY.

Step 2. Net benefit at an illustrative threshold. At GBP 30,000 per QALY, incremental net monetary benefit and its variance are

$$\text{INMB} = \lambda \Delta E - \Delta C, \qquad \text{Var(INMB)} = \lambda^{2}\,\text{Var}(\Delta E) + \text{Var}(\Delta C) - 2\lambda\,\text{Cov}(\Delta E, \Delta C)$$

where $\lambda$ is the threshold value of a QALY, $\Delta E$ and $\Delta C$ are the incremental QALYs and cost, and Var and Cov are their sampling variances and covariance. INMB = 30,000 × 0.060 − 1,200 = GBP 600. The threshold term is 900,000,000 × 0.000625 = 562,500 and the covariance term is 2 × 30,000 × 3.0 = 180,000, so the variance is 562,500 + 160,000 − 180,000 = 542,500 and the standard error is GBP 736.55.

Step 3. The net benefit interval. Since 1.96 × 736.55 = 1,443.63, the interval is 600 ± 1,443.63, or minus GBP 844 to GBP 2,044 per patient. Ignoring the correlation would give a variance of 722,500, a standard error of GBP 850 and a wider interval of minus GBP 1,066 to GBP 2,266.

Step 4. The matching ICER interval. Hoch and colleagues show that the thresholds at which the lower and upper net benefit limits cross zero mark the Fieller interval for the ICER. Solving for those points gives GBP 7,361 and GBP 93,515 per QALY; at GBP 7,361 the net benefit of minus GBP 758 is exactly 1.96 standard errors below zero, and at GBP 93,515 the net benefit of GBP 4,411 is exactly 1.96 standard errors above it.

Interpretation. Each separate interval excludes zero, yet at GBP 30,000 per QALY the data are compatible with a net loss of about GBP 844 or a net gain of about GBP 2,044 per patient. The threshold lies inside the Fieller interval, which is the same finding in ratio form. Reporting the two separate intervals would overstate how settled the decision is.

Using reported intervals to parameterise decision models

Decision models often take their inputs from published results that give an estimate and a 95% interval but no standard error. The ISPOR-SMDM task force report on parameter estimation and uncertainty observes that standard estimation methods give a point estimate with a measure of precision, such as standard errors or 95% confidence intervals, and that the range for a one-way deterministic sensitivity analysis can be based on a 95% interval. The Cochrane Handbook recovers a standard error by dividing the width of a 95% interval by 3.92, after taking natural logs for a ratio measure.

As an illustration, a hazard ratio of 0.75 reported with a 95% interval of 0.60 to 0.94 gives SE(ln HR) = (ln 0.94 − ln 0.60) / 3.92 = (−0.0619 + 0.5108) / 3.92 = 0.1145. That value can parameterise a distribution for ln HR in a probabilistic sensitivity analysis. Rebuilding the interval as exp(−0.2877 ± 1.96 × 0.1145) returns 0.60 to 0.94, a useful check that the extraction is right.

Interpretation and decision use in HTA

A confidence interval summarises sampling variation and nothing else. Briggs and Gray stress that statistical analysis only summarises uncertainty due to sampling variation, leaving sensitivity analysis to handle inputs without sample data, such as unit costs, and issues of generalisability, extrapolation and methods. Structural uncertainty and bias do not appear in the limits either.

For a decision, the interval is read against values that matter. Greenland and colleagues advise looking at the interval to see which effect sizes of clinical or practical importance are compatible with the data, and point out that precision is shown by width on the appropriate scale, not by whether the null value is excluded. In their example an interval of 5 to 40 excludes zero but is more than twice as wide as an interval of −5 to 10. Briggs and Gray also note that cost-effectiveness acceptability curves give decision makers more information than confidence intervals because they do not adhere to conventional significance levels, and Löthgren and Zethraeus set out the formal relation between the curve and inference based on intervals and P values. For trial reporting, the CONSORT 2025 statement (item 26) asks for each outcome's estimated effect size and its precision, such as a 95% confidence interval.

Credible intervals, P values and future studies

Three neighbouring ideas are often run together with the confidence interval. A credible interval states posterior probability under a prior and model, which a confidence interval cannot. A P value tests one value, while the interval shows the whole range of values with P above the cut-off, and statistical significance is just the question of whether the null value lies outside. Greenland and colleagues also warn that an observed 95% interval does not predict where 95% of future estimates will fall. For two independent, unbiased studies with equal standard errors, the chance that the first interval contains the second estimate is 83%.

Misreadings that recur

Most errors with confidence intervals fall into a few groups, several of them listed by Greenland and colleagues. Each one gives the interval a meaning that its construction does not support.

1. Probability for one interval. Saying the reported interval has a 95% chance of containing the true value gives it a Bayesian meaning it does not have.
2. Values outside are refuted. The interval rests on many assumptions, and declaring a value excluded needs stronger conditions than falling outside the limits.
3. Overlap means no difference. Intervals of 1.04 to 4.96 and 4.16 to 19.84 overlap, yet the test of no difference gives P = 0.03; groups should be compared with an interval for the difference.
4. Including the null shows no effect. A wide interval that includes zero may reflect too little data.
5. Separate cost and QALY intervals as joint uncertainty. Hoch and colleagues note that an ICER interval cannot be made from the separate intervals because that ignores the correlation between costs and effects.
6. Reading the limits as total uncertainty. Bias, model structure and inputs without sample data sit outside the limits.

Sources

• Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technology Assessment. 1999;3(2). https://doi.org/10.3310/hta3020
• Briggs AH, Mooney CZ, Wonderling DE. Constructing confidence intervals for cost-effectiveness ratios: an evaluation of parametric and non-parametric techniques using Monte Carlo simulation. Statistics in Medicine. 1999;18(23):3245-3262. https://pubmed.ncbi.nlm.nih.gov/10602149/
• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722-732. https://doi.org/10.1177/0272989X12458348
• Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350. https://doi.org/10.1007/s10654-016-0149-3
• Higgins JPT, Li T, Deeks JJ (editors). Chapter 6: Choosing effect measures and computing estimates of effect (last updated August 2023). In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-06
• Hoch JS, Hay A, Isaranuwatchai W, et al. Advantages of the net benefit regression framework for trial-based economic evaluations of cancer treatments: an example from the Canadian Cancer Trials Group CO.17 trial. BMC Cancer. 2019;19:552. https://doi.org/10.1186/s12885-019-5779-x
• Hopewell S, Chan AW, Collins GS, et al. CONSORT 2025 statement: updated guideline for reporting randomised trials. BMJ. 2025;389:e081123. https://doi.org/10.1136/bmj-2024-081123
• Löthgren M, Zethraeus N. Definition, interpretation and calculation of cost-effectiveness acceptability curves. Health Economics. 2000;9(7):623-630. https://pubmed.ncbi.nlm.nih.gov/11103928/
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200. https://doi.org/10.1136/bmj.320.7243.1197