Frequentist Analysis

Frequentist Analysis in Trial-Based Economic Evaluation and HTA

In health economics, frequentist analysis is the usual way to analyse the costs and outcomes collected in a randomised controlled trial, reporting a mean difference with a confidence interval and a P value. Its probabilities describe how the data, the estimates and the tests would behave if the trial were repeated many times under an assumed model, and they say nothing directly about how likely a treatment effect or a cost-effectiveness claim is. The framework matters in health technology assessment because trial-based cost-effectiveness analysis and much pairwise meta-analysis are frequentist, while NICE prefers decision models analysed probabilistically, which fit more easily with Bayesian thinking. Bayesian analysis instead gives the probability of an effect given the data and a prior, and frequentist meta-analysis applies the same repeated-sampling logic to pooling trials.

Probability as the frequency of data under repeated sampling

Greenland and colleagues explain that in conventional methods probability refers to hypothetical frequencies of data patterns under an assumed statistical model, not to hypotheses, which is why these methods are called frequentist. The unknown quantity, such as the mean difference in cost per patient between two arms, is treated as fixed. What varies is the sample, and with it the estimate.

Every frequentist statement is therefore about a procedure. A 95% confidence interval comes from a method that, in valid applications, covers the true value 95% of the time on average, and Greenland and colleagues stress, after Neyman, that this coverage is a property of a long sequence of intervals and not of any single one. A P value is the probability that the test statistic would be at least as large as its observed value if every model assumption, including the test hypothesis, were correct. Neither is a probability that the hypothesis is true.

Estimates, tests and intervals and their operating characteristics

Most frequentist estimation starts from a likelihood, the probability of the data as a function of the parameters. Maximum likelihood estimation picks the values that make the observed data most probable.

$$\hat{\theta} = \arg\max_{\theta} \prod_{i=1}^{n} f(x_i \mid \theta)$$

where $\theta$ is the vector of unknown parameters, $x_i$ is the observation for patient $i$ of $n$, $f(x_i \mid \theta)$ is its density or mass under the model and $\hat{\theta}$ is the estimate.

Large-sample hypothesis tests and intervals then use the estimate and its standard error.

$$\hat{\theta} \pm z_{1-\alpha/2}\,\text{SE}(\hat{\theta}), \qquad Z = \frac{\hat{\theta} - \theta_0}{\text{SE}(\hat{\theta})}$$

where $\text{SE}(\hat{\theta})$ is the standard error of the estimate, $z_{1-\alpha/2}$ is the standard normal quantile (1.96 for a 95% interval when $\alpha$ is 0.05), $\theta_0$ is the value under the null hypothesis (usually zero difference) and $Z$ is the Wald test statistic.

Frequentist methods are judged by their operating characteristics. The FDA's January 2026 draft guidance on Bayesian methodology in drug trials, still a draft on 25 October 2026, lists them for frequentist trials. For tests they are the familywise Type I error rate and statistical power, conventionally fixed by design at 0.025 one-sided and 80% or 90%. For estimation they are bias and mean squared error of point estimates and the coverage probability and width of confidence intervals. These can be calculated before the trial because, as the draft explains, frequentist inference rests on the probability of the data given fixed parameter values.

Frequentist methods in trial-based cost-effectiveness analysis

An economic evaluation alongside a trial raises two problems that clinical endpoints rarely do. Cost data are highly skewed, and the quantity of interest combines a difference in cost with a difference in effect. Thompson and Barber argue that, despite the skew, the arithmetic mean is the informative measure, because a budget holder needs the cost of treating all patients, and that comparing medians or log costs may mislead. They point to the non-parametric bootstrap for differences in arithmetic means. Briggs and Gray likewise advise basing analyses on means and reporting confidence limits on the original scale.

Underpowered cost comparisons are common. Briggs and Gray took 32 mean costs with a measure of variance from published studies and found a mean sample size of 78 and a mean coefficient of variation of 1.01. At the mean cost of 37,938 and mean standard deviation of 40,150, 78 per arm could not detect less than a 50% difference in cost, and detecting a 10% difference would need 1,910 patients per arm, so they judged most such studies grossly underpowered.

For the joint question, the frequentist toolkit offers the Fieller method or the bootstrap for an interval around the ICER, and incremental net benefit, which replaces the ratio with a difference that has an ordinary standard error. The NICE manual (PMG36, section 4.7.15) lists confidence ellipses and scatter plots on the cost-effectiveness plane and cost-effectiveness acceptability curves among appropriate ways of presenting parameter uncertainty.

Frequentist evidence synthesis feeding decision models

Inverse-variance pooling and the random-effects method of DerSimonian and Laird are the classic frequentist route to a pooled treatment effect. NICE Decision Support Unit Technical Support Document 2 (TSD 2) says frequentist software produces perfectly valid evidence synthesis, and for pairwise meta-analysis and indirect comparisons of two-arm trials without covariates the maximum likelihood estimates and standard errors can be used directly to create distributions for Monte Carlo sampling.

Networks with loops, multi-arm trials, baseline models or covariates need more care. TSD 2 gives two options, the bootstrap or a multivariate normal distribution built from the maximum likelihood estimates and their covariance matrix, which should be approximately the same as a Bayesian posterior. TSD 2 also notes that frequentist random-effects analyses can carry the uncertainty in the between-study heterogeneity but rarely do.

Choosing between frequentist and Bayesian analysis in HTA

The FDA draft names the prior as the main feature distinguishing Bayesian from frequentist approaches. The draft also describes Bayesian analyses inside an overall frequentist framework, calibrated to control the Type I error rate. Briggs states that a traditional frequentist approach is equivalent to a Bayesian approach with no prior information. In his analysis of blood pressure control in the UK Prospective Diabetes Study, a Bayesian analysis with an uninformative prior gave exactly the same cost-effectiveness results as a standard frequentist analysis.

The difference lies in what the results mean and how they are used. Briggs argues that reading an acceptability curve as the probability that a technology is cost-effective given the data requires a Bayesian interpretation, which is how PMG36 describes the curve. Claxton goes further and argues that rules of inference are irrelevant to the adoption decision, which should rest on mean net benefit irrespective of statistical significance, with the spread of net benefit relevant only to whether more information is needed. That argument underlies value of information analysis.

Chapters 3 and 4 of the NICE manual (PMG36) name neither framework. Section 3.4.11 states that NICE prefers the network meta-analysis methods of the DSU evidence synthesis series, and TSD 2 takes a Bayesian approach, and section 4.7.12 says the committee's preferred cost-effectiveness estimate should come from a probabilistic analysis when possible, unless the model is linear. TSD 2 adds that model critique and model choice matter in either framework. A frequentist analysis of trial data, turned into distributions for probabilistic sensitivity analysis, therefore fits NICE methods.

Worked example linking net benefit tests to the acceptability curve and trial size

All numbers are illustrative. A trial with 200 patients per arm estimates an incremental cost of GBP 620 (standard error GBP 180) and an incremental gain of 0.030 QALYs (standard error 0.012). For simplicity the two estimates are taken as uncorrelated.

Step 1. Test each difference. Since 1.96 × 180 = 352.8, the cost interval is 620 ± 352.8, or GBP 267 to GBP 973, and the test statistic is 620 / 180 = 3.44, with a two-sided P of 0.0006. The QALY interval is 0.030 ± 0.024, or 0.006 to 0.054, with a test statistic of 2.5 and P of 0.012. Both differences are statistically significant, and the ICER is 620 / 0.030 = GBP 20,667 per QALY.

Step 2. Net benefit and its standard error. Incremental net benefit at threshold $\lambda$ and its variance are

$$\text{INB} = \lambda \Delta E - \Delta C, \qquad \text{Var(INB)} = \lambda^{2}\,\text{Var}(\Delta E) + \text{Var}(\Delta C)$$

where $\lambda$ is the value placed on a QALY, $\Delta E$ is the incremental QALYs, $\Delta C$ is the incremental cost and the covariance term is zero by assumption. Both ends of the GBP 25,000 to GBP 35,000 per QALY range in PMG36 (section 4.7.15) are used. At GBP 25,000 the QALY gain is worth 25,000 × 0.030 = 750, so INB is 750 − 620 = GBP 130, the variance is 90,000 + 32,400 = 122,400 and the standard error is GBP 349.86. At GBP 35,000 the gain is worth 35,000 × 0.030 = 1,050, so INB is 1,050 − 620 = GBP 430, the variance is 176,400 + 32,400 = 208,800 and the standard error is GBP 456.95.

Step 3. One-sided tests of positive net benefit. At GBP 25,000 the test statistic is 130 / 349.86 = 0.37 and the one-sided P is 0.355. At GBP 35,000 the statistic is 430 / 456.95 = 0.94 and P is 0.173.

Step 4. The link to the acceptability curve. Under the normal approximation, the acceptability curve at a threshold equals one minus that one-sided P, so 0.645 at GBP 25,000 and 0.827 at GBP 35,000. This is the formal relation between the curve and P values that Löthgren and Zethraeus set out. Calling 0.645 a probability of cost-effectiveness is the Bayesian reading with no prior information that Briggs describes.

Step 5. Power for net benefit. To detect an INB of GBP 130 with 80% power at a two-sided 5% level, the standard error would need to fall to 130 / (1.96 + 0.84) = GBP 46.43. Since the variance shrinks in proportion to sample size, the trial would need about (349.86 / 46.43)² = 56.8 times as many patients, roughly 11,360 per arm.

Interpretation. Both differences are significant, yet confirming positive net benefit at GBP 25,000 would need a trial about 57 times larger. On Claxton's argument the technology is still preferred at both thresholds because expected net benefit is positive, and the uncertainty bears on the value of further research.

Misreadings and limits

Most errors with frequentist results come from reading a property of the procedure as a statement about the result in hand. Five recur in trial-based analyses.

1. Treating P as the probability that the null hypothesis is true. Greenland and colleagues reject this outright, since the P value assumes the test hypothesis.
2. Treating a reported 95% interval as having a 95% chance of containing the true value. A credible interval answers the question many readers have in mind.
3. Reading P above 0.05 as no difference. Underpowered cost comparisons make this especially misleading.
4. Basing adoption on significance. In Claxton's framework the decision rests on mean net benefit, and the spread informs the value of further evidence.
5. Forgetting the model. Every frequentist probability is computed under an assumed model, so selective reporting, multiple testing or a wrong distribution for costs leaves the stated error rates unreliable.

Sources

• Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis. Health Economics. 1999;8(3):257-261. https://pubmed.ncbi.nlm.nih.gov/10348420/
• Briggs AH. A Bayesian approach to stochastic cost-effectiveness analysis: an illustration and application to blood pressure control in type 2 diabetes. International Journal of Technology Assessment in Health Care. 2001;17(1):69-82. https://doi.org/10.1017/s0266462301104071
• Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technology Assessment. 1999;3(2). https://doi.org/10.3310/hta3020
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/S0167-6296(98)00039-3
• DerSimonian R, Laird N. Meta-analysis in clinical trials. Controlled Clinical Trials. 1986;7(3):177-188. https://doi.org/10.1016/0197-2456(86)90046-2
• Dias S, Welton NJ, Sutton AJ, Ades AE. NICE DSU Technical Support Document 2: A generalised linear modelling framework for pairwise and network meta-analysis of randomised controlled trials. Decision Support Unit; August 2011, last updated September 2016. Section 7. https://www.sheffield.ac.uk/sites/default/files/2022-02/TSD2-General-meta-analysis-corrected-2Sep2016v2.pdf
• Food and Drug Administration. Use of Bayesian Methodology in Clinical Trials of Drug and Biological Products. Draft guidance for industry, January 2026. https://www.fda.gov/media/190505/download
• Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. European Journal of Epidemiology. 2016;31(4):337-350. https://doi.org/10.1007/s10654-016-0149-3
• Löthgren M, Zethraeus N. Definition, interpretation and calculation of cost-effectiveness acceptability curves. Health Economics. 2000;9(7):623-630. https://pubmed.ncbi.nlm.nih.gov/11103928/
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 3.4.11, 4.7.12 and 4.7.15. https://www.nice.org.uk/process/pmg36
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200. https://doi.org/10.1136/bmj.320.7243.1197