Asymptotic Normality

Asymptotic Normality and Large-Sample Intervals for Mean Costs, ICERs and Model Parameters

Many confidence intervals in economic evaluation, around a mean cost difference, a QALY gain, a regression coefficient or an incremental net benefit, are built by treating the estimate as if it came from a normal distribution. Asymptotic normality is the large-sample result that justifies this, even when patient-level costs are far from normal. The result supports standard intervals for mean costs and model parameters, but the approximation is poor for heavily skewed costs in small trials and for ratio statistics such as the incremental cost-effectiveness ratio. An illustrative trial compares delta-method, Fieller and net benefit intervals, followed by the alternatives used in practice.

From the central limit theorem to the large-sample interval

The starting point is the central limit theorem, which concerns averages. For independent and identically distributed observations with a finite mean and variance, the sample mean, centred on the population mean and scaled by the square root of the sample size, converges in distribution to a normal variable with mean zero and the population variance, whatever the shape of the population distribution. In symbols, the result is as follows.

$$\sqrt{n}\,(\bar{X}_n - \mu) \xrightarrow{d} N(0, \sigma^2)$$

where $n$ is the number of observations, $\bar{X}_n$ is the sample mean, $\mu$ is the population mean, $\sigma^2$ is the finite population variance, and $\xrightarrow{d}$ denotes convergence in distribution. An estimator is asymptotically normal when a statement of this form holds for it, with its own limiting variance.

The practical consequence is the large-sample (Wald) interval. Once the standard error has been estimated, an approximate 95% confidence interval is given below.

$$\hat{\theta} \pm z_{0.975} \times \widehat{SE}(\hat{\theta})$$

where $\hat{\theta}$ is the estimate, $\widehat{SE}(\hat{\theta})$ is its estimated standard error, and $z_{0.975}$ is the 97.5th percentile of the standard normal distribution, approximately 1.96. The interval is symmetric by construction and its coverage is only approximately 95%.

The same reasoning covers maximum likelihood estimation. Under regularity conditions, including a correctly specified model and a true value inside the parameter space, the estimator satisfies the following.

$$\sqrt{n}\,(\hat{\theta}_{ML} - \theta) \xrightarrow{d} N\big(0, I(\theta)^{-1}\big)$$

where $\hat{\theta}_{ML}$ is the maximum likelihood estimate, $\theta$ is the true parameter value, and $I(\theta)$ is the Fisher information per observation. The inverse of the observed information gives the estimated variance-covariance matrix that software reports with a fitted model.

Why economic evaluation leans on the normal approximation

Health economic analysis relies on this result at several points, often without saying so.

For trial costs the quantity of interest is the arithmetic mean, because the mean multiplied by the number of patients gives the total cost of treating them. Thompson and Barber concluded that the arithmetic mean is the informative measure for cost data in pragmatic trials. A difference in arithmetic means is a difference of averages, so the central limit theorem applies and a normal-approximation interval is valid in large samples even when individual costs are skewed.

For regression-based parameters, such as the coefficients of a parametric survival model, asymptotic normality supplies the joint distribution of the estimates. The NICE Decision Support Unit's survival analysis guidance (TSD 14) notes that parameter uncertainty can be taken into account using the variance-covariance matrices of the fitted models. In a probabilistic model the coefficients are then usually drawn together from a multivariate normal distribution through a Cholesky decomposition, which preserves their correlation; this is a standard method in decision modelling texts such as Briggs, Claxton and Sculpher.

For other inputs, the ISPOR-SMDM good practice task force on parameter estimation states that a parameter informed by plenty of data can be given a normal distribution in probabilistic sensitivity analysis, because the sampling distribution of an arithmetic mean is normal with sufficient sample size. NICE's methods manual (PMG36) asks that distributions represent the available evidence and are not chosen arbitrarily.

Skewed cost data and the question of sample size

Asymptotic normality says nothing about how large a sample must be. With costs the answer depends on skewness. A long right tail, produced by a few patients with complications or long stays, slows the approach to normality.

Thompson and Barber describe highly skewed distributions as typical of cost data and note that normal-theory methods are fairly robust to non-normality, especially in large samples, but that robustness for a particular dataset is hard to judge. In a simulation study, Nixon, Wonderling and Grieve found that central limit theorem and bootstrap standard errors for incremental net benefit were both accurate when samples were moderate to large (more than 50 in their experiments), even with highly skewed costs; with small, highly skewed samples the central limit theorem errors were slightly more accurate.

O'Hagan and Stevens argued that, even where the bootstrap and methods based on asymptotic normality of sample means are technically valid, they may give inefficient and even misleading inferences for skewed costs, and that methods recognising the skewness should be used. A generalised linear model with a gamma or similar distribution is one such method, and Barber and Thompson note that it makes inferences about the mean cost directly.

Why the ICER resists a normal approximation

The ICER is a ratio of two estimated differences, and ratios can approach normality slowly. A large-sample variance can still be derived with the delta method, also called the Taylor series method. It linearises the ratio around the two estimated differences, as follows.

$$\widehat{Var}(\hat{R}) \approx \hat{R}^2 \left[ \frac{Var(\Delta \hat{C})}{\Delta \hat{C}^2} + \frac{Var(\Delta \hat{E})}{\Delta \hat{E}^2} - \frac{2\,Cov(\Delta \hat{C}, \Delta \hat{E})}{\Delta \hat{C}\,\Delta \hat{E}} \right]$$

where $\hat{R} = \Delta \hat{C} / \Delta \hat{E}$ is the estimated ICER, $\Delta \hat{C}$ and $\Delta \hat{E}$ are the estimated differences in mean cost and mean effect, and $Var$ and $Cov$ are their variances and covariance. The interval built from this variance is symmetric around $\hat{R}$.

When the true incremental effect is not zero, the ICER estimator is still asymptotically normal, and the delta method relies on exactly that; the difficulty is that convergence is slow when the effect difference is small relative to its standard error, so in trials of realistic size the symmetric interval can be a poor guide. If the estimated effect difference has positive probability density at zero, as a normally distributed difference does, the ratio estimator has no finite mean or variance in any finite sample, so the delta-method variance describes the limiting distribution only. A negative ICER is also ambiguous, since it arises both for a cheaper, more effective option and for a dearer, less effective one, in opposite quadrants of the cost-effectiveness plane.

Polsky and colleagues compared four methods by simulation and found that the bootstrap and Fieller intervals had miscoverage probabilities closer to the nominal level than the box and Taylor series methods, with the Taylor series method underestimating the upper limit. Stinnett and Mullahy argued that confidence intervals for cost-effectiveness ratios have theoretical limitations that make them inappropriate in many situations, and proposed net health benefit as the alternative.

Worked example with one trial and three intervals

The figures are illustrative. A trial randomises 250 patients to each arm. The new treatment costs £2,000 more on average, with a per-patient standard deviation of £6,708 in each arm, and yields 0.10 more QALYs, with a per-patient standard deviation of 0.447. The cost and QALY differences are assumed uncorrelated.

1. Standard errors from the central limit theorem.

$$SE(\Delta \hat{C}) = \sqrt{\frac{6708^2}{250} + \frac{6708^2}{250}} \approx 600$$

where $SE(\Delta \hat{C})$ is the standard error of the cost difference in pounds, 6,708 is the standard deviation of cost in each arm and 250 is the number of patients per arm. The same calculation with 0.447 gives a standard error of the QALY difference of about 0.04.

2. Delta-method interval. The ICER is £2,000 / 0.10 = £20,000 per QALY, and with zero covariance the interval is as follows.

$$SE(\hat{R}) = 20000 \times \sqrt{\left(\frac{600}{2000}\right)^2 + \left(\frac{0.04}{0.10}\right)^2} = 20000 \times \sqrt{0.25} = 10000$$

where $SE(\hat{R})$ is the delta-method standard error of the ICER in pounds per QALY. The approximate 95% interval is £20,000 ± 1.96 × £10,000, from £400 to £39,600 per QALY.

3. Fieller's interval. The Fieller method applies the normal approximation to the two differences rather than to their ratio. Its limits are the values of $R$ that solve the following equation.

$$(\Delta \hat{C} - R\,\Delta \hat{E})^2 = z^2 \left[ Var(\Delta \hat{C}) + R^2\,Var(\Delta \hat{E}) - 2R\,Cov(\Delta \hat{C}, \Delta \hat{E}) \right]$$

where $R$ is a candidate ICER, $z$ is 1.96 and the other terms are as above. With the example's figures (variances 600² and 0.04², zero covariance) this is the quadratic $a R^2 - 400 R + 2617024 = 0$ with $a = 0.0038534$, whose roots give a 95% interval of about £7,017 to £96,786 per QALY.

4. Net benefit at an illustrative threshold of £30,000 per QALY. The incremental net benefit is linear in the two differences, so no ratio is involved.

$$INB = 30000 \times 0.10 - 2000 = 1000$$

where $INB$ is the incremental net monetary benefit in pounds. Its variance is 30,000² × 0.04² + 600² = 1,440,000 + 360,000 = 1,800,000, so its standard error is about £1,342, the 95% interval runs from about −£1,630 to £3,630, and the normal approximation gives a probability of about 0.77 that net benefit is positive, which is one point on the acceptability curve.

5. Interpretation. The delta method puts the upper limit at £39,600, whereas Fieller's interval reaches almost £97,000, because a much smaller QALY gain is still plausible and would push the ratio up sharply; this is the pattern Polsky and colleagues reported. Fieller and net benefit agree, since the Fieller limits are the thresholds at which the net benefit interval touches zero (at £7,017 per QALY net benefit is about −£1,298, equal in size to 1.96 times its standard error), and £30,000 lies inside Fieller's interval while the net benefit interval at £30,000 includes zero.

Had the QALY standard error been 0.06, the effect difference would not be significant at the 5% level (0.10 / 0.06 is about 1.67). The delta method would still return a tidy interval, about −£6,296 to £46,296, but Fieller's set would become two unbounded pieces, every value below about −£110,622 and every value above about £6,177. The unbounded set is the honest answer, because the data cannot place an upper limit on the ICER.

Alternatives used in economic evaluation

The bootstrap, Fieller's method and net benefit regression still rest on large-sample arguments, so the choice among them is about where the approximation is applied; a parametric Bayesian model avoids it, at the price of a distributional assumption.

The non-parametric bootstrap resamples patients with replacement and recalculates the statistic many times. Briggs, Wonderling and Mooney noted that percentile bootstrap intervals for the ICER do not depend on parametric assumptions about its sampling distribution, while cautioning that bootstrap estimates of bias and standard error can be unstable. For skewed mean costs in moderately sized samples, Barber and Thompson preferred the bootstrap-t and bias-corrected and accelerated percentile methods. Fieller's method, applied to cost-effectiveness ratios by Willan and O'Brien, keeps the normal approximation but places it on the differences, which are averages. Net benefit regression, set out by Hoch, Briggs and Willan, places cost-effectiveness in a standard regression framework, which allows adjustment for imperfect randomisation and the identification of subgroups.

Bayesian analysis replaces the sampling distribution with a posterior distribution, which under a fully specified parametric model is exact given the model and the prior rather than a large-sample approximation.

Assumptions and common misunderstandings

Each application inherits the conditions of a limit result. Observations should be independent, or their dependence modelled, as with cluster-robust standard errors in a cluster-randomised trial; variances should be finite; and the sample should be large relative to the skewness and the number of parameters. For maximum likelihood, a true parameter value on the boundary of the parameter space, such as a variance component equal to zero, breaks the standard result, and close to the boundary the approximation is merely poor; a misspecified model makes the model-based variance-covariance matrix unreliable. Asymptotic normality also differs from the law of large numbers, which says the estimate converges to the true value but not how its distribution looks on the way.

A common error is to believe the data must be normal; the approximation concerns the estimator's sampling distribution, so skewed costs are acceptable in large samples, although a few very expensive patients can leave a modest trial far from the limit. A second is to make the normal a default distribution in probabilistic models, where a probability or utility near a bound can take impossible values unless its standard error is small. Finally, a normal-approximation interval describes sampling uncertainty in one dataset only, not the choice of data source, model structure or extrapolation, which NICE's manual treats as separate sources of uncertainty.

Sources

• Barber JA, Thompson SG. Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. Statistics in Medicine. 2000;19(23):3219-3236.
• Barber J, Thompson S. Multiple regression of cost data: use of generalised linear models. Journal of Health Services Research and Policy. 2004;9(4):197-204.
• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force-6. Value in Health. 2012;15(6):835-842.
• Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Economics. 1997;6(4):327-340.
• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press; 2006.
• Casella G, Berger RL. Statistical Inference. 2nd ed. Pacific Grove, CA: Duxbury; 2002.
• Hoch JS, Briggs AH, Willan AR. Something old, something new, something borrowed, something blue: a framework for the marriage of health econometrics and cost-effectiveness analysis. Health Economics. 2002;11(5):415-430.
• Latimer N. NICE DSU Technical Support Document 14: Survival analysis for economic evaluations alongside clinical trials, extrapolation with patient-level data. Sheffield: Decision Support Unit; 2011, updated March 2013. Section 3.7.
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Section 4.7.
• Nixon RM, Wonderling D, Grieve RD. Non-parametric methods for cost-effectiveness analysis: the central limit theorem and the bootstrap compared. Health Economics. 2010;19(3):316-333.
• O'Hagan A, Stevens JW. Assessing and comparing costs: how robust are the bootstrap and methods based on asymptotic normality? Health Economics. 2003;12(1):33-49.
• Polsky D, Glick HA, Willke R, Schulman K. Confidence intervals for cost-effectiveness ratios: a comparison of four methods. Health Economics. 1997;6(3):243-252.
• Stinnett AA, Mullahy J. Net health benefits: a new framework for the analysis of uncertainty in cost-effectiveness analysis. Medical Decision Making. 1998;18(2 Suppl):S68-S80.
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200.
• Willan AR, O'Brien BJ. Confidence intervals for cost-effectiveness ratios: an application of Fieller's theorem. Health Economics. 1996;5(4):297-305.