Central Limit Theorem

Central Limit Theorem and the Analysis of Mean Costs and QALYs with Normal Methods

The central limit theorem is the reason a normal-theory confidence interval can be placed around the mean cost of a trial arm even when individual patient costs are heavily skewed. It is a statement about averages. As the number of independent patients grows, the sampling distribution of their mean cost, mean QALYs or mean net benefit comes closer to a normal shape, and its standard error falls in proportion to one over the square root of the sample size. Health economists rely on it for t-tests and intervals on arithmetic mean costs, for normal distributions in probabilistic models, and for judging how many simulations a probabilistic sensitivity analysis needs.

The theorem, from patient costs to the distribution of their mean

The theorem starts from a trial arm of $n$ patients whose costs are independent draws from one distribution with a finite mean and variance. It makes no claim about individual costs. Its classical form, as given in standard statistical texts such as Casella and Berger, concerns the standardised sample mean, as follows.

$$\frac{\sqrt{n}\,(\bar{X}_n - \mu)}{\sigma} \xrightarrow{d} N(0, 1)$$

where $n$ is the number of patients, $\bar{X}_n$ is their sample mean cost, $\mu$ is the population mean cost, $\sigma$ is the population standard deviation, $N(0, 1)$ is the standard normal distribution and $\xrightarrow{d}$ denotes convergence in distribution as $n$ grows. Nothing is assumed about the shape of the cost distribution beyond a finite variance.

Rearranged, the result says that in a large sample the mean is approximately normal with mean $\mu$ and variance $\sigma^2 / n$. The square root of that variance is the standard error of the mean, $\sigma / \sqrt{n}$, so quadrupling the number of patients halves the standard error. In practice $\sigma$ is unknown and is replaced by the sample standard deviation $s$, which does not change the large-sample result; the t distribution is the usual choice in smaller samples.

The theorem describes how the mean would vary across repeated samples, not how costs vary across patients. It is also a limit statement and gives no sample size at which the approximation becomes good enough.

Why economic evaluation leans on the theorem

Thompson and Barber argue that the arithmetic mean is the informative measure for cost data in pragmatic trials, because multiplying it by the number of patients to be treated gives the total budget a health service needs. They describe highly skewed distributions as typical of cost data, with a long right-hand tail reflecting the patients who incur high costs through complications, reoperation or extended hospital stay. Inference about arithmetic means of skewed data is exactly what the theorem supports.

A t-test on untransformed costs compares arithmetic means. Thompson and Barber note that it is strictly valid only for normal data but will be reliable if either the skewness is not too extreme or the sample sizes are moderately large. Their low back pain example shows what is at stake. Mean costs of GBP 360 and GBP 508 gave a t-based 95% interval for the GBP 148 difference of −GBP 146 to GBP 442 (P = 0.32), whereas a Mann-Whitney U test on the same data gave P = 0.02, which they say would have been extremely misleading had it been used.

The same reasoning carries into cost-effectiveness. The review by Mihaylova and colleagues lists the central limit theorem and the bootstrap as two common non-parametric approaches to uncertainty in trial costs and effects. Nixon, Wonderling and Grieve applied both to the incremental net benefit, which is a difference of means and so suits the theorem. In decision models, the ISPOR-SMDM task force on parameter estimation states that a parameter informed by plenty of data can be given a normal distribution in probabilistic sensitivity analysis because of the central limit theorem, with the arithmetic mean and standard error informing both the distribution and the confidence interval.

Conditions, and how fast skewed costs converge

Applying the theorem to cost data rests on three requirements, and cost data can strain each of them. The third, sample size relative to skewness, is the one most often misjudged.

Independence. Patients must contribute independent observations. In a trial randomised by practice or hospital, outcomes within a cluster are correlated, and the effective number of independent units falls between the number of clusters and the number of patients, depending on the intracluster correlation (see cluster randomisation and the design effect). Gomes and colleagues note that most cost-effectiveness analyses of cluster randomised trials use methods that ignore clustering. In their simulations seemingly unrelated regression without a robust (cluster-adjusted) standard error gave interval coverage of 0.89 against a nominal 0.95, and in their case study ignoring clustering underestimated statistical uncertainty and the value of further research.

Finite variance. The standard theorem needs a finite variance. Observed costs are bounded, so this holds in practice, but a very large variance driven by a few extreme patients leaves the mean unstable in samples of ordinary size.

Sample size relative to skewness. The skewness of the sample mean falls with the square root of the sample size, as follows.

$$\gamma_{\bar{X}} = \frac{\mu_3 / n^2}{(\sigma^2 / n)^{3/2}} = \frac{\mu_3}{\sigma^3 \sqrt{n}} = \frac{\gamma}{\sqrt{n}}$$

where $\gamma_{\bar{X}}$ is the skewness of the sample mean, $\mu_3$ is the third central moment of individual costs, $\mu_3 / n^2$ is the third central moment of the mean of $n$ independent costs, $\sigma^2 / n$ is the variance of the mean and $\gamma = \mu_3 / \sigma^3$ is the skewness of individual costs. Halving the skewness of the mean therefore takes four times as many patients, and cost distributions with a long tail need large samples before the mean looks symmetric.

Nixon, Wonderling and Grieve found that central limit theorem and bootstrap methods both estimated the true standard errors of incremental net benefit accurately when samples were moderate to large (more than 50 in their experiments), even with highly skewed costs, and that with small, highly skewed samples the central limit theorem gave slightly more accurate standard errors. O'Hagan and Stevens argued that even where the bootstrap and methods based on asymptotic normality of sample means are technically valid, they may give inefficient and even misleading inferences for skewed costs, and that methods recognising the skewness should be used. An accurate standard error does not guarantee that a symmetric interval covers the true mean at the stated rate, as the example below shows.

Worked example of rare expensive patients and interval coverage

The cost distribution and every number here are illustrative. In a trial arm, 90% of patients have an uncomplicated course costing GBP 500 and 10% have a complication costing GBP 10,000. The question is how often a standard 95% interval, the sample mean plus or minus 1.96 estimated standard errors, contains the true mean cost.

1. Population values. The true mean is 0.9 × 500 + 0.1 × 10,000 = GBP 1,450. The variance is 0.9 × 500² + 0.1 × 10,000² − 1,450² = 225,000 + 10,000,000 − 2,102,500 = 8,122,500, so the standard deviation is GBP 2,850. For a two-point distribution with probability 0.1 of the high value, the skewness is (1 − 0.2) / √(0.1 × 0.9) = 0.8 / 0.3 = 2.67.

2. Standard error and skewness of the mean. With 20 patients the standard error is 2,850 / √20 = GBP 637 and the skewness of the mean is 2.67 divided by the square root of 20, or 0.60; with 400 patients they are GBP 142.50 and 0.13.

3. One sample of 20, by hand. The chance that 20 patients include no complicated case is 0.9 raised to the power 20, or 0.122. Every cost in such a sample is GBP 500, the sample standard deviation is zero and the interval collapses to GBP 500, missing the true mean. A sample with exactly one complicated patient has mean 500 + 9,500 / 20 = GBP 975, and its standard deviation follows from the formula for $k$ complicated patients among $n$, as follows.

$$s = 9500 \sqrt{\frac{k(n - k)}{n(n - 1)}} = 9500 \sqrt{\frac{1 \times 19}{20 \times 19}} = 9500 \times 0.2236 = 2124$$

where $s$ is the sample standard deviation in pounds, $k$ is the number of complicated patients, $n$ is the sample size and 9,500 is the gap between the two costs. The standard error is 2,124 / √20 = GBP 475 and 1.96 × 475 = 931, so the interval runs from GBP 44 to GBP 1,906 and contains GBP 1,450.

4. Exact coverage across all samples. Repeating step 3 for every possible $k$ and weighting each result by its binomial probability gives the exact coverage of the nominal 95% interval.

5. Interpretation. Coverage nears 95% only at a few hundred patients, and even at 400 an interval is more than twice as likely to lie wholly below the true mean as above it. The misses come from samples with too few complicated patients, so the error understates cost and the budget built on it. The example shows how slowly the theorem takes hold when the mean depends on rare, expensive events.

Monte Carlo error in probabilistic models

A probabilistic sensitivity analysis reports expected costs, QALYs and net benefit as averages over its iterations, a standard method set out in decision modelling texts such as Briggs, Claxton and Sculpher. Each average is a Monte Carlo estimate with its own sampling error. Koehler, Brown and Haneuse define this Monte Carlo error as the standard deviation of the estimate across hypothetical repeats of the whole simulation, and use the central limit theorem to estimate it as follows.

$$\widehat{MCE}(\hat{\phi}_R) = \frac{\hat{\sigma}_{\phi}}{\sqrt{R}}$$

where $\hat{\phi}_R$ is the estimate from $R$ iterations, $\hat{\sigma}_{\phi}$ is the standard deviation of the output across iterations and $R$ is the number of iterations.

As an illustration, if incremental net benefit has a standard deviation of GBP 5,000 across iterations, 1,000 iterations give a Monte Carlo error of 5,000 / √1,000 = GBP 158 and 4,000 iterations give GBP 79. Halving the error takes four times the iterations, the same square-root rule as for patients in a trial.

Related results and where each applies

The law of large numbers says that the sample mean converges to the true mean; Koehler and colleagues use the central limit theorem for the size and shape of the error around it. Asymptotic normality extends the same large-sample reasoning to estimators other than simple means, such as regression coefficients and maximum likelihood estimates, and that page covers interval methods for the ICER, including the delta method and the Fieller method. The non-parametric bootstrap resamples the observed patients instead of assuming a normal sampling distribution. Methods that model the skewness directly, such as a generalised linear model or a Bayesian analysis with a skewed distribution, are the alternatives the Mihaylova review and O'Hagan and Stevens discuss.

Common misunderstandings

Most errors with the theorem come from applying it to the wrong object or the wrong unit. Four recur in economic evaluation.

1. Believing the costs must be normal. Skewed patient costs are no bar to a normal-theory confidence interval for the mean in a large sample, and log-transforming costs changes the quantity compared.
2. Treating a fixed sample size as always enough. No sample size works for all cost data; in the example, 50 patients per arm gave 88% coverage for a nominal 95% interval.
3. Counting rows instead of independent units. Repeated visits by one patient or patients within one cluster do not add independent information at the rate the row count suggests.
4. Applying the theorem directly to a ratio. The theorem applies directly to the two mean differences in an ICER; the ratio itself can stay far from normal in trials of realistic size, as the asymptotic normality page explains.

The theorem also controls only random sampling error. It does nothing about bias from an unrepresentative sample, missing cost data or a misspecified outcome, and a large trial with these problems can produce a tight interval around the wrong value.

Sources

• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press; 2006.
• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722-732. https://doi.org/10.1177/0272989X12458348
• Casella G, Berger RL. Statistical Inference. 2nd ed. Pacific Grove, CA: Duxbury; 2002.
• Gomes M, Ng ES, Grieve R, Nixon R, Carpenter J, Thompson SG. Developing appropriate methods for cost-effectiveness analysis of cluster randomized trials. Medical Decision Making. 2012;32(2):350-361. https://doi.org/10.1177/0272989X11418372
• Koehler E, Brown E, Haneuse SJ. On the assessment of Monte Carlo error in simulation-based statistical analyses. The American Statistician. 2009;63(2):155-162. https://doi.org/10.1198/tast.2009.0030
• Mihaylova B, Briggs A, O'Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health Economics. 2011;20(8):897-916. https://doi.org/10.1002/hec.1653
• Nixon RM, Wonderling D, Grieve RD. Non-parametric methods for cost-effectiveness analysis: the central limit theorem and the bootstrap compared. Health Economics. 2010;19(3):316-333. https://doi.org/10.1002/hec.1477
• O'Hagan A, Stevens JW. Assessing and comparing costs: how robust are the bootstrap and methods based on asymptotic normality? Health Economics. 2003;12(1):33-49. https://doi.org/10.1002/hec.699
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200. https://doi.org/10.1136/bmj.320.7243.1197