Bootstrap Standard Error

Bootstrap Standard Error for Skewed Trial Costs and Net Benefit

The bootstrap standard error replaces a variance formula with computation, so that the observed distribution of the data, rather than an assumed normal shape, determines how precisely an estimate is known. In trial-based economic evaluation it is used mainly for mean costs, incremental costs and incremental net benefit, where costs are skewed. Resampling in general and bootstrap intervals belong to the wider topic of bootstrapping.

From replicate estimates to a standard error

The standard error of an estimate is the standard deviation of its sampling distribution, which depends on the unknown population distribution. The bootstrap puts the empirical distribution of the data, with mass $\frac{1}{n}$ on each observation, in its place. For most statistics the result is then evaluated by Monte Carlo sampling, as follows.

$$\widehat{SE}_B = \sqrt{\frac{1}{B-1}\sum_{b=1}^{B}\left(\hat{\theta}^*_b - \bar{\theta}^*\right)^2}, \qquad \bar{\theta}^* = \frac{1}{B}\sum_{b=1}^{B}\hat{\theta}^*_b$$

where $\widehat{SE}_B$ is the bootstrap standard error from $B$ replicates, $\hat{\theta}^*_b$ is the statistic recalculated on bootstrap sample $b$, drawn with replacement and of the same size $n$ as the data, and $\bar{\theta}^*$ is the mean of the replicates. As $B$ grows, $\widehat{SE}_B$ approaches the ideal bootstrap standard error, the value over every possible resample. The replicate standard deviation is itself the standard error and is not divided by $\sqrt{B}$. In a spreadsheet, STDEV.S applied to the replicate column gives it directly, since Microsoft documents that the function uses the n-1 method.

The sample mean as a check on the method

For a sample mean the ideal bootstrap standard error has a closed form, derived by Efron and Tibshirani. It shows exactly how the bootstrap relates to the textbook formula. The two expressions are given below.

$$\widehat{SE}_{\infty}(\bar{x}) = \frac{1}{n}\sqrt{\sum_{i=1}^{n}\left(x_i - \bar{x}\right)^2} = \sqrt{\frac{n-1}{n}}\;\frac{s}{\sqrt{n}}$$

where $\widehat{SE}_{\infty}(\bar{x})$ is the ideal bootstrap standard error of the mean, $x_i$ is observation $i$, $\bar{x}$ is the sample mean, $n$ is the sample size and $s$ is the sample standard deviation with divisor $n-1$. With 50 patients the factor $\sqrt{(n-1)/n}$ is about 0.990, the kind of difference Efron and Tibshirani describe as too small to matter in most applications. With six patients it is about 0.913, so the bootstrap understates the usual standard error by roughly 9% before skewness is even considered.

Why trial-based evaluations apply it to costs

Costs collected in trials are usually right-skewed, with a tail of patients who incur high costs through complications or long stays. Thompson and Barber argue that the arithmetic mean is still the informative measure, because mean cost multiplied by the number of patients gives the budget a decision maker needs; medians and geometric means do not. The bootstrap is attractive because it uses the observed cost distribution instead of assuming normality.

Barber and Thompson recommended bootstrap methods either as a check on standard parametric methods or as the primary analysis for moderately sized samples of highly skewed costs. Their BMJ paper adds that t test methods were adequate in most realistic situations. In a menorrhagia trial with 78 and 70 patients per group, 95% intervals for a mean cost difference of £320 were £204 to £437 by a t test method and £192 to £426 by the bootstrap. Nixon, Wonderling and Grieve found that bootstrap and central limit theorem standard errors for incremental net benefit were both accurate with more than 50 patients, and that with small, highly skewed samples the central limit theorem was slightly more accurate. They concluded that either method is generally appropriate, but that the central limit theorem is easier to implement and gives standard errors at least as accurate as the bootstrap. Barber and Thompson's option of the bootstrap as the primary analysis for moderately sized, highly skewed samples still stands, and the bootstrap remains the route to a standard error for statistics with no simple formula.

Resampling costs and effects together within each arm

The bootstrap has to copy the design that generated the data. Briggs and Gray describe sampling $n_A$ cost and effect pairs with replacement from arm A, $n_B$ pairs from arm B, and then calculating the statistic. Resampling within arms keeps the randomised group sizes, and resampling pairs keeps each patient's cost with that patient's outcome. Pairing matters because the correlation enters the variance of incremental net monetary benefit, as shown below.

$$\operatorname{Var}^*(\widehat{INMB}) = \lambda^2\operatorname{Var}^*(\Delta\bar{E}) + \operatorname{Var}^*(\Delta\bar{C}) - 2\lambda\operatorname{Cov}^*(\Delta\bar{E}, \Delta\bar{C})$$

where $\operatorname{Var}^*$ and $\operatorname{Cov}^*$ are the variance and covariance across bootstrap replicates, $\Delta\bar{E}$ and $\Delta\bar{C}$ are the differences in mean effect and mean cost between arms, and $\lambda$ is the value of one unit of effect. Resampling costs and effects separately forces the covariance to zero. When costlier patients have worse outcomes the covariance is negative, so separate resampling understates the standard error.

Which statistics a bootstrap standard error suits

The method is most dependable for statistics that behave like averages, such as mean costs, mean quality-adjusted life years (QALYs), their differences and net benefit. For these a normal interval of estimate plus or minus 1.96 standard errors, centred on the original estimate, is a reasonable first summary, though Efron and Tibshirani note that this standard interval is sometimes good and sometimes not.

The incremental cost-effectiveness ratio is the main exception. Briggs and Gray point out that there may be a non-negligible probability of a zero or near-zero denominator, so the moments of the ratio may be undefined, and an undefined variance cannot be estimated by resampling. Briggs, Wonderling and Mooney found percentile intervals for the ratio promising, but reported that successive bootstrap estimates of its standard error and bias may be unstable as replicates are added. Uncertainty about the ratio is better shown on the cost-effectiveness plane or by Fieller intervals.

How many replicates a standard error needs

A standard error computed from a finite number of replicates carries simulation noise on top of its sampling variability. Efron and Tibshirani quantify that extra noise. Their approximation for the coefficient of variation of $\widehat{SE}_B$ is as follows.

$$CV(\widehat{SE}_B) \approx \sqrt{CV(\widehat{SE}_{\infty})^2 + \frac{E(\hat{\delta}) + 2}{4B}}$$

where $CV(\widehat{SE}_B)$ is the coefficient of variation of the standard error from $B$ replicates, $CV(\widehat{SE}_{\infty})$ is that of the ideal bootstrap standard error, typically 0.10 to 0.30 in their account, and $E(\hat{\delta})$ is the expected excess kurtosis of the bootstrap distribution (zero for a normal distribution). With the kurtosis term set to zero, as for a normal distribution, and $CV(\widehat{SE}_{\infty}) = 0.10$, 100 replicates give the following.

$$CV(\widehat{SE}_{100}) \approx \sqrt{0.10^2 + \frac{2}{400}} = \sqrt{0.015} \approx 0.122$$

where the terms are as defined above. The simulation adds little, which is why Efron and Tibshirani judge 50 to 200 replicates adequate for a standard error in most situations, against a rough minimum of 1,000 for bias-corrected intervals and about 250 for a percentile interval. In trial analyses the count is set by the most demanding output; Briggs and Gray used 1,000 replicates for cost differences and 5,000 for the cost-effectiveness plane. Simulation noise in model outputs is covered under Monte Carlo error.

Worked example with six patients per arm, paired and unpaired

The figures below are illustrative. A trial compares a new treatment (T) with usual care (U), six patients per arm. Costs are right-skewed, and the costliest patient in each arm has the lowest QALYs.

1. Point estimates. Mean costs are £2,000 (T) and £1,000 (U); mean QALYs are 0.75 and 0.70. The incremental cost is £1,000 and the incremental effect 0.05 QALYs, a ratio of £20,000 per QALY. At an illustrative threshold of £30,000 per QALY, incremental net monetary benefit is £500, from 30,000 × 0.05 − 1,000 = 500.

2. Mean costs. Squared cost deviations sum to 10,000,000 in arm T and 580,000 in arm U.

$$\widehat{SE}_{\infty}(\bar{C}_T) = \frac{\sqrt{10{,}000{,}000}}{6} \approx 527.0, \qquad \widehat{SE}_{\infty}(\bar{C}_U) = \frac{\sqrt{580{,}000}}{6} \approx 126.9$$

where $\bar{C}_T$ and $\bar{C}_U$ are the arm mean costs in pounds. The usual $s/\sqrt{n}$ gives £577.4 and £139.0, larger by the expected factor of 1/0.913.

3. Incremental cost. The arms are resampled independently, so their variances add.

$$\widehat{SE}_{\infty}(\Delta\bar{C}) = \sqrt{527.0^2 + 126.9^2} \approx 542.1$$

where $\Delta\bar{C}$ is the difference in mean cost. The analytic value allowing unequal variances is £593.9.

4. Net benefit, paired. Each patient's net benefit at $\lambda = 30{,}000$ is $\lambda \times \text{QALYs} - \text{cost}$, giving £23,600, £22,800, £22,000, £21,200, £20,200 and £13,200 in arm T (mean £20,500), and £21,600, £20,800, £20,100, £20,000, £19,300 and £18,200 in arm U (mean £20,000). Squared deviations sum to 71,020,000 and 6,940,000 respectively.

$$\widehat{SE}_{\infty}(\widehat{INMB}) = \frac{\sqrt{71{,}020{,}000 + 6{,}940{,}000}}{6} \approx 1471.6$$

where $\widehat{INMB}$ is the estimated incremental net monetary benefit in pounds.

5. Net benefit, unpaired. Separate resampling keeps the variances but drops the covariance. Squared QALY deviations sum to 0.031 (T) and 0.004 (U), contributing 30,000 × 30,000 × 0.031 = 27,900,000 and 30,000 × 30,000 × 0.004 = 3,600,000 respectively.

$$\widehat{SE}_{\infty, \text{unpaired}}(\widehat{INMB}) = \frac{\sqrt{27{,}900{,}000 + 10{,}000{,}000 + 3{,}600{,}000 + 580{,}000}}{6} \approx 1081.2$$

where the subscript marks the incorrect unpaired scheme. The result is about 26.5% too small, because the negative cost and QALY correlation, driven by the costliest patient in arm T, is lost.

6. Interpretation. The paired standard error of £1,471.6 is almost three times the estimate of £500. A normal interval of 500 ± 1.96 × 1,471.6 runs from about -£2,384 to £3,384, so the data cannot say whether adoption is worthwhile at this threshold. With six patients per arm and one extreme cost, both the normal approximation and the bootstrap are weak, so the example shows mechanics only; in practice $B$ Monte Carlo replicates approximate these ideal values.

Small samples, clusters and missing data

The bootstrap treats the sample as the population, so its standard error is only as good as the sample's picture of costs. In small trials it is also biased downwards, by the factor shown above for the mean.

Flynn and Peters studied two cluster bootstrap methods, one resampling whole clusters within each arm and one resampling clusters and then individuals, and found 86.4% to 93.8% coverage for nominal 95% intervals, against 93.5% to 94.8% for a cluster-adjusted robust standard error; they suggested about 24 clusters per arm as the minimum before considering the bootstrap instead. Gomes and colleagues, studying adjustment for systematic covariate imbalance in trials using cluster randomisation, found that multilevel models gave close to nominal coverage even with few clusters of unequal size.

With missing data, Brand and colleagues compared ten ways of combining multiple imputation and the bootstrap for skewed cost-effectiveness trial data. Single imputation with added noise, nested in the bootstrap percentile method, had the best statistical properties but needed long computation, while the same approach within the bootstrap-t method gave intervals that were too narrow. A standard unpaired t-test with standard multiple imputation and no bootstrap performed acceptably as an alternative.

Bootstrap standard error beside other precision measures

Several methods estimate precision and differ in what they assume. The bootstrap needs fewest distributional assumptions but depends on a representative sample and a resampling scheme that copies the design.

The bootstrap standard error describes sampling uncertainty in a trial, whereas the input distributions of a probabilistic sensitivity analysis represent parameter uncertainty in a decision model.

Where bootstrap standard errors are misread

A common spreadsheet error divides the replicate standard deviation by $\sqrt{B}$, giving the simulation error of the replicate mean, which shrinks towards zero as replicates are added. Another centres the normal interval on the replicate mean instead of the original estimate.

Sources

• Barber JA, Thompson SG. Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. Statistics in Medicine. 2000;19(23):3219-3236.
• Brand J, van Buuren S, le Cessie S, van den Hout W. Combining multiple imputation and bootstrap in the analysis of cost-effectiveness trial data. Statistics in Medicine. 2019;38(2):210-220.
• Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technology Assessment. 1999;3(2). Chapters 4 and 5.
• Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Economics. 1997;6(4):327-340.
• Efron B, Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science. 1986;1(1):54-75. Sections 1, 2 and 9.
• Flynn TN, Peters TJ. Use of the bootstrap in analysing cost data from cluster randomised trials: some simulation results. BMC Health Services Research. 2004;4:33.
• Gomes M, Grieve R, Nixon R, Ng ES, Carpenter J, Thompson SG. Methods for covariate adjustment in cost-effectiveness analysis that use cluster randomised trials. Health Economics. 2012;21(9):1101-1118.
• Microsoft. STDEV.S function. Microsoft Support. Accessed 1 October 2026.
• Nixon RM, Wonderling D, Grieve RD. Non-parametric methods for cost-effectiveness analysis: the central limit theorem and the bootstrap compared. Health Economics. 2010;19(3):316-333.
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200.