Bootstrapping

Bootstrapping Trial Data to Show Cost-Effectiveness Uncertainty

In health economics, bootstrapping is used mainly to express sampling uncertainty in cost-effectiveness analyses run alongside randomised trials. Each patient's cost and outcome are resampled together within each trial arm, incremental costs, incremental quality-adjusted life years (QALYs) and net benefit are recalculated thousands of times, and the spread of those results gives confidence intervals, a scatter on the cost-effectiveness plane and an acceptability curve without assuming that skewed costs follow a normal distribution. Intervals for the ratio itself can mislead, so net benefit and acceptability curves do much of the work. The standard error from replicates has its own page, bootstrap standard error.

The observed sample as a stand-in for the population

Efron introduced the bootstrap in 1979 as a general way to estimate the sampling distribution of a statistic from one observed sample. The empirical distribution, which puts probability $\frac{1}{n}$ on each observation, takes the place of the unknown population, and samples of size $n$ are drawn from it with replacement. The statistic is recalculated on each sample as follows.

$$\hat{\theta}^*_b = t(x^*_{1b}, \ldots, x^*_{nb}), \qquad b = 1, \ldots, B$$

where $\hat{\theta}^*_b$ is the estimate from bootstrap sample $b$, $t$ is the estimator applied to the original data, $x^*_{ib}$ is the $i$th observation drawn into sample $b$, $n$ is the original sample size and $B$ is the number of replicates. Because drawing is with replacement, some patients appear several times in a replicate and others not at all. The non-parametric bootstrap distribution is conditional on the data, so it can only recombine what the trial observed; a rare catastrophic cost that no patient incurred cannot appear in any replicate. A parametric bootstrap instead simulates data from a fitted model, so it can produce unobserved values but inherits any error in that model.

Resampling a two-arm trial in cost and effect pairs

Briggs, Wonderling and Mooney set out non-parametric bootstrap confidence intervals for the incremental cost-effectiveness ratio using patient-level data from a clinical trial, noting that such intervals do not depend on parametric assumptions about the ratio's sampling distribution. Briggs and Gray describe the procedure in three stages, which are to sample $n_A$ cost and effect pairs with replacement from arm A, sample $n_B$ pairs from arm B, then calculate the statistic, repeating many times. Replicate $b$ gives the following quantities.

$$\Delta C^*_b = \bar{C}^*_{T,b} - \bar{C}^*_{U,b}, \qquad \Delta E^*_b = \bar{E}^*_{T,b} - \bar{E}^*_{U,b}, \qquad INMB^*_b(\lambda) = \lambda\,\Delta E^*_b - \Delta C^*_b$$

where $\bar{C}^*_{T,b}$ and $\bar{C}^*_{U,b}$ are mean costs in the resampled treatment and usual-care arms, $\bar{E}^*_{T,b}$ and $\bar{E}^*_{U,b}$ are mean QALYs, $\Delta C^*_b$ and $\Delta E^*_b$ are the incremental cost and effect, $\lambda$ is the value placed on one QALY and $INMB^*_b(\lambda)$ is incremental net monetary benefit. Drawing within each arm holds the randomised group sizes fixed, and drawing whole patients carries the correlation between each patient's cost and outcome into every replicate.

Percentile, bias-corrected and BCa intervals

Efron and Tibshirani present three related ways of turning replicates into a confidence interval. The percentile method takes the interval from $\hat{G}^{-1}(\alpha)$ to $\hat{G}^{-1}(1-\alpha)$, the lower and upper $\alpha$ quantiles of the replicates. The bias-corrected (BC) and bias-corrected and accelerated (BCa) methods also read off percentiles of the replicates, but move the cut points as follows.

$$\hat{\theta}_{BC_a}[\alpha] = \hat{G}^{-1}\left(\Phi\left\{z_0 + \frac{z_0 + z^{(\alpha)}}{1 - a\,(z_0 + z^{(\alpha)})}\right\}\right), \qquad z_0 = \Phi^{-1}\{\hat{G}(\hat{\theta})\}$$

where $\hat{\theta}_{BC_a}[\alpha]$ is the interval endpoint that leaves probability $\alpha$ in its tail (a central interval leaves $\alpha$ in each tail), $\hat{G}$ is the cumulative distribution of the replicates, $\Phi$ is the standard normal cumulative distribution, $z^{(\alpha)}$ is its $\alpha$ quantile, $\hat{\theta}$ is the original estimate, $z_0$ is the bias-correction constant and $a$ is the acceleration constant. With $a = 0$ the formula reduces to the BC endpoint $\hat{G}^{-1}(\Phi\{2z_0 + z^{(\alpha)}\})$, and if exactly half the replicates fall below $\hat{\theta}$ then $z_0 = 0$ and BC equals the percentile interval. Efron and Tibshirani treat about 1,000 replicates as a rough minimum for BC or BCa intervals, against about 250 for a useful percentile interval. For mean trial costs, Barber and Thompson preferred the bootstrap-t, its variance-stabilised form and the BCa method.

Why bootstrap intervals for the ICER break down

The ratio of two differences is where naive bootstrapping goes wrong. Briggs and Gray note that ranking replicate ICERs for percentile methods fails because a negative ratio from a cost saving cannot be told apart from a negative ratio from a health loss, and that where this happens it may undermine the validity of the approach. They also point out that the ICER is only meaningful when incremental cost and effect share the same sign. In simulation studies, Briggs, Mooney and Wonderling found that methods assuming a normal sampling distribution performed poorly, as did combining separate intervals for costs and effects, and advised avoiding normal approximations when the sampling distribution is suspected to be skewed.

Net benefit avoids the ratio, because $INMB^*_b(\lambda)$ is defined for every replicate, whatever its quadrant, so its percentiles stay interpretable. Where an interval for the ratio itself is wanted, Fieller's method is the main parametric alternative.

From replicates to the plane and the acceptability curve

Each replicate gives one point $(\Delta E^*_b, \Delta C^*_b)$ on the cost-effectiveness plane. Briggs and Gray proposed using the replicates as an estimate of the joint distribution and, for each ceiling ratio, counting the share that falls on the acceptable side. They listed two advantages over integrating a joint normal density, namely that the approach is easier and that it makes no parametric assumption about the joint distribution. The estimated probability that treatment is cost-effective is given below.

$$\hat{P}(\lambda) = \frac{1}{B}\sum_{b=1}^{B} \mathbf{1}\left[INMB^*_b(\lambda) > 0\right]$$

where $\hat{P}(\lambda)$ is the estimated probability at value $\lambda$ per QALY and $\mathbf{1}[\cdot]$ equals 1 when the condition holds and 0 otherwise. Plotting $\hat{P}(\lambda)$ across values of $\lambda$ gives the cost-effectiveness acceptability curve. Löthgren and Zethraeus defined the curve formally through net benefit and related it to confidence intervals and P values. Fenwick, O'Brien and Briggs add reading rules. The curve meets the vertical axis at the share of points in the cost-saving quadrants, as the value per QALY rises it levels off at the share in the health-gaining quadrants, and it is not a cumulative distribution function.

Worked example with eight patients per arm and 2,000 replicates

The data are illustrative. A trial compares a new treatment (T) with usual care (U), eight patients per arm; patient 7 in arm T has a long admission.

1. Observed results. Means are GBP 3,650 and 0.775 QALYs (T) against GBP 2,300 and 0.715 (U). The incremental cost is GBP 1,350 and the incremental effect 0.06 QALYs, an ICER of 1,350 / 0.06 = 22,500 GBP per QALY. Net benefit is 20,000 × 0.06 − 1,350 = −150 GBP at 20,000 GBP per QALY and 30,000 × 0.06 − 1,350 = 450 GBP at 30,000 GBP per QALY.

2. One replicate. Arm T draws patients 1, 5, 5, 7, 1, 2, 5, 5, so patient 5 appears four times and patients 3, 4, 6 and 8 are absent; arm U draws 7, 5, 6, 7, 7, 5, 2, 5. Mean costs are GBP 3,550 and GBP 2,575 and mean QALYs 0.78 and 0.69, so the replicate gives an incremental cost of GBP 975, an incremental effect of 0.09 and net benefit of 30,000 × 0.09 − 975 = 1,725 GBP.

3. Intervals. Over 2,000 replicates (fixed random seed), the 95% percentile interval for net benefit at 30,000 GBP per QALY runs from −2,975 to 3,262 GBP. Because 49.7% of replicates fall below 450, $z_0 = \Phi^{-1}(0.497) = -0.0075$, so the BC cut points move to $\Phi(2 \times -0.0075 - 1.960) = 0.0241$ and 0.9741, giving −2,984 to 3,238 GBP.

4. Quadrants. 1,873 replicates lie north-east; 121 show a QALY loss and 6 show none, so 127 lack a meaningful ICER. Percentiles of the raw replicate ratios run from about −168,900 to 221,100 GBP per QALY, a figure that mixes quadrants and means nothing.

5. Acceptability curve. The share of positive net benefit is 0 at a value of zero (no replicate saves money), 0.177 at 10,000, 0.463 at 20,000, 0.606 at 30,000, 0.742 at 50,000 and 0.855 at 100,000 GBP per QALY. The curve can rise no higher than 1,873 / 2,000 = 0.9365, the share that gains QALYs.

6. Interpretation. At 30,000 GBP per QALY the point estimate favours treatment, but the probability of cost-effectiveness is only 0.606 and the net benefit interval spans zero. With eight patients per arm the replicates show the mechanics, not a reliable decision.

Clusters and missing data inside the resampling

The resampling has to copy the way the data were generated. In cluster randomised trials, Flynn and Peters kept the sampling structure by selecting clusters with replacement from each arm; resampling individuals instead would treat patients in the same practice or ward as independent. Their coverage results for few clusters are summarised on the bootstrap standard error page.

Missing data need the imputation inside the loop, so that each replicate is imputed afresh. Of ten designs Brand and colleagues tested on skewed cost-effectiveness data, imputing once per replicate with added noise and reading off percentiles performed best, though it was slow to compute. Any regression adjustment or multiple imputation step belongs inside each replicate for the same reason.

What bootstrap replicates do and do not represent

Briggs and Gray stress that statistical analysis summarises only uncertainty due to sampling variation. Unit costs not sampled in the trial, generalisability, extrapolation beyond follow-up and methodological choices still need sensitivity analysis. A trial bootstrap therefore differs from probabilistic sensitivity analysis, which samples parameter uncertainty in a decision model from specified distributions.

Several misreadings recur. Briggs, Wonderling and Mooney found successive bootstrap estimates of bias and standard error for the ratio unstable as replicates were added, and urged caution with them. Resampling costs and QALYs separately discards their correlation. More replicates reduce Monte Carlo error but add no information about patients, and an acceptability curve value of 0.606 is the share of replicates with positive net benefit, not the probability that the treatment works.

Sources

• Barber JA, Thompson SG. Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. Statistics in Medicine. 2000;19(23):3219-3236.
• Brand J, van Buuren S, le Cessie S, van den Hout W. Combining multiple imputation and bootstrap in the analysis of cost-effectiveness trial data. Statistics in Medicine. 2019;38(2):210-220.
• Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technology Assessment. 1999;3(2). Chapter 5.
• Briggs AH, Mooney CZ, Wonderling DE. Constructing confidence intervals for cost-effectiveness ratios: an evaluation of parametric and non-parametric techniques using Monte Carlo simulation. Statistics in Medicine. 1999;18(23):3245-3262.
• Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Economics. 1997;6(4):327-340.
• Efron B. Bootstrap methods: another look at the jackknife. Annals of Statistics. 1979;7(1):1-26.
• Efron B, Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science. 1986;1(1):54-75. Sections 7 and 9.
• Fenwick E, O'Brien BJ, Briggs A. Cost-effectiveness acceptability curves: facts, fallacies and frequently asked questions. Health Economics. 2004;13(5):405-415.
• Flynn TN, Peters TJ. Use of the bootstrap in analysing cost data from cluster randomised trials: some simulation results. BMC Health Services Research. 2004;4:33.
• Löthgren M, Zethraeus N. Definition, interpretation and calculation of cost-effectiveness acceptability curves. Health Economics. 2000;9(7):623-630.