Signature
theta_bar = sum_(b=1)^B [theta_b] / B; SE_B = sqrt(sum_(b=1)^B [(theta_b - theta_bar)^2] / (B - 1))
| Inputs | Definition | Unit |
|---|---|---|
theta_b | Value of the statistic recalculated on bootstrap sample b, drawn with replacement and of the same size as the data | unit of the statistic |
B | Number of bootstrap samples drawn and replicate values computed | count |
theta_bar | Mean of the B replicate values | unit of the statistic |
|---|---|---|
SE_B | Bootstrap standard error estimated from B replicates | unit of the statistic, for example pounds |
Function
Bootstrap standard error of a trial statistic from resampled data
Maps a statistic and the data that produced it to the standard deviation of the statistic across samples drawn with replacement from the data, each of the same size as the original sample. The empirical distribution of the observations, with mass 1/n on each, stands in for the unknown population distribution, so no normal shape is assumed. In trial-based economic evaluation the statistic is usually a mean cost, an incremental cost or incremental net benefit, and the resampling copies the trial design by drawing patients within each arm with their costs and effects kept together.
Computational function
Computational function: bootstrap standard error and normal interval from a vector of replicates
Takes the vector of bootstrap replicate values produced by resampling, together with the estimate from the original data, and returns the bootstrap standard error and a normal interval centred on the original estimate. It applies HE-FM-BSE-001 to the whole vector and uses the result in a second step, so the inputs are the stored replicates and the original estimate rather than the single values the formula's variables hold.
Inputs and outputs:
theta_star: Vector of B replicate values of the statistic, for example incremental net benefit, each from one resample drawn within arms with each patient's cost and effect kept together; required, at least two values. Unit: unit of the statistic, for example pounds.;theta_hat: Value of the statistic in the original data; required. Unit: unit of the statistic.;z: Standard normal critical value, 1.96 for a 95% interval; required, above zero. Unit: none.;SE_B: Bootstrap standard error, the standard deviation of the replicates with divisor B minus 1. Unit: unit of the statistic.;L: Lower limit of the normal interval, theta_hat minus z times SE_B. Unit: unit of the statistic.;U: Upper limit of the normal interval, theta_hat plus z times SE_B. Unit: unit of the statistic.Assumption: The replicates come from resampling that copies the trial design, and the statistic behaves like an average, so a normal interval is a reasonable first summary. Efron and Tibshirani note that this standard interval is sometimes good and sometimes not, and it does not suit the incremental cost-effectiveness ratio, whose variance may be undefined.
Worked example (Five illustrative replicates around an estimate of 500): Five invented replicate values, far fewer than the 50 to 200 needed in practice, show the arithmetic. They are not drawn from the article's six-patient trial, whose own interval is about -2,384 to 3,384. Their mean of 600 differs from the original estimate of 500, and the interval is centred on 500.
theta_star = (600, -1400, 2600, 1100, 100); theta_hat = 500; z = 1.96; SE_B = 1457.74; L = -2357.17; U = 3357.17Worked example (Identical replicates give a zero-width interval): When every replicate equals the original estimate, the standard error is zero and both limits equal the estimate, a limiting case that checks the implementation.
theta_star = (500, 500, 500); theta_hat = 500; z = 1.96; SE_B = 0; L = 500; U = 500Excel:
=STDEV.S(Replicates)in a cell named SE, with=Estimate-ZValue*SEand=Estimate+ZValue*SEfor the limits. With the replicate values in a range named Replicates, the original estimate in Estimate and the critical value in ZValue, the three cells return the standard error and the interval.R:
boot_se_interval <- function(reps, estimate, z = 1.96) { se <- sd(reps); c(se = se, lower = estimate-z*se, upper = estimate+z*se) }The R function sd uses the divisor B minus 1, so no further correction is needed;boot_se_interval(c(600, -1400, 2600, 1100, 100), 500)reproduces the first example.Python:
def boot_se_interval(reps, estimate, z=1.96): se = statistics.stdev(reps); return se, estimate-z*se, estimate+z*seUses the statistics module, whose stdev function also uses the divisor B minus 1.Test (Interval is symmetric about the original estimate): The midpoint of the two limits equals the original estimate, not the replicate mean. Expected result: TRUE. Excel check:
=ABS((Lower+Upper)/2-Estimate)<1E-9Test (Interval width equals twice z times the standard error): The distance between the limits is twice the critical value times the replicate standard deviation. Expected result: TRUE. Excel check:
=ABS((Upper-Lower)-2*ZValue*STDEV.S(Replicates))<1E-9Common error (Centring the interval on the replicate mean): Centring on the replicate mean of 600 instead of the original estimate of 500 moves the limits to about -2,257.17 and 3,457.17 instead of -2,357.17 and 3,357.17. The normal interval is the original estimate plus or minus z standard errors.
Source: Efron B, Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science. 1986;1(1):54-75. Section 1, equation 1.7, which centres the standard interval on the original estimate, and section 2, equation 2.4, which defines the bootstrap standard error from B replications.
SE_B = sd(theta_star); L = theta_hat - z * SE_B; U = theta_hat + z * SE_B
Try this function
Implementations
Excel
Bootstrap standard error from a replicate column
Microsoft documents that STDEV.S uses the n-1 method, so applied to the column of B replicate values it returns the bootstrap standard error directly.
=STDEV.S(Replicates)
Assumptions
Bootstrap samples copy the trial design
Each bootstrap sample has the same size as the data and is drawn with replacement in the way the data were generated: within each trial arm, keeping each patient's cost with the same patient's outcome.
Sample stands in for the population of costs
The bootstrap treats the sample as the population, so the standard error is only as good as the sample's picture of costs. With few patients or one extreme cost it is unreliable, and in small samples it is biased downwards.
Statistic with a finite variance that behaves like an average
The method is most dependable for statistics that behave like averages, such as mean costs, mean QALYs, their differences and net benefit. For the incremental cost-effectiveness ratio a zero or near-zero denominator can leave the variance undefined, and successive bootstrap estimates of its standard error may be unstable.
Worked examples
Five illustrative net benefit replicates
Five replicate values of incremental net benefit, far fewer than the 50 to 200 needed in practice, show the arithmetic: the mean is 600, the squared deviations sum to 8,500,000 and the standard error is about 1,457.74 pounds.
theta_b = [600, -1400, 2600, 1100, 100]; B = 5; theta_bar = 600; SE_B = 1457.74
Identical bootstrap replicates give a zero standard error
When every replicate takes the same value the standard error is zero, a limiting case that checks the implementation.
theta_b = [1000, 1000, 1000]; B = 3; theta_bar = 1000; SE_B = 0
Common errors
Dividing the replicate standard deviation by the square root of B
Dividing by the square root of B gives the simulation error of the replicate mean, which shrinks towards zero as replicates are added. With the five illustrative replicates it gives about 651.92 pounds instead of 1,457.74.
Using STDEV.P for the replicate standard deviation
STDEV.P divides by B rather than B minus 1. With the five illustrative replicates it gives about 1,303.84 pounds instead of 1,457.74; the gap shrinks as B grows, but only STDEV.S matches the definition.
Sources
Efron and Tibshirani Monte Carlo estimate of the bootstrap standard error
Efron B, Tibshirani R. Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical Science. 1986;1(1):54-75. Section 2, equation 2.4, which defines the estimate from B bootstrap replications with divisor B minus 1 and shows that it approaches the ideal bootstrap standard error as B grows.
Microsoft documentation of the STDEV.S divisor
Microsoft. STDEV.S function. Microsoft Support. Description, which states that the standard deviation is calculated using the n-1 method.
Briggs and Gray on undefined moments of the incremental cost-effectiveness ratio
Briggs AH, Gray AM. Handling uncertainty when performing economic evaluation of healthcare interventions. Health Technology Assessment. 1999;3(2). Chapter 5, which notes that there may be a non-negligible probability of a zero or near-zero denominator, so that the moments of the incremental cost-effectiveness ratio may be undefined.
Briggs, Wonderling and Mooney on unstable bootstrap standard errors for the ratio
Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Economics. 1997;6(4):327-340. Abstract, which reports that successive bootstrap estimates of the bias and standard error of the ratio may be unstable as replications increase.
Canonical Identity
Stable URI · Machine-readable · Resolvable · CC BY 4.0