Jackknife

Jackknife Estimation of Bias and Standard Error

The jackknife is a way of judging how far an estimate from patient data, such as a mean cost, a unit cost or a regression coefficient, can be trusted when no simple formula gives its standard error. It drops each observation in turn, recomputes the estimate, and reads bias and spread from how much the recomputed values move. Health economists meet it less often as a stand-alone method than as a component inside other tools, most visibly the bias-corrected and accelerated bootstrap interval for skewed cost data and the replicate weights supplied with complex survey datasets. The method is deterministic, so two analysts with the same data obtain the same answer. Its weaknesses are just as specific, since it fails for the median and its variance estimate tends to run high.

How leaving one observation out reveals bias and variance

Efron describes the Quenouille-Tukey jackknife as a nonparametric method for estimating the bias and variance of a statistic by recomputing it $n$ times, each time removing one observation. The logic is that an estimator whose value shifts a great deal when a single patient is removed is one whose value would also shift a great deal in a fresh sample.

$$\hat\theta_{(i)} = s(x_1, \dots, x_{i-1}, x_{i+1}, \dots, x_n), \qquad \bar\theta_{(\cdot)} = \frac{1}{n}\sum_{i=1}^{n} \hat\theta_{(i)}$$

where $x_1, \dots, x_n$ are the $n$ observations, $s(\cdot)$ is the rule that turns data into the estimate, $\hat\theta$ is the estimate from the full sample, $\hat\theta_{(i)}$ is the leave-one-out estimate with observation $i$ removed, and $\bar\theta_{(\cdot)}$ is the average of the $n$ leave-one-out estimates.

Bias correction and pseudo-values

If the leave-one-out estimates sit systematically above or below the full-sample estimate, the jackknife reads that gap as a sign of small-sample bias and scales it up by $n-1$. Stata's manual calls the resulting corrected value a first-order unbiased estimator, meaning that the leading bias term of order one over $n$ is removed but not every bias. The same quantities give each observation a pseudo-value, and Efron sets out both the corrected estimate and the pseudo-values in this form.

$$\widehat{\text{bias}}_{\text{jack}} = (n-1)\left(\bar\theta_{(\cdot)} - \hat\theta\right), \qquad \tilde\theta = \hat\theta - \widehat{\text{bias}}_{\text{jack}}, \qquad \tilde\theta_i = n\hat\theta - (n-1)\hat\theta_{(i)}$$

where $\widehat{\text{bias}}_{\text{jack}}$ is the jackknife estimate of bias, $\tilde\theta$ is the bias-corrected jackknife estimate, and $\tilde\theta_i$ is the pseudo-value for observation $i$. The mean of the pseudo-values equals $\tilde\theta$. For a sample mean the pseudo-values are the original observations, and the jackknife treats the pseudo-values of any estimate as if they were $n$ roughly independent observations whose mean is the corrected estimate.

The jackknife variance and standard error

Tukey's step, credited to him in the Stata manual, was to treat the standard error of the mean of the pseudo-values as the standard error of the estimate. Written with the leave-one-out values instead, the multiplier $(n-1)/n$ on a sum of squares is far larger than the $\frac{1}{n-1}$ of an ordinary variance, because any two leave-one-out samples share $n-2$ observations and their estimates therefore vary much less than independent estimates would.

$$\widehat{\text{Var}}_{\text{jack}} = \frac{n-1}{n}\sum_{i=1}^{n}\left(\hat\theta_{(i)} - \bar\theta_{(\cdot)}\right)^2 = \frac{1}{n(n-1)}\sum_{i=1}^{n}\left(\tilde\theta_i - \tilde\theta\right)^2$$

where $\widehat{\text{Var}}_{\text{jack}}$ is the jackknife variance and its square root is the jackknife standard error. The two forms are algebraically identical.

Where the jackknife appears in health economic work

Most uses in health economics are indirect. The method supplies a building block for another procedure, or it handles a design where a closed-form variance is awkward to obtain.

Inside the BCa bootstrap interval for costs

Barber and Thompson, analysing cost data from two randomised trials, recommended the bootstrap-t and the bias-corrected and accelerated (BCa) percentile methods for comparing arithmetic mean costs in moderately sized samples of highly skewed data. The acceleration constant in the BCa interval is computed from jackknife values. Stata's bootstrap manual gives the formula and calls it the jackknife estimate of acceleration.

$$a = \frac{\sum_{i=1}^{n}\left(\bar\theta_{(\cdot)} - \hat\theta_{(i)}\right)^3}{6\left\{\sum_{i=1}^{n}\left(\bar\theta_{(\cdot)} - \hat\theta_{(i)}\right)^2\right\}^{3/2}}$$

where $a$ is the acceleration, and the other symbols are as defined above. A few very expensive patients make the cubed deviations lopsided, so $a$ departs from zero, and the interval limits shift to reflect the skew. With this formula, a BCa interval for a trial-based cost difference or ICER rests on $n$ jackknife recomputations as well as the bootstrap replicates, which matters when each recomputation is slow.

Survey replicate weights for model inputs

Large surveys used to estimate model inputs often have stratified cluster designs, and leaving out one cluster at a time gives a jackknife variance that respects that design. Onyimadu and colleagues estimated transition probabilities between childhood weight status categories from the UK Millennium Cohort Study for use in cost-effectiveness models of weight management. The survey's design variables suited Taylor series linearisation, which did not work with their multistate model function, so they derived 398 jackknife replicate weights, matching the number of clusters in the survey, and used them to estimate variances.

Cost-effectiveness ratios and influence checks

Cook and Heyse proposed an angular transformation of the standardised cost-effectiveness ratio to stabilise its variance, noting that a 95% interval on the transformed scale is easily obtained with either the jackknife or the bootstrap. Stata's manual adds that jackknife pseudo-values remain useful for finding overly influential observations, something the bootstrap does not provide, and in cost data these are typically the few very expensive patients.

Choices that change the jackknife answer

The unit that is deleted must match the unit that was sampled. In a cluster randomised trial, or with patients nested within hospitals, deleting single patients treats correlated observations as independent and understates the standard error. Stata's cluster option leaves out one cluster per recomputation, with $n$ becoming the number of clusters.

The size of the deleted group also matters. Efron explains that the ordinary jackknife fails for the median because the statistic changes too irregularly for its extrapolation to work, while a grouped jackknife that removes observations in sufficiently large groups gives the correct asymptotic variance for the median.

A reference distribution has to be chosen before an interval can be formed. Tukey suggested treating the difference between the bias-corrected estimate and the true value, divided by the jackknife standard error, as Student's t with $n-1$ degrees of freedom. Efron regards this as optimistic, with theoretical support only in large samples, where the choice between t and normal reference values hardly matters.

Whether to apply the bias correction is a further choice. The correction is itself estimated with error, so reporting the corrected and uncorrected estimates together shows how much it matters.

Worked example with patient costs and a ward unit cost

All figures in this example are illustrative. Steps 1 to 4 apply the jackknife to a sample mean, where the answer is known, and steps 5 to 9 to a ratio.

1. Mean cost of five patients. Annual costs are GBP 800, 1,200, 1,500, 2,100 and 6,400, summing to GBP 12,000, so the mean is 12,000 / 5 = 2,400.

2. Leave-one-out means. Removing each patient in turn leaves four patients, giving (12,000 − 800) / 4 = 2,800, then 2,700, 2,625, 2,475 and (12,000 − 6,400) / 4 = 1,400. Their average is 12,000 / 5 = 2,400, so the estimated bias is 4 × (2,400 − 2,400) = 0. The jackknife correctly finds no bias in a sample mean.

3. Jackknife standard error. The deviations of the leave-one-out means from 2,400 are 400, 300, 225, 75 and minus 1,000. Their squares sum to 1,306,250, and (4/5) × 1,306,250 = 1,045,000, a standard error of GBP 1,022.25. The classical formula gives the sample variance 20,900,000 / 4 = 5,225,000 and 5,225,000 / 5 = 1,045,000, exactly the same. The pseudo-values, 5 × 2,400 − 4 × 2,800 = 800 and so on, reproduce the five original costs.

4. Acceleration for a BCa interval. The cubed differences between the average and each leave-one-out mean sum to 897,187,500, and 6 × 1,306,250 to the power 1.5 is about 8,957,580,000, so $a$ is about 0.10. The single patient costing GBP 6,400 drives this positive value, and a BCa interval would stretch further above the mean than below it.

5. Unit cost per bed day. Five wards report annual costs of GBP 900, 1,500, 1,200, 2,700 and 700 thousand and 2,000, 3,600, 2,500, 4,000 and 1,900 bed days. The pooled unit cost is total cost over total bed days, 7,000,000 / 14,000 = 500, or GBP 500 per bed day, a ratio of means with no simple exact standard error. Here the ward is the sampled unit, so each recomputation leaves out one ward.

6. Leave-one-ward-out ratios. Dropping each ward gives 6,100,000 / 12,000 = 508.33, 5,500,000 / 10,400 = 528.85, 5,800,000 / 11,500 = 504.35, 4,300,000 / 10,000 = 430.00 and 6,300,000 / 12,100 = 520.66. Their average is 498.44.

7. Bias and corrected estimate. The estimated bias is 4 × (498.44 − 500) = minus 6.25, so the corrected unit cost is 500 + 6.25 = GBP 506.25 per bed day.

8. Standard error and interval. The squared deviations of the five ratios from 498.44 are 97.92, 924.67, 34.93, 4,683.72 and 493.88, summing to 6,235.13. Then (4/5) × 6,235.13 = 4,988.10, a jackknife standard error of 70.63. Using Tukey's t with 4 degrees of freedom (2.776), the interval is 506.25 ± 2.776 × 70.63, or about GBP 310 to GBP 702 per bed day.

9. Interpretation. The pseudo-value for the fourth ward is 5 × 500 − 4 × 430 = 780, far above the others, which lie between about 385 and 483. That ward, at GBP 675 per bed day, contributes most of the variance, and with only five wards the interval is too wide to support a precise tariff. With so few units the t reference distribution is a rough guide only.

Jackknife, bootstrap and leave-one-out cross-validation

The jackknife and bootstrapping both reuse the observed data, but they sample differently. Efron notes that the bootstrap draws samples of size $n$ with replacement, while the ordinary jackknife can be thought of as drawing samples of size $n-1$ without replacement, and he shows the jackknife to be a linear approximation to the bootstrap. He describes bootstrap methods as more widely applicable and more dependable, and the Stata manual says the jackknife variance estimate has been largely replaced by the bootstrap. Briggs, Wonderling and Mooney, bootstrapping a trial ICER, found successive bootstrap estimates of bias and standard error possibly unstable and advised cautious interpretation of them.

Efron describes the infinitesimal jackknife as the delta method, so the ordinary jackknife and the delta method usually agree for smooth statistics in large samples. The jackknife needs no derivatives, only the ability to rerun the estimation.

Leave-one-out cross-validation uses the same deletion pattern for a different purpose. It predicts each left-out observation with a model fitted to the remaining data and averages the prediction errors, so its target is predictive accuracy, as in cross-validation more generally. The jackknife instead asks how much the estimate itself moves, and its targets are bias and sampling variance.

Limitations and common misreadings

The jackknife standard error of a mean equals the classical standard error, as the worked example shows, so it gives no protection against the skewness of cost data. Efron and Stein proved that the jackknife variance estimate tends to be biased upwards, so its intervals lean towards being conservative.

The method is unreliable for non-smooth statistics. Efron shows that the ordinary jackknife variance for the sample median is not even asymptotically consistent, and the same weakness affects other quantiles of cost distributions unless a grouped version is used.

Bias correction does not make an estimator unbiased in small samples. It removes only the leading term, and for a ratio such as an ICER whose denominator can approach zero, both the leave-one-out values and the correction can be unstable.

The jackknife describes sampling uncertainty in the data at hand. It says nothing about structural choices in a decision model, and its standard errors feed parameter uncertainty only after a distribution has been chosen for probabilistic sensitivity analysis.

Sources

• Barber JA, Thompson SG. Analysis of cost data in randomized trials: an application of the non-parametric bootstrap. Statistics in Medicine. 2000;19(23):3219-3236. https://pubmed.ncbi.nlm.nih.gov/11113956/
• Briggs AH, Wonderling DE, Mooney CZ. Pulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimation. Health Economics. 1997;6(4):327-340. https://pubmed.ncbi.nlm.nih.gov/9285227/
• Cook JR, Heyse JF. Use of an angular transformation for ratio estimation in cost-effectiveness analysis. Statistics in Medicine. 2000;19(21):2989-3003. https://pubmed.ncbi.nlm.nih.gov/11042628/
• Efron B. Bootstrap methods: another look at the jackknife. Annals of Statistics. 1979;7(1):1-26. https://doi.org/10.1214/aos/1176344552
• Efron B, Stein C. The jackknife estimate of variance. Annals of Statistics. 1981;9(3):586-596. https://doi.org/10.1214/aos/1176345462
• Onyimadu O, Astbury NM, Achana F, Petrou S, Violato M. Childhood transitions between weight status categories: evidence from the UK Millennium Cohort Study. PharmacoEconomics. 2024;42(6):649-661. https://doi.org/10.1007/s40273-024-01361-3
• StataCorp. bootstrap: Bootstrap sampling and estimation. Stata Base Reference Manual, release 19. College Station, TX: Stata Press; 2025. https://www.stata.com/manuals/rbootstrap.pdf
• StataCorp. jackknife: Jackknife estimation. Stata Base Reference Manual, release 19. College Station, TX: Stata Press; 2025. https://www.stata.com/manuals/rjackknife.pdf