Tolerance Interval

Tolerance Interval for Variation Between Patients

A tolerance interval answers a question about individual patients rather than about an average. In health economics the question might be the range of annual treatment costs within which nearly all patients of a service fall, or a length of stay that almost no patient will exceed, with an allowance for the fact that the limits are estimated from a finite sample. Two numbers define the interval, the coverage (the share of individual values the limits must enclose) and the confidence that the coverage is achieved, so a 95%/95% interval aims to enclose at least 95% of patients with 95% confidence. Because it describes spread between patients, a tolerance interval is much wider than a confidence interval around a mean cost, and it does not shrink towards zero width as the sample grows. It settles instead on the true population percentiles.

Coverage and confidence as two separate probabilities

The NIST/SEMATECH e-Handbook of Statistical Methods draws the contrast directly. Confidence limits are limits within which a population parameter such as the mean is expected to lie, while statistical tolerance limits are limits within which a stated proportion of the population is expected to lie. The handbook adds that statistical tolerance intervals have a probabilistic interpretation and are unrelated to engineering tolerances, which are limits of acceptability set by a designer.

$$\Pr\left[\, F(U) - F(L) \ge P \,\right] \ge \gamma$$

where $L$ and $U$ are the lower and upper tolerance limits calculated from the sample, $F$ is the cumulative distribution function of the individual values in the population, $F(U)-F(L)$ is the coverage (the share of the population between the limits), $P$ is the required coverage and $\gamma$ is the confidence level. Young, describing the R package tolerance, gives the same definition and the one-sided versions, in which an upper limit $U$ satisfies $\Pr[F(U) \ge P] \ge \gamma$. That one-sided condition is the same statement as saying that $U$ is an upper confidence limit, at level $\gamma$, for the population value below which a share $P$ of patients fall.

The coverage of any one calculated interval is unknown, since $F$ is unknown. The confidence describes the procedure, which in repeated samples produces limits enclosing at least a share $P$ of the population in a proportion $\gamma$ of samples. Young notes that $\gamma$ and $P$ are usually chosen from 0.90, 0.95 and 0.99.

Patient variability and parameter uncertainty in economic evaluation

The ISPOR-SMDM task force report on parameter estimation and uncertainty separates concepts that are often confused. Stochastic uncertainty, also called first-order uncertainty or variability, is random variability in outcomes between identical patients. Parameter uncertainty, or second-order uncertainty, is the uncertainty in estimating the parameter of interest, with the standard error as its regression analogue. Heterogeneity is the part of between-patient variability that can be attributed to patient characteristics.

Confidence intervals for mean costs and QALYs, and the distributions sampled in probabilistic sensitivity analysis, describe parameter uncertainty. A tolerance interval describes the spread of individual values, which is stochastic variability plus any heterogeneity that has not been conditioned on. The task force notes that patient-level models such as microsimulation and discrete-event simulation simulate numerous virtual patients, and that assessing parameter uncertainty in them requires the stochastic uncertainty to be eliminated. A tolerance interval calculated from the simulated patients of one run with fixed parameters therefore describes first-order variation conditional on those parameter values.

Three intervals are easily run together. A confidence interval targets a parameter such as the mean cost. A prediction interval targets one future observation, for example the cost of the next patient treated. A tolerance interval targets a stated share of the whole population of patients. When heterogeneity matters, limits can be calculated within subgroups, and Young's package also provides regression tolerance intervals that condition on covariates.

Normal-theory tolerance factors

For data from a normal distribution, the limits take the familiar form of the sample mean plus or minus a multiple of the sample standard deviation. The multiple, the tolerance factor, is larger than the usual normal critical value because it must allow for sampling error in both the mean and the standard deviation.

$$L = \bar{x} - k s, \qquad U = \bar{x} + k s$$

where $\bar{x}$ is the sample mean, $s$ is the sample standard deviation and $k$ is the tolerance factor, which depends on the sample size $n$, the coverage $P$ and the confidence $\gamma$. Young notes that the one-sided factor has an exact solution based on the noncentral t distribution, while the two-sided factor is usually approximated. The NIST handbook gives the approximation due to Howe.

$$k_2 = z_{(1+P)/2} \sqrt{\frac{(n-1)\left(1 + \frac{1}{n}\right)}{\chi^{2}_{1-\gamma,\, n-1}}}$$

where $k_2$ is the two-sided factor, $z_{(1+P)/2}$ is the standard normal value with cumulative probability $(1+P)/2$, which is 1.96 when $P$ is 0.95, and $\chi^{2}_{1-\gamma,\, n-1}$ is the value of the chi-square distribution with $n-1$ degrees of freedom that is exceeded with probability $\gamma$.

The factor falls towards the normal critical value as the sample grows, because the mean and standard deviation become known. The table gives 95%/95% factors calculated for this article by numerical integration and checked by simulation, with Howe's approximation alongside.

Distribution-free limits from the smallest and largest values

When no distribution can be assumed, the limits are taken from the ordered data. Young states that a nonparametric tolerance interval needs only a continuous distribution and uses order statistics as the limits, the $r$th smallest and $q$th smallest observations, with $r$ less than $q$. The NIST handbook warns that normal-theory intervals will not give the intended coverage when the data are not normal, and that distribution-free methods may give substantially wider intervals.

For the interval between the smallest and largest of $n$ observations, the handbook gives the confidence that coverage $P$ is reached.

$$\gamma = 1 - n P^{\,n-1} + (n - 1) P^{\,n}$$

where $n$ is the sample size, $P$ is the coverage and $\gamma$ is the resulting confidence. With 25 observations, the handbook's table shows only 0.358 confidence that the sample range covers 95% of the population. The smallest sample for a 95%/95% interval from the extremes is 93, which gives a confidence of 0.9500, while 92 gives 0.9479. The handbook's approximate formula also gives 93. For a one-sided upper limit set at the largest value, the confidence is one minus $P^{n}$, so 59 observations are needed for 95%/95%. These sample sizes matter for non-parametric estimation from small trial cost samples, which often cannot support a distribution-free 95%/95% statement at all.

Skewed costs and the log scale

Thompson and Barber describe highly skewed distributions as typical of cost data, with a long right-hand tail because some patients incur high costs from complications, reoperation or extended hospital stay. For a mean, the log scale changes the question, since a t test on log costs compares geometric means.

A tolerance interval behaves differently. The share of patients with costs between two limits equals the share whose log costs lie between the logs of those limits, so coverage survives any increasing transformation. Young describes the procedure for log-normal data, calculating a normal tolerance interval for the logged values and exponentiating both limits. A log-normal distribution often fits positive, right-skewed costs better than a normal one, and the resulting limits are asymmetric around the median, as the skewness of the data requires.

Worked example with annual costs for 150 patients

All numbers in this example are illustrative. A service evaluation records annual costs for 150 patients. The log costs have mean 8.50 and standard deviation 0.60, and on the original scale the arithmetic mean is GBP 5,900 and the standard deviation GBP 3,850. The analysis wants a 95%/95% two-sided tolerance interval for annual cost per patient.

1. Tolerance factor. With 150 patients, the numerator of Howe's ratio is 149 × 1.00667 = 149.99. The chi-square value with 149 degrees of freedom exceeded with probability 0.95 is 121.79, so the ratio is 149.99 / 121.79 = 1.2316, its square root is 1.1098 and Howe's factor is 1.96 × 1.1098 = 2.175. The exact factor is 2.176, used as 2.18.

2. Limits on the log scale. The half-width is 2.18 × 0.60 = 1.308. Subtracting 1.308 from 8.50 gives a lower limit of 7.192, and adding it gives an upper limit of 9.808.

3. Limits on the cost scale. Exponentiating gives $e^{7.192}$ = GBP 1,329 and $e^{9.808}$ = GBP 18,179. The median cost is $e^{8.50}$ = GBP 4,915, so the interval stretches much further above the median than below it.

4. A normal interval on raw costs. Applying the same factor to the raw costs gives 2.18 × 3,850 = 8,393. Around the mean of GBP 5,900 the limits would then be minus GBP 2,493 and GBP 14,293. A negative lower limit is impossible for costs, and the upper limit understates the right tail.

5. The confidence interval for the mean. The standard error of the mean is 3,850 / $\sqrt{150}$ = 314.35, and with a t value of 1.976 for 149 degrees of freedom the half-width is 1.976 × 314.35 = 621. The 95% confidence interval for mean cost is GBP 5,279 to GBP 6,521.

6. A one-sided upper limit. The exact one-sided 95%/95% factor for 150 patients is 1.870, so the half-width is 1.870 × 0.60 = 1.122. The upper limit is 8.50 plus 1.122, or 9.622 on the log scale, which is $e^{9.622}$ = GBP 15,093.

7. Distribution-free check. With 150 patients, the interval between the cheapest and the most expensive patient covers at least 95% of the population with confidence 0.996 by the formula above. Using the second cheapest and second most expensive patients instead gives a confidence of 0.945, just short of 95%.

Interpretation. Under the log-normal assumption, there is 95% confidence that at least 95% of patients like these have annual costs between about GBP 1,330 and GBP 18,180, and that at least 95% cost less than about GBP 15,090. The mean is pinned down far more tightly, at GBP 5,279 to GBP 6,521. A budget for the whole service rests on the mean, while planning for individual cases, such as a reserve for expensive patients, rests on the tolerance limits.

Uses in health economic work

Trial-based evaluations and decision models are built around means, so a tolerance interval does not replace a confidence interval or a probabilistic analysis. It adds a different piece of information, how widely individual patients vary around the mean. That can help in describing patient-level costs, lengths of stay or utilities alongside the mean, and in judging whether a single average tariff or cost per case hides wide variation in cost between individual cases.

In model validation, a tolerance interval calculated from observed patient data gives a range within which most simulated patients from a well-specified patient-level model should fall, conditional on the parameter values used. A simulation whose patients spread much more or much less widely than the observed limits suggests the model misrepresents first-order variation, even when its mean is right. A one-sided upper limit can also support a rule for flagging unusually costly or long cases, because it states, with confidence, a level that only a small share of patients should exceed.

Misreadings and limits

Most errors with tolerance intervals come from treating them as another form of confidence interval. Others come from forgetting the assumptions behind the tolerance factor.

1. Mean plus or minus 1.96 standard deviations. That plug-in range estimates where 95% of values lie but carries no confidence allowance. With 150 patients the 95%/95% factor is 2.18, and with 20 patients it is 2.76.
2. Reading the confidence as coverage. A 95%/95% interval is a statement about the procedure and the share of the population, not a 95% probability that one named patient falls inside.
3. Normal factors on skewed costs. Raw-scale normal limits can be negative and can miss the right tail, as step 4 shows.
4. Expecting a narrow interval from a large sample. More data narrow the confidence interval for the mean, but the tolerance interval settles on the population spread.
5. Ignoring dependence. The methods assume independent, identically distributed observations, which repeated measures on the same patients or clustering by centre can break.
6. Pooling unlike patients. A single interval for a mixed population may describe no subgroup well, so heterogeneity calls for subgroup or regression tolerance intervals.

Sources

• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722-732. https://doi.org/10.1177/0272989X12458348
• National Institute of Standards and Technology. NIST/SEMATECH e-Handbook of Statistical Methods, section 7.2.6, What intervals contain a fixed percentage of the population values (sections 7.2.6.3 and 7.2.6.4). Accessed 28 October 2026. https://www.itl.nist.gov/div898/handbook/prc/section2/prc26.htm
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200. https://doi.org/10.1136/bmj.320.7243.1197
• Young DS. tolerance: an R package for estimating tolerance intervals. Journal of Statistical Software. 2010;36(5):1-39. https://doi.org/10.18637/jss.v036.i05