Standard Error

Standard Error in Health Economic Estimates and Models

The standard error tells an analyst how far an estimate, such as the mean cost per patient in a trial arm or a log hazard ratio taken from a published study, is likely to sit from the population value because of which patients happened to be sampled. It describes the estimate and not the patients, and it shrinks as the sample grows while the spread of individual costs does not. In health economics it sits behind confidence intervals and tests, sets study weights in meta-analysis and fixes the width of input distributions in probabilistic sensitivity analysis. Confusing it with the standard deviation misstates the uncertainty in a model input by a factor equal to the square root of the sample size. Derived ratios, small skewed samples and clustered data are covered under the delta method, the bootstrap standard error and the robust standard error.

Spread of the estimate and spread of the patients

Altman and Bland, in a BMJ statistics note, explain that a sample mean varies between samples in a way described by its sampling distribution, and that the standard deviation of that distribution is the standard error of the mean. They add that it can be read as a measure of the precision of the sample mean, and that the contrast with the standard deviation reflects the distinction between data description and inference. The ISPOR-SMDM task force report on parameter estimation draws the same line for decision models. It contrasts the standard deviation, describing how individual observations vary, with the standard error, the precision of an estimated quantity, and files the standard error under parameter uncertainty, not under stochastic variation between identical patients.

For a mean the relation is short.

$$SE(\bar{x}) = \frac{s}{\sqrt{n}}$$

where $SE(\bar{x})$ is the estimated standard error of the sample mean $\bar{x}$, $s$ is the sample standard deviation of the individual observations and $n$ is the number of observations.

Altman and Bland note that the standard error falls as the sample size increases while the standard deviation does not tend to change, and that this idea underlies sample size calculation for a controlled trial. The central limit theorem explains why the mean of right-skewed costs is close to normal in large samples, so that its standard error can be used with normal-theory methods.

Standard errors for the estimates evaluations report

Altman and Bland note that a standard error can be calculated for any quantity estimated from a sample, such as a proportion, a regression coefficient, a risk ratio or a difference between two means.

For a difference in mean cost between two independent trial arms, the variances add.

$$SE(\bar{x}_1 - \bar{x}_2) = \sqrt{\frac{s_1^{2}}{n_1} + \frac{s_2^{2}}{n_2}}$$

where $\bar{x}_1$ and $\bar{x}_2$ are the arm means, $s_1$ and $s_2$ are the arm standard deviations and $n_1$ and $n_2$ are the numbers of patients in each arm. This form does not assume equal standard deviations, the approach Thompson and Barber prefer for cost data in general. The classical t test instead puts a single pooled standard deviation in place of both $s_1$ and $s_2$.

For a proportion, such as the share of patients readmitted within a year, the standard error is as follows.

$$SE(\hat{p}) = \sqrt{\frac{\hat{p}(1 - \hat{p})}{n}}$$

where $\hat{p}$ is the observed proportion and $n$ is the number of patients. The Cochrane Handbook (chapter 6) states that risk ratios, odds ratios and hazard ratios are analysed on the natural log scale, and that the standard error then refers to the log of the ratio. For a cost or utility regression, the fitted model reports a standard error for each coefficient and a variance-covariance matrix for the set, and the ISPOR-SMDM report notes that multivariable estimation also gives the covariance between estimated parameters.

Incremental cost and incremental QALYs measured on the same patients are usually correlated. The standard error of a combination such as incremental net benefit therefore needs their covariance as well as their separate standard errors, a calculation worked through on the confidence interval page.

Where the standard error is used

Altman and Bland describe the standard error as most useful for calculating a confidence interval, a large-sample 95% interval being 1.96 standard errors either side of the mean, and note that it is also used to calculate P values. The ratio of an estimate to its standard error is the Wald test statistic, compared with the standard normal distribution, and for a mean from a small sample the same ratio is referred to the t distribution in a t test.

In evidence synthesis the standard error sets each study's weight. The Cochrane Handbook explains that software for generic inverse-variance meta-analysis mostly takes an effect estimate and its standard error from each study. With weights equal to the inverse of the squared standard error, halving a study's standard error quadruples its weight.

In decision models, the ISPOR-SMDM report observes that standard estimation methods give a point estimate with a measure of precision, such as standard errors or 95% confidence intervals. Where much information is available, it says the normal distribution can be used in PSA and a standard confidence interval in a deterministic sensitivity analysis, with consistency kept because the arithmetic mean and standard error inform both. It names beta distributions as a natural match for binomial data and gamma or log normal distributions for right-skewed parameters. Such distributions are often fitted by the method of moments, which sets the distribution's mean equal to the estimate and its standard deviation equal to the estimate's standard error.

A gamma distribution with shape $k$ and scale $\theta$ has mean $k\theta$ and variance $k\theta^{2}$, so matching gives the following.

$$k = \frac{\mu^{2}}{SE^{2}}, \qquad \theta = \frac{SE^{2}}{\mu}$$

where $\mu$ is the estimated mean, for example a mean cost, and $SE$ is its standard error. For a beta distribution with parameters $\alpha$ and $\beta$ fitted to a probability, matching the mean and variance gives the following.

$$\alpha + \beta = \frac{\mu(1 - \mu)}{SE^{2}} - 1, \qquad \alpha = \mu(\alpha + \beta), \qquad \beta = (1 - \mu)(\alpha + \beta)$$

where $\mu$ is the estimated probability and $SE$ is its standard error. Fitted with the patient standard deviation instead, they describe individual patients, not uncertainty about the mean.

Recovering a standard error from a published report

Model inputs often come from papers that report an estimate with a confidence interval or a P value but no standard error. The Cochrane Handbook (chapter 6, sections 6.3.1 and 6.3.2) gives the conversions for this case.

$$SE = \frac{U - L}{3.92}, \qquad SE = \frac{\hat{\theta}}{Z}$$

where $U$ and $L$ are the upper and lower limits of a 95% confidence interval, $\hat{\theta}$ is the reported estimate and $Z$ is the standard normal value that matches its two-sided P value. For a ratio measure both limits are first converted to natural logs, and the result is the standard error of the log ratio. The P value route assumes a test of the estimate divided by its standard error. The handbook's example is a risk difference of 0.03 with P = 0.008, for which Z = 2.652 and the standard error is 0.03 / 2.652 = 0.011.

The same chapter adds that a group standard deviation can be obtained by multiplying the standard error of a group mean by the square root of the group size, but the standard error must come from within a single group, never from the difference between groups.

Worked example with an illustrative cost and readmission input

All numbers in this example are illustrative. A cohort of 400 patients supplies two inputs for a cost-effectiveness model, the mean annual cost of managing the condition and the probability of readmission within a year.

Step 1. Standard error of the mean cost. The mean cost is GBP 8,200 and the standard deviation across patients is GBP 2,400. The square root of 400 is 20, so the standard error is 2,400 / 20 = GBP 120.

Step 2. Confidence interval. Since 1.96 × 120 = 235.2, the 95% interval for the mean is GBP 7,964.80 to GBP 8,435.20.

Step 3. Gamma distribution for the PSA. The shape is k = 8,200² / 120² = 67,240,000 / 14,400 = 4,669.4 and the scale is θ = 14,400 / 8,200 = 1.7561. As a check, k × θ = 4,669.4 × 1.7561 = 8,200 and √k × θ = 68.33 × 1.7561 = 120. The 2.5th and 97.5th percentiles of this gamma distribution are GBP 7,966 and GBP 8,437, close to the normal interval.

Step 4. The same fit with the standard deviation. Using GBP 2,400 by mistake gives k = 67,240,000 / 5,760,000 = 11.67 and θ = 5,760,000 / 8,200 = 702.4. The central 95% range becomes GBP 4,192 to GBP 13,530, and 33.7% of PSA draws exceed GBP 9,000, against effectively none with the correct distribution. The standard deviation of the draws is 20 times too large, the square root of the sample size, so the model would treat a well-estimated input as highly uncertain.

Step 5. Readmission probability. If 48 of the 400 patients are readmitted, the proportion is 0.12 and its standard error is √(0.12 × 0.88 / 400) = √0.000264 = 0.0162. For a beta distribution, 0.1056 / 0.000264 = 400, and subtracting 1 gives α + β = 399, so α = 0.12 × 399 = 47.88 and β = 0.88 × 399 = 351.12. Its central 95% range is 0.090 to 0.154.

Interpretation. The mean is precise even though patients vary widely, and that precision is what the model input must carry. A cohort of 1,600 would halve the standard error to GBP 60, while the patient standard deviation would stay near GBP 2,400.

Checking a published comparison of trial costs

Thompson and Barber reproduce a table from a trial of hospital at home against inpatient care for hip replacement patients. Mean hospital costs over three months were GBP 515.42 (standard deviation 473.20) for 36 hospital-at-home patients and GBP 776.30 (standard deviation 364.53) for 49 inpatients, with a reported difference of minus GBP 260.87 and a 95% interval of minus GBP 441.56 to minus GBP 80.19.

Pooled standard deviation. The pooled variance is (35 × 473.20² + 48 × 364.53²) / 83 = (7,837,138.4 + 6,378,341.8) / 83 = 171,270.8, so the pooled standard deviation is 413.85.

Pooled standard error. The standard error of the difference is 413.85 × √(1/36 + 1/49) = 413.85 × 0.2195 = GBP 90.85. With 83 degrees of freedom the t value is 1.989, and 1.989 × 90.85 = 180.69. The rounded means differ by 260.88, so the interval is minus GBP 441.57 to minus GBP 80.19, which reproduces the published limits and shows they assumed equal standard deviations.

Unequal-variance standard error. Without that assumption the standard error is √(473.20² / 36 + 364.53² / 49) = √(6,219.95 + 2,711.88) = √8,931.83 = GBP 94.51, about 4% larger. With the Satterthwaite approximation of 63.4 degrees of freedom the t value is 1.998, giving minus GBP 449.72 to minus GBP 72.04.

Interpretation. Both intervals exclude zero, so the conclusion stands, but the smaller arm had the larger spread and the equal-variance interval is slightly too narrow, the situation in which Thompson and Barber prefer methods that do not assume equal standard deviations.

What a standard error leaves out

A standard error measures sampling variation within one study, under that study's model, and bias and structural uncertainty do not appear in it. The ISPOR-SMDM report warns that relying on a single study is likely to underestimate uncertainty.

The simple formulas also assume independent patients and a reasonably large sample. Patients in the same practice or ward resemble one another, so $s / \sqrt{n}$ understates the standard error in cluster randomised trials and a cluster-adjusted estimator is needed. Thompson and Barber advise checking small or extremely skewed cost samples with the bootstrap. A ratio such as the ICER has no simple standard error, so the delta method gives an approximate one and the Fieller method builds the interval directly.

Misreadings that recur

Each of these misreadings changes the uncertainty that reaches a decision. The PSA case, using the patient standard deviation in place of the standard error, is shown in step 4.

1. Reporting the standard error to describe patients. Altman and Bland advise the standard deviation for how widely measurements are scattered and the standard error for uncertainty around the mean. They also cite a review of 88 articles published in 2002 in which 12 (14%) did not say which measure of dispersion they reported.
2. Deriving a group standard deviation from the standard error of a difference. The Cochrane Handbook allows the conversion only for a standard error from within one group.
3. Adding standard errors instead of variances. Independent variances add, and correlated estimates also need their covariance.

Sources

• Altman DG, Bland JM. Standard deviations and standard errors. BMJ. 2005;331(7521):903. https://doi.org/10.1136/bmj.331.7521.903
• Briggs AH, Weinstein MC, Fenwick EAL, Karnon J, Sculpher MJ, Paltiel AD. Model parameter estimation and uncertainty analysis: a report of the ISPOR-SMDM Modeling Good Research Practices Task Force Working Group-6. Medical Decision Making. 2012;32(5):722-732. https://doi.org/10.1177/0272989X12458348
• Higgins JPT, Li T, Deeks JJ (editors). Chapter 6: Choosing effect measures and computing estimates of effect (last updated August 2023). In: Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; 2024. Sections 6.3.1, 6.3.2 and 6.5.2.2. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-06
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320(7243):1197-1200. https://doi.org/10.1136/bmj.320.7243.1197