Degrees of Freedom

Degrees of Freedom in Health Economic Statistics

Every estimate taken from data uses up some of its information, and the degrees of freedom are the running count of what is left for judging uncertainty. That count picks the exact t, chi-squared or F curve against which a statistic is read, so it decides how wide a confidence interval is and how surprising a test result looks. In health economics it matters most where samples are small or models are rich, as with a mean cost from a pilot trial, a meta-analysis of five trials or a choice between survival curves. Getting the count wrong usually makes uncertainty look smaller than it is, and the understatement can carry into the parameter distributions used in probabilistic sensitivity analysis.

How estimation uses up information

The simplest case is the sample variance. Its deviations are measured from the sample mean, which was computed from the same observations, and deviations from a mean always sum to zero. Once $n-1$ deviations are known the last is fixed, so the sum of squares is divided by $n-1$ instead of $n$.

$$s^2 = \frac{\sum_{i=1}^{n} (x_i - \bar{x})^2}{n - 1}$$

where $s^2$ is the sample variance, $x_i$ is the value for patient $i$, $\bar{x}$ is the sample mean and $n$ is the patient count. The NIST/SEMATECH e-Handbook uses this form for the variance and standard deviation.

The same accounting applies to fitted models. In least-squares regression analysis the NIST handbook divides the residual sum of squares by $n-p$, where $p$ is the number of parameters in the functional part of the model. Each estimated coefficient removes one residual degree of freedom, which links it to overfitting, since a model with almost as many parameters as observations leaves little information to judge its error.

Degrees of freedom as the shape of reference distributions

The main reference distributions are each indexed by degrees of freedom. The NIST handbook describes the chi-squared distribution as the sum of $\nu$ squared independent standard normal variables, with mean $\nu$, and the F-distribution as the ratio of two chi-squared variables, each divided by its degrees of freedom $\nu_1$ or $\nu_2$. The t-distribution has heavier tails than the normal distribution and approaches it as $\nu$ grows, with a good approximation above 30.

The heavier tails matter because the critical value for a two-sided 95% interval grows as the degrees of freedom fall. The t percentiles below were computed here.

Small samples and mean costs in trial-based evaluations

A t-based confidence interval is the standard parametric interval for a trial mean cost or QALY. The NIST handbook builds the interval for a mean from the t percentile with $N-1$ degrees of freedom multiplied by the standard error, $s/\sqrt{N}$. Chapter 6 of the Cochrane Handbook expects t-based intervals when groups are small, giving fewer than 60 participants per group as a guide, with $N-1$ degrees of freedom for one group and $N_E + N_C - 2$ for a two-sample t-test.

Thompson and Barber argue that only a t test on untransformed costs compares arithmetic means, the quantity a budget needs, and they prefer methods that do not assume equal standard deviations in the two arms. The NIST handbook gives the Welch-Satterthwaite degrees of freedom for that case, usually a non-integer value.

$$\nu = \frac{\left(s_1^2/N_1 + s_2^2/N_2\right)^2}{\frac{(s_1^2/N_1)^2}{N_1 - 1} + \frac{(s_2^2/N_2)^2}{N_2 - 1}}$$

where $s_1^2$ and $s_2^2$ are the cost variances in the two arms and $N_1$ and $N_2$ are the arm sizes. When one arm's variance term dominates, $\nu$ falls towards that arm's size minus one and the interval widens. Thompson and Barber note that t-based methods strictly assume normal data, so small or very skewed samples should be checked by bootstrapping.

Cochran's Q, I-squared and tau-squared in meta-analysis

Heterogeneity statistics in a meta-analysis are read against the study count minus one. Higgins and colleagues describe Cochran's Q as the weighted sum of squared deviations of each study from the pooled estimate, compared with a chi-squared distribution on $k-1$ degrees of freedom, and the Cochrane Handbook reads a large Q relative to its degrees of freedom as evidence of heterogeneity. Since a chi-squared variable has mean equal to its degrees of freedom, $k-1$ is the value Q is expected to take when studies differ by chance alone.

$$I^2 = \max\left(0, \frac{Q - (k-1)}{Q}\right) \times 100\%, \qquad \hat{\tau}^2 = \max\left(0, \frac{Q - (k-1)}{\sum w_i - \sum w_i^2 / \sum w_i}\right)$$

where $Q$ is Cochran's statistic, $k$ is the number of studies and $w_i$ is the inverse-variance weight of study $i$. I-squared is the share of Q in excess of its degrees of freedom, and the DerSimonian and Laird tau-squared used in a random-effects model scales the same excess, set to zero whenever Q is no larger than its degrees of freedom. Higgins and colleagues warn that the test has low power with few studies, so a non-significant Q cannot be taken as evidence of homogeneity.

Knapp-Hartung adjustment and prediction intervals

The usual Wald interval for a pooled random-effects estimate multiplies its standard error by 1.96, as though tau-squared were known. Chapter 10 of the Cochrane Handbook notes this can be overly narrow with few studies. The Hartung-Knapp-Sidik-Jonkman (HKSJ) adjustment generally inflates the variance of the summary mean and uses a t-distribution with $k-1$ degrees of freedom.

$$SE_{HKSJ} = \sqrt{\frac{\sum w_i^{*} (y_i - \hat{\mu})^2}{(k-1)\sum w_i^{*}}}$$

where $y_i$ is the estimate from study $i$, $\hat{\mu}$ is the random-effects pooled estimate and $w_i^{*}$ are the random-effects weights. IntHout, Ioannidis and Borm found lower error rates than with the DerSimonian and Laird method in simulations with heterogeneity, but advise caution with five or fewer studies of very unequal size.

Degrees of freedom explain one of the method's failure modes. With two studies the multiplier is the t percentile on 1 degree of freedom, 12.71, and the Cochrane Handbook reports excessively wide intervals with two or even three studies, while intervals can be overly narrow when tau-squared is estimated as zero. The Handbook's prediction interval for the effect in a new setting uses the same $k-1$ degrees of freedom.

Counting parameters when choosing survival models

In survival extrapolation the idea appears as a count of estimated parameters. Burnham and Anderson give the Akaike information criterion as minus twice the maximised log-likelihood plus twice the number of parameters $K$, which must include every estimated parameter. NICE DSU Technical Support Document 14 explains that an exponential model has one parameter, so two-parameter models such as the Weibull and Gompertz are penalised relative to it, and that the Bayesian information criterion penalises extra parameters more heavily.

TSD 14 also warns that comparing log-likelihoods is valid only for nested models. For nested models the likelihood ratio test refers twice the gain in log-likelihood to a chi-squared distribution with degrees of freedom equal to the difference in the number of free parameters. The exponential is a Weibull with shape fixed at 1, and the generalised gamma includes the Weibull, exponential and log-normal as special cases.

Flexible parametric models use the term directly. The flexsurv documentation of the Royston-Parmar model gives degrees of freedom equal to the number of knots plus one, so a model with no knots on the hazard scale is a Weibull with one degree of freedom, and the simulation study in NICE DSU TSD 21 compared models with 1 to 5. Extra degrees of freedom let the hazard bend more inside the trial, but TSD 14 notes that fit to observed data says nothing about behaviour beyond follow-up.

Worked example with a pilot trial, a small meta-analysis and survival curves

All numbers in this example are illustrative and were recomputed for this article. They show evidence read against different degrees of freedom.

1. Interval for a mean cost. A pilot of 12 patients has mean cost GBP 2,400 and standard deviation GBP 900, so the standard error is GBP 259.8 on 11 degrees of freedom. The t multiplier of 2.201 gives a half-width of $SE \times 2.201 = 571.8$ and an interval of GBP 1,828 to GBP 2,972. The normal multiplier would give a half-width of 509.2 and an interval about 11% too narrow.

2. Heterogeneity across five trials. Five trials report log hazard ratios of -0.40, -0.05, -0.35, 0.10 and -0.30 with standard errors of 0.15, 0.12, 0.20, 0.14 and 0.10. The inverse-variance weights sum to 289.91, and Q is 9.46 on 4 degrees of freedom, a P value of 0.051.

$$I^2 = \frac{9.462 - 4}{9.462} \times 100 = 57.7, \qquad \hat{\tau}^2 = \frac{9.462 - 4}{220.83} = 0.0247$$

where 220.83 is the denominator term for these weights and I-squared is in per cent. Q exceeds its degrees of freedom, so tau-squared is positive although Q is not significant at 5%.

3. Interval for the pooled hazard ratio. The random-effects log hazard ratio is -0.190 with a DerSimonian and Laird standard error of 0.0939, giving a hazard ratio of 0.83 with Wald interval 0.69 to 0.99. The HKSJ standard error is 0.0947 and the multiplier on 4 degrees of freedom is 2.776, giving 0.64 to 1.08. The standard error barely moves, so the reference distribution alone makes the interval include no effect. The 95% prediction interval is 0.50 to 1.38.

4. Choosing a survival curve. Five models are fitted to one trial arm.

The Weibull has the lowest AIC. Against the exponential the likelihood ratio statistic is 15.0 on 1 degree of freedom, above the 5% critical value of 3.841, so the shape parameter earns its place, while against the generalised gamma it is 1.6 on 1 degree of freedom, below 3.841, so the third parameter does not. The NICE manual (PMG36, sections 4.6.20 and 4.6.25) asks for the limitations of such fit statistics to be documented and the clinical plausibility of each hazard to be assessed.

Degrees of freedom compared with sample size

Degrees of freedom follow sample size but differ from it. A trial of 12 patients has 11 for one mean, while a meta-analysis of 5,000 patients in five trials has only 4 for heterogeneity, because the unit of replication is the study.

Common misreadings

Most errors claim more degrees of freedom than the data hold. Each makes uncertainty look smaller.

1. Using 1.96 for small samples. Mean costs or utilities from a few patients need the t multiplier.
2. Reading a non-significant Q as homogeneity. With few studies Q has low power.
3. Applying HKSJ mechanically. With two or three studies its intervals can be too wide, and with tau-squared estimated as zero too narrow.
4. Comparing non-nested survival models by log-likelihood. Only nested models have a valid likelihood ratio test.
5. Undercounting parameters. An AIC that leaves out a variance or shape parameter penalises the richer model too lightly.

Sources

• NIST/SEMATECH. e-Handbook of Statistical Methods. National Institute of Standards and Technology; 2012. Chapters 1.3.5, 1.3.6.6 and 4.4.3. https://www.itl.nist.gov/div898/handbook/
• Higgins JPT, Li T, Deeks JJ (editors). Chapter 6: Choosing effect measures and computing estimates of effect. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 6.5.2.2 and 6.5.2.3. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-06
• Deeks JJ, Higgins JPT, Altman DG, McKenzie JE, Veroniki AA (editors). Chapter 10: Analysing data and undertaking meta-analyses. In: Cochrane Handbook for Systematic Reviews of Interventions version 6.5. Cochrane; 2024. Sections 10.10.2 and 10.10.4.3 to 10.10.4.5. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-10
• Deeks JJ, Higgins JPT. Statistical algorithms in Review Manager. Cochrane; 2022. https://documentation.cochrane.org/revman-kb/files/210600101/210600103/1/1654774257333/Statistical_methods_in_revman.pdf
• Higgins JPT, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327:557-560. https://doi.org/10.1136/bmj.327.7414.557
• IntHout J, Ioannidis JPA, Borm GF. The Hartung-Knapp-Sidik-Jonkman method for random effects meta-analysis is straightforward and considerably outperforms the standard DerSimonian-Laird method. BMC Medical Research Methodology. 2014;14:25. https://doi.org/10.1186/1471-2288-14-25
• Thompson SG, Barber JA. How should cost data in pragmatic randomised trials be analysed? BMJ. 2000;320:1197-1200. https://doi.org/10.1136/bmj.320.7243.1197
• Burnham KP, Anderson DR. Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods and Research. 2004;33(2):261-304. https://doi.org/10.1177/0049124104268644
• Chen Y, Moustaki I, Zhang H. A note on likelihood ratio tests for models with latent variables. Psychometrika. 2020;85(4):996-1012. https://doi.org/10.1007/s11336-020-09735-0
• Latimer N. NICE DSU Technical Support Document 14: survival analysis for economic evaluations alongside clinical trials. University of Sheffield; 2011, updated 2013. Sections 2.2, 2.6, 3.3 and 3.5. https://www.sheffield.ac.uk/nice-dsu/tsds/survival-analysis
• Rutherford MJ, Lambert PC, Sweeting MJ, et al. NICE DSU Technical Support Document 21: flexible methods for survival analysis. University of Sheffield; 2020. https://doi.org/10.15131/shef.data.33849118
• Jackson C. flexsurvspline help page. flexsurv R package, CRAN; 2024. https://search.r-project.org/CRAN/refmans/flexsurv/html/flexsurvspline.html
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). 2022, last updated 13 October 2026. Sections 4.6.20 and 4.6.25. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2