Design Effect

Design Effect in Clustered Samples and Economic Evaluation

A design effect says how many times larger the sampling variance of an estimate is because people were sampled in clusters or with unequal weights, compared with the same number of people drawn independently. A value of 2 means a clustered sample of 600 carries about as much information as 300 independent observations. In health economics it decides how many practices or wards a cluster trial or survey must recruit, and how much uncertainty a survey-based or trial-based input should carry into a cost-effectiveness model. Ignoring it makes a prevalence, a cost or an incremental net benefit look more precise than the data allow.

Variance ratio, effective sample size and its square root

The design effect is attached to one statistic, not to a whole survey. The NHANES variance estimation tutorial from the US National Center for Health Statistics (NCHS) calls it an attribute of a statistic calculated on a particular variable, and notes that it can differ between variables and between subgroups of the same survey.

$$\text{DEFF} = \frac{V_{\text{design}}(\hat{\theta})}{V_{\text{SRS}}(\hat{\theta})}, \qquad n_{\text{eff}} = \frac{n}{\text{DEFF}}, \qquad \text{DEFT} = \sqrt{\text{DEFF}}$$

where $\hat{\theta}$ is the estimate (a proportion, mean cost or treatment effect), $V_{\text{design}}$ is its variance allowing for clustering, stratification and weighting, $V_{\text{SRS}}$ is its variance under a simple random sample of the same size $n$, $n_{\text{eff}}$ is the effective sample size and DEFT is the factor by which the standard error grows. The WHO vaccination coverage survey manual notes that some surveys, including the Demographic and Health Surveys, report DEFT instead of DEFF, in which case the effective sample size is the sample size divided by DEFT squared. Reading a reported DEFT as if it were a DEFF therefore understates the variance inflation.

Where the inflation comes from in clustered and weighted samples

Patients in the same practice, hospital or village tend to resemble each other, so each extra patient from a cluster adds less new information than a patient drawn from elsewhere. The similarity is summarised by the intraclass correlation $\rho$. The CONSORT extension for cluster randomised trials gives the inflation factor, or design effect, for clusters of equal size.

$$\text{DEFF} = 1 + (m - 1)\rho$$

where $m$ is the number of people per cluster and $\rho$ is the intracluster correlation coefficient, the share of total outcome variance lying between clusters. CONSORT notes that $\rho$ is typically small, often below 0.05, but its effect can be considerable when clusters are large. Because $m$ multiplies $\rho$, adding more people per cluster raises the design effect.

Many surveys also use unequal weights. The WHO manual (WHO/IVB/18.09) multiplies the clustering term by a weighting term, and it describes the assumption in its 2005 predecessor of a design effect of 2 for every survey as misleading, because the value changes with the number of respondents per cluster.

$$\text{DEFF} = \left[1 + (m - 1)\rho\right]\left[1 + CV_w^{2}\right]$$

where $CV_w$ is the coefficient of variation of the survey weights. Clusters of unequal size also inflate variance. Rutterford, Copas and Eldridge, reviewing sample size methods for cluster randomised trials, report that using the mean cluster size in the simple formula underestimates the required sample size, and that adjustment is recommended when the coefficient of variation of cluster size exceeds 0.23. One adjustment they give, for an appropriately weighted cluster-level analysis, needs only the mean and spread of cluster size, and they note that it overestimates the sample size needed for most individual-level analyses.

$$\text{DEFF} = 1 + \left[(CV^{2} + 1)\bar{m} - 1\right]\rho$$

where $CV$ is the standard deviation of cluster size divided by the mean cluster size $\bar{m}$. With equal clusters $CV = 0$ and the formula returns to the equal-cluster version above.

Planning a cluster survey or cluster randomised trial

At the design stage the sample size that a simple random sample would need is multiplied by the design effect. Rutterford and colleagues describe this as the simplest approach to sample size under cluster randomisation, and the WHO manual builds its survey sample size from an effective sample size multiplied by DEFF and then inflated for non-response. A design effect therefore sits inside the wider sample size calculation, alongside the effect size and statistical power.

The choice between more clusters and more people per cluster follows from the formula. CONSORT states that power is increased more easily by adding clusters than by enlarging them. The illustrative table below takes a precision target that a simple random sample of 801 would meet (a 95% confidence interval of plus or minus 3 percentage points around a proportion of 25%) and an intraclass correlation of 0.05.

Each row multiplies 801 by the design effect and rounds the number of clusters up. Doubling the cluster size from 20 to 40 saves 19 clusters but needs 820 more people, so the cheaper design depends on the relative cost of recruiting a site and recruiting a patient.

Assumptions that change the design effect

CONSORT notes that $\rho$ is often not known when a trial is planned and is only estimated with error after it is completed. The WHO manual suggests an observed figure from a recent survey in the same area or a conservative value slightly larger than expected. The value also depends on the outcome and on adjustment. Rutterford and colleagues identify inclusion of the baseline value of the outcome as a covariate as arguably the strongest factor in reducing the intraclass correlation, so a published value should be matched to the planned analysis.

A design effect can fall below 1. The NCHS reliability tutorial notes this possibility, which produces an effective sample size larger than the actual sample, and caps the effective sample size at the actual sample size when applying minimum sample size rules. The same tutorial explains that software defines the comparison "simple random sample of the same size" in different ways, so reported design effects from different packages are not always comparable.

Precision also depends on the number of clusters, not only on the design effect. The NHANES tutorial gives the design degrees of freedom as the number of primary sampling units minus the number of strata, which is 15 for most two-year NHANES releases. NCHS data presentation standards suppress an estimated proportion if either the actual or the effective sample size is below 30.

Design effects in cost-effectiveness analysis and model inputs

When a cost-effectiveness analysis runs alongside a cluster randomised trial, costs and outcomes are both clustered. Gomes and colleagues, comparing statistical methods for such analyses, used a median intraclass correlation of 0.05 for costs from their case studies and 0.02 for outcomes from their literature review, so costs and health outcomes can carry different design effects. In their case study of 40 clinics and 1,851 patients, with intraclass correlations of around 0.01, the standard error of the incremental net benefit was GBP 286 when clustering was ignored and between GBP 648 and GBP 733 under methods that allowed for it. The per-patient expected value of perfect information was GBP 114 when clustering was ignored and between GBP 262 and GBP 293 otherwise. The authors advise that methods ignoring clustering should not be justified on the basis of low estimated intraclass correlations, and conclude that multilevel models and a two-stage bootstrap that resamples clusters as well as individuals were appropriate in the settings studied.

The same logic applies to model inputs taken from cluster surveys. In probabilistic sensitivity analysis a proportion is often given a beta distribution. Matching the beta variance to the design-based variance gives a simple rule.

$$\frac{\hat{p}(1 - \hat{p})}{\alpha + \beta + 1} = \text{DEFF} \times \frac{\hat{p}(1 - \hat{p})}{n} \quad \Rightarrow \quad \alpha + \beta = n_{\text{eff}} - 1, \qquad \alpha = \hat{p}(\alpha + \beta)$$

where $\hat{p}$ is the survey estimate and $\alpha$ and $\beta$ are the beta parameters. Building the distribution from the raw sample size instead treats a clustered sample as if every respondent were independent and understates parameter uncertainty in the model.

Weighting raises a parallel issue in population-adjusted indirect comparisons. NICE DSU Technical Support Document 18 approximates the effective sample size of a weighted population as the squared sum of the weights divided by the sum of the squared weights. That quantity equals $n/(1 + CV_w^2)$ when $CV_w$ is computed with divisor $n$, so the divisor is the weighting term of the WHO formula. TSD 18 adds that with estimated weights this approximation is likely to underestimate the true effective sample size.

Worked example with a practice survey feeding a model input

All numbers are illustrative. A model of a chronic condition needs the proportion of diagnosed patients whose disease is uncontrolled, taken from a survey of 20 patients in each of 30 general practices, 600 patients in total. The survey finds 25% uncontrolled, and the intraclass correlation is 0.05.

1. Design effect. With 20 patients per practice, DEFF = 1 + (20 − 1) × 0.05 = 1.95, so DEFT is the square root of 1.95, or 1.396.

2. Variance and standard error. Under simple random sampling the variance is 0.25 × 0.75 / 600 = 0.0003125 and the standard error 0.0177. The design-based variance is 1.95 × 0.0003125 = 0.000609 and the standard error 0.0247.

3. Confidence interval. Ignoring clustering, the 95% interval is 0.25 ± 1.96 × 0.0177, or 21.5% to 28.5%. Allowing for it, the interval is 0.25 ± 1.96 × 0.0247, or 20.2% to 29.8%, about 40% wider.

4. Effective sample size. 600 / 1.95 = 307.7, so the survey carries the information of about 308 independent patients.

5. Beta distribution for the model. The sum of the parameters is 307.7 − 1 = 306.7, so $\alpha$ = 0.25 × 306.7 = 76.7 and $\beta$ = 0.75 × 306.7 = 230.0. Using the raw sample size would give parameters summing to 599, close to a Beta(150, 450), whose variance is about half the correct value.

6. Other sources of inflation. If the number of patients surveyed per practice had varied around the mean of 20 with a coefficient of variation of 0.5, the unequal-cluster formula first gives (0.25 + 1) × 20 = 25 and then 1 + (25 − 1) × 0.05 = 2.2, with an effective sample size of 600 / 2.2 = 273. If instead survey weights had a coefficient of variation of 0.4, the WHO formula gives 1.95 × (1 + 0.16) = 2.26 and an effective sample size of 600 / 2.26 = 265.

7. Interpretation. The point estimate of 25% does not change, but the model input is less certain than a naive analysis suggests. Feeding the wider distribution into the model widens the spread of simulated costs and QALYs and, as in the Gomes case study of trial data, can raise the estimated value of further research.

Boundaries with related concepts

The intraclass correlation describes how alike people within a cluster are, and the design effect converts that similarity, together with cluster size and weighting, into a variance multiplier. The cluster effect is the underlying tendency for outcomes to group. The variance inflation factor in regression is a different idea, measuring how much collinearity among predictors inflates the variance of a coefficient. A robust standard error or a hierarchical model estimates the uncertainty that clustering creates directly from the data, while the design effect is mainly a planning and reporting summary of it.

Where design effects are misread

A single design effect for a whole survey is a simplification, because each statistic, variable and subgroup has its own. A fixed value of 2 borrowed from older guidance ignores the number of people per cluster. Applying a planning multiplier to a finished analysis is no substitute for software that uses the actual strata, clusters and weights, which the NHANES tutorial says should be used for all statistical tests and confidence limits. The design effect deals only with sampling variance, so it does not correct bias from non-response, missing data or poor measurement. With few clusters the degrees of freedom stay small whatever the design effect, and CONSORT asks cluster trial reports to give the intracluster correlation for each primary outcome.

Sources

• Campbell MK, Piaggio G, Elbourne DR, Altman DG. Consort 2010 statement: extension to cluster randomised trials. BMJ. 2012;345:e5661. https://doi.org/10.1136/bmj.e5661
• Gomes M, Ng ES, Grieve R, Nixon R, Carpenter J, Thompson SG. Developing appropriate methods for cost-effectiveness analysis of cluster randomized trials. Medical Decision Making. 2012;32(2):350-361. https://doi.org/10.1177/0272989X11418372
• National Center for Health Statistics. NHANES tutorials, Variance Estimation module and Reliability of Estimates module. Centers for Disease Control and Prevention (accessed 11 October 2026). https://wwwn.cdc.gov/nchs/nhanes/tutorials/varianceestimation.aspx and https://wwwn.cdc.gov/nchs/nhanes/tutorials/reliabilityofestimates.aspx
• Phillippo DM, Ades AE, Dias S, Palmer S, Abrams KR, Welton NJ. NICE DSU Technical Support Document 18: Methods for population-adjusted indirect comparisons in submissions to NICE. Decision Support Unit, ScHARR, University of Sheffield; 2016. https://sheffield.ac.uk/nice-dsu/tsds/population-adjusted
• Rutterford C, Copas A, Eldridge S. Methods for sample size determination in cluster randomized trials. International Journal of Epidemiology. 2015;44(3):1051-1067. https://doi.org/10.1093/ije/dyv113
• World Health Organization. Vaccination Coverage Cluster Surveys: Reference Manual. WHO/IVB/18.09. Geneva: WHO; 2018. https://www.who.int/publications/i/item/WHO-IVB-18.09