Design Effect

A statistic quantifying how much a complex survey or clustered study's variance differs from what a simple random sample of the same size would give.

Source: Kish 1965

A design effect quantifies how much the variance of an estimate from a complex or clustered study exceeds what a simple random sample of the same size would give. A value above one means the design has lost precision, typically because people within clusters resemble one another, so the sample carries less information than its headcount suggests. Knowing it lets researchers inflate sample sizes to compensate and analyse the data correctly. Measuring the precision lost to design is its purpose. Kirkwood and Sterne (2003) describe this.

Source: Kirkwood & Sterne 2003

A design effect is calculated as the ratio of the variance of an estimate under the actual, complex sampling design to the variance that would be obtained under simple random sampling of the same size. For clustered designs, it is often approximated from the intraclass correlation and the cluster size, increasing with both. So a design effect is calculated by comparing the design's variance with the simple random sampling variance, and for clustering it can be estimated from how correlated observations within clusters are and how large the clusters are, which shows how much the clustering inflates the variance beyond what independent sampling would produce.

Source: Kish 1965

A design effect indicates how much a study's design affects the precision of its estimates compared with simple random sampling: a value of one indicates equivalence, above one indicates greater variance and reduced efficiency, as typically occurs with clustering, and below one indicates improved efficiency, as can occur with stratification. So a design effect indicates the loss or gain in precision due to the sampling or study design, with values above one, common in clustered studies, showing that the design yields less precise estimates than an equivalent simple random sample, which must be accounted for in analysis and sample size planning.

Source: Kish 1965

The design effect is important because complex designs, especially those involving clustering, do not achieve the precision that a simple random sample of the same size would, so ignoring the design effect leads to underestimated variances, overly narrow confidence intervals, and inflated significance. It is also needed to plan adequate sample sizes. So the design effect matters for both valid analysis and proper study design, since accounting for it ensures that the reduced precision of clustered or complex designs is reflected in the standard errors and that studies are made large enough, by inflating the sample size by the design effect, to reach the intended precision.

Source: Kish 1965

The design effect relates to the effective sample size in that the effective sample size is the actual sample size divided by the design effect, giving the size of a simple random sample that would achieve the same precision as the complex design. A design effect above one thus yields an effective sample size smaller than the actual number. So the design effect and effective sample size are directly linked, with the design effect quantifying the inflation of variance and the effective sample size expressing the equivalent independent sample, which together convey how much the design reduces the information per observation, important for analysis and for planning studies with clustered or complex designs.

Source: Kish 1965