Random Sampling

Random Sampling in Health Surveys, Value Sets and Cost Studies

Random sampling is what allows a mean cost, a prevalence or a population utility norm measured in a few thousand people to stand for a whole country, because the chance that each person entered the sample is fixed by the design and can be calculated. Statisticians also call it probability sampling. In health economics it sits behind the national surveys that supply costs and age-related utility values to decision models and behind the general-population samples whose preferences become a value set. It answers a different question from the random allocation of patients in a trial, which makes treatment groups comparable but says nothing about whom the trial represents.

Known selection probabilities as the defining condition

The United Nations handbook on household survey samples sets three requirements for probability sampling. Each element must have a known mathematical chance of being selected, that chance must be greater than zero, and it must be numerically calculable. The chances need not be equal and can vary with the objectives of the survey. The handbook explains that this mathematical basis lets estimates be inferred to the population from which the sample was drawn and lets sampling errors be estimated from the sample itself, so a standard error and confidence interval can be reported.

Quota, convenience, purposive and random walk samples do not meet these conditions. The same handbook states that no statistical theory guides their use, that survey estimates from them will be biased, and that the size and often the direction of the bias will be unknown. A sample can look like the population on age and sex and still fail this test, because the people who were reachable or willing may differ on the outcome being measured.

The probabilities refer to a defined target population and a sampling frame, the list or set of areas from which the sample is actually drawn. Groups that the frame leaves out have a zero chance of selection, and the handbook recommends redefining the target population to match what the survey really covers. The Medical Expenditure Panel Survey (MEPS), for example, covers the US civilian noninstitutionalised population, and the address sample for the England EQ-5D-5L valuation study excluded communal establishments such as care homes.

Simple, systematic, stratified, cluster and multistage designs

Most health surveys combine several random selection methods. The UN handbook describes simple random sampling as rarely used in large household surveys because listing and travel make it costly, but it remains the baseline for comparing the efficiency of other designs.

Strata should be internally similar and different from each other, while clusters work best when they are internally varied. Clusters usually contain similar people, so a clustered sample carries less information than its size suggests, a loss measured by the design effect. In a multistage design the overall probability is the product of the stage probabilities. In the UN handbook's illustration, 10 villages are drawn at random from 100 and 1 in 5 households is then drawn systematically in each, so every household has an overall chance of $\frac{10}{100} \times \frac{1}{5} = \frac{1}{50}$.

Weights that turn a sample into population estimates

Because selection probabilities can differ, each respondent carries a weight. The UN handbook defines the base or design weight as the reciprocal of the probability of selection, and it states that estimates which ignore weights for differential selection rates will in general be biased.

$$w_i = \frac{1}{p_i}, \qquad \bar{y}_w = \frac{\sum_i w_i y_i}{\sum_i w_i}$$

where $w_i$ is the weight of respondent $i$, $p_i$ is that respondent's probability of selection, $y_i$ is the observed value such as annual cost or an EQ-5D index score and $\bar{y}_w$ is the weighted mean. Base weights are then adjusted for non-response and calibrated so that totals by age and sex agree with known population figures.

For a stratified sample the estimate and its variance are built from the strata, as set out in the handbook's annex.

$$\bar{y}_{st} = \sum_h W_h \bar{y}_h, \qquad V(\bar{y}_{st}) = \sum_h W_h^2 (1 - f_h) \frac{s_h^2}{n_h}$$

where $W_h$ is the share of the population in stratum $h$, $\bar{y}_h$ is the sample mean in that stratum, $s_h$ is its sample standard deviation, $n_h$ is the number sampled and $f_h$ is the sampling fraction, whose correction $(1 - f_h)$ can be dropped when the fraction is small.

Worked example with an oversampled older stratum

All numbers in this example are illustrative. A health authority wants the mean annual NHS cost per adult in a population of 1,000,000, of whom 200,000 are aged 65 or over. It draws a stratified random sample of 500 older and 500 younger adults, oversampling the older group because their costs are more variable.

1. Compare unweighted and weighted means. The unweighted mean is $\frac{2400 + 600}{2} = 1500$ GBP. The weighted mean gives the older stratum its population share of 0.2 and the younger stratum 0.8, so it is GBP 480 plus GBP 480, or GBP 960, and ignoring the weights would overstate the cost per adult by 56%. Multiplying by 1,000,000 adults gives a budget estimate of GBP 960 million.

2. Compute the standard error. The sampling fractions are tiny, so the correction is dropped.

$$V(\bar{y}_{st}) = \frac{0.2^2 \times 3000^2}{500} + \frac{0.8^2 \times 1200^2}{500} = 720 + 1843.2 = 2563.2$$

where the result is in GBP squared. The standard error is 50.63 GBP and the 95% interval is 960 plus or minus 99.2, or GBP 861 to GBP 1,059.

3. Compare with other allocations. A proportionate sample of 200 older and 800 younger adults gives a variance of 1,800 + 1,152 = 2,952 and a standard error of 54.33. The population variance of cost is 2,952,000 within strata plus 518,400 between them (stratum means lie 1,440 and 360 from GBP 960), or 3,470,400. A simple random sample of 1,000 therefore has a standard error of 58.91, the square root of 3,470.4, and the oversampled design has a design effect of 2,563.2 / 3,470.4 = 0.74. The UN handbook notes that unequal weights across strata can improve precision when stratum variances differ, but raise the variance when they are the same. With a common standard deviation of GBP 2,000 in both strata, the same 500 and 500 split would have a variance 1.36 times that of proportionate allocation.

4. Interpret. With full response, the random design makes the weighted mean an unbiased estimator of the population mean, and its standard error of GBP 50.63 is the uncertainty a model should carry in a probabilistic sensitivity analysis.

Uses in costing, utilities and value sets

MEPS, sponsored by the US Agency for Healthcare Research and Quality, is a nationally representative sample of the US civilian noninstitutionalised population and provides data on healthcare use, expenditures and sources of payment. Each annual panel is a subsample of households that responded to the National Health Interview Survey, a multistage area probability sample. The MEPS methodology report for 2023 states that analyses must incorporate the sample weights and the stratification and clustering of the design, and that software assuming a simple random sample will usually underestimate the variance of MEPS estimates. Cost-of-illness studies based on MEPS therefore need its strata and cluster variables as well as its weights.

In England each wave of the Health Survey for England (HSE) from 2022 to 2025 was planned to include around 8,000 adults and 2,000 children recruited from everyone living at randomly selected private addresses. A report by the NICE Decision Support Unit estimated EQ-5D-3L by age and sex from the 2014 wave, noting that the HSE uses a clustered, multistage stratified design with weights for selection, non-response and population profile, and that its estimates take all three into account. The NICE manual (PMG36, section 4.3.7) asks that baseline utility values extrapolated over long horizons be adjusted to reflect falling general population health-related quality of life with age, using a recent and robust population source. The same report also analysed an online panel survey from April 2020 that was not intended to sample from a representative group of the UK population. It found that the differences from HSE 2014 could not be attributed to sample characteristics, pointed instead to fieldwork during the Covid-19 lockdown, and advised that the age profile based on HSE 2014 was the best one to use for the time being.

Value sets rely on the same logic. The NICE manual (section 4.3.4) states that the valuation of health-related quality of life should be based on public preferences from a representative sample of the UK population using a choice-based method. In the EQ-5D-5L valuation study for England, Devlin and colleagues randomly selected 2,220 addresses from 66 primary sampling units based on postcode sectors from the Postcode Address File, selected addresses systematically within each unit, and chose one adult per household at random with no substitutes. The response rate was 47.7%, with 996 people completing the time trade-off and discrete choice experiment tasks, and the modelling adjusted for the age distribution of the population in England.

Random sampling compared with random allocation

Random sampling decides who is studied, while randomisation decides which treatment each participant receives. The NICE manual (section 3.3.5) explains that randomisation makes baseline differences between trial groups, including unmeasured ones, a matter of chance, which is a property of internal validity. External validity is assessed separately, by whether the results apply to wider patient groups and routine practice (section 3.3.6). Trial participants are volunteers who meet eligibility criteria, and the ICH E9 guideline on statistical principles for clinical trials states that no individual trial can be expected to be totally representative of future users. A randomised controlled trial therefore gives a fair comparison without a random sample, and a survey can give a representative estimate without any comparison.

A third use of the phrase belongs to modelling. Monte Carlo simulation and Latin hypercube sampling draw random values from parameter distributions to propagate uncertainty through a model. These draws come from a probability distribution, not from a population of people, and their precision depends on the number of simulations.

Limitations and common misreadings

Most problems arise after the random draw, when the realised sample departs from the designed one or the design is ignored in analysis. Each of the errors below changes either the estimate or its standard error.

1. Treating haphazard selection as random. Interviewing whoever is available or recruiting from an online panel with quotas gives no known selection probabilities, so the bias cannot be quantified.
2. Ignoring non-response. A random invitation with a response rate near one half can still give a biased estimate if responders differ from non-responders, which weighting can reduce only for characteristics that are measured.
3. Forgetting frame coverage. People missing from the frame have no chance of selection, so estimates cover only the framed population.
4. Using unweighted means from an unequal-probability design. The worked example shows a 56% overstatement from this error alone.
5. Computing standard errors as if the sample were simple random. Clustering usually inflates the variance, and ignoring it understates the uncertainty passed to a decision model.
6. Confusing random sampling with random allocation. A randomised trial supports causal comparison but is not thereby generalisable.

Sources

• United Nations Statistics Division. Designing Household Survey Samples: Practical Guidelines. Studies in Methods, Series F No. 98. New York: United Nations; 2005. Chapters 3, 6 and 7 and Annex A.2. https://unstats.un.org/unsd/demographic/sources/surveys/Handbook23June05.pdf
• Agency for Healthcare Research and Quality. Methodology Report #38: Estimation Procedures for the 2023 Medical Expenditure Panel Survey Household Component. Rockville, MD: AHRQ. Sections 1.1 and 6. https://meps.ahrq.gov/data_files/publications/mr38/mr38.shtml
• Health Research Authority. The Health Survey for England 2022 to 2025. Research summary, REC reference 21/EM/0144. https://www.hra.nhs.uk/planning-and-improving-research/application-summaries/research-summaries/the-health-survey-for-england-2022-to-2025/
• Hernández Alava M, Pudney S, Wailoo A. Estimating EQ-5D by Age and Sex for the UK. NICE Decision Support Unit report. Sheffield: University of Sheffield; 12 January 2022. https://sheffield.ac.uk/nice-dsu/methods-development/estimating-eq-5d
• Devlin NJ, Shah KK, Feng Y, Mulhern B, van Hout B. Valuing health-related quality of life: an EQ-5D-5L value set for England. Health Economics. 2018;27(1):7-22. https://doi.org/10.1002/hec.3564
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 2022, last updated 13 October 2026. Sections 3.3.5, 3.3.6, 4.3.4 and 4.3.7. https://www.nice.org.uk/process/pmg36/chapter/evidence and https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2
• International Council for Harmonisation. ICH E9 Statistical Principles for Clinical Trials. Step 4, 5 February 1998. Section 2.2.1. https://database.ich.org/sites/default/files/E9_Guideline.pdf