Experimental Design

Experimental Design for Trials and Choice Experiments

Experimental design is the set of decisions, made before any data exist, about which units receive which intervention, how many units are studied and how known sources of variation are handled. In health economics the units are usually patients or survey respondents, and the design decides whether a difference in costs and QALYs, or in stated choices, can be read as an effect of the intervention. The same principles govern a randomised controlled trial that carries an economic evaluation and a discrete choice experiment that measures preferences. Sample size is where the economic angle bites hardest, because a trial sized for a clinical endpoint is often too small to answer the cost-effectiveness question.

Fisher's principles of randomisation, replication and blocking

Ronald Fisher's The Design of Experiments, first published in 1935, set out the logic that clinical and economic studies still follow. Fisher argued that the simple precaution of randomisation is enough to guarantee the validity of the test of significance, because chance, and not the experimenter, decides which unit receives which treatment. ICH E9, the international guideline on statistical principles for clinical trials, describes randomisation as a deliberate element of chance in treatment assignment that gives a sound statistical basis for evaluating treatment effects and tends to produce groups with similar prognostic factors, known and unknown.

Replication means applying each treatment to several units. Fisher gave it two purposes. The variance of a mean falls inversely with the number of replications, and replication is the only way to estimate the error against which a difference is judged.

Blocking groups similar units before randomising within each group. Fisher divided his field into compact blocks so that as much as possible of the variation due to soil fell between blocks and was removed from the experimental error. The clinical equivalent is stratified randomisation, usually run as block randomisation within each stratum. ICH E9 notes that stratifying by important baseline prognostic factors can promote balance and helps most in small trials, and that stratification factors should be accounted for in the analysis.

The design written as a statistical model

Every experimental design implies a model for the outcome. Writing that model down shows where precision comes from. For a trial with treatments $i$ randomised within strata or centres $j$, Fisher's randomised block layout gives

$$y_{ijk} = \mu + \tau_i + \beta_j + \varepsilon_{ijk}$$

where $y_{ijk}$ is the outcome for patient $k$ given treatment $i$ in block $j$, $\mu$ is the overall mean, $\tau_i$ is the effect of treatment $i$, $\beta_j$ is the effect of block $j$, and $\varepsilon_{ijk}$ is residual error with variance $\sigma^2$. Because $\beta_j$ is estimated separately, variation between blocks no longer inflates $\sigma^2$, and the model can be fitted by linear regression.

For a two-arm comparison the precision of the estimated effect depends on the arm sizes.

$$\operatorname{Var}(\hat{\tau}) = \sigma^2 \left( \frac{1}{n_1} + \frac{1}{n_0} \right)$$

where $\hat{\tau}$ is the estimated difference in mean outcome, $\sigma^2$ is the within-arm variance, and $n_1$ and $n_0$ are the numbers in the intervention and control arms. For a fixed total, equal allocation gives the smallest variance. With 500 participants a 250 and 250 split gives $\sigma^2 (1/250 + 1/250) = 0.0080\sigma^2$, while a 333 and 167 split gives $\sigma^2 (1/333 + 1/167) = 0.0090\sigma^2$, about 12% higher.

Baseline measurements act like blocking. Borm and colleagues showed that if the baseline and follow-up outcome correlate at $\rho$, an analysis of covariance on groups of $(1 - \rho^2)n$ subjects has the same power as a t-test on groups of $n$.

Design configurations used in health technology assessment

Each layout trades simplicity against efficiency. The NICE manual for technology appraisal (PMG36, section 3.3.2) states a strong preference for high-quality randomised controlled trials for relative treatment effects, with non-randomised studies as a complement or fallback.

ICH E9 calls the parallel group design the most common for confirmatory trials and names carryover as the chief difficulty of crossover designs. For factorial designs it warns that the interaction test may lack power when the sample is sized for main effects. For cluster trials the Cochrane Handbook (chapter 23) gives the design effect as approximately $\text{DE} = 1 + (M - 1)\,\text{ICC}$, where $M$ is the average cluster size and ICC is the intraclass correlation coefficient.

Designing a trial that will carry an economic evaluation

When a trial also supports a cost-effectiveness analysis, clinical design choices constrain the economic one. The ISPOR task force report by Ramsey and colleagues, published in 2015, recommends designing for effectiveness rather than efficacy where possible, collecting resource use and health state utilities directly from participants, and fully integrating economic data collection into the study.

ICH E9 states that the sample size is usually set by the primary objective. That objective is usually clinical, so the economic analysis inherits a sample chosen for another question, while the decision quantity for a health system is incremental net monetary benefit, which carries the variance of both costs and effects. Willan showed that net health benefit lets standard procedures be used for the power and sample size of cost-effectiveness data.

$$n = \frac{2\,(z_{1-\alpha/2} + z_{1-\beta})^2\, \sigma^2}{\Delta^2}$$

where $n$ is the number per arm, $z_{1-\alpha/2}$ and $z_{1-\beta}$ are standard normal quantiles for the two-sided significance level $\alpha$ and the power $(1 - \beta)$, $\sigma$ is the within-arm standard deviation of the outcome, and $\Delta$ is the difference to be detected. For net monetary benefit at a value $\lambda$ per QALY the per-patient variance is

$$\sigma_{NMB}^2 = \lambda^2 \sigma_E^2 + \sigma_C^2 - 2\lambda\rho_{EC}\,\sigma_E \sigma_C$$

where $\sigma_E$ and $\sigma_C$ are the standard deviations of QALYs and costs, and $\rho_{EC}$ is the correlation between a patient's QALYs and costs.

Worked example sizing a two-arm trial for QALYs and net benefit

All numbers in this example are illustrative. A trial of a new rehabilitation service against usual care expects a one-year QALY gain of 0.05 with a within-arm standard deviation of 0.20, and an extra cost of GBP 500 per patient with a standard deviation of GBP 3,000. With a two-sided 5% level and 80% statistical power, $z_{1-\alpha/2} = 1.960$ and $z_{1-\beta} = 0.8416$, so $(1.960 + 0.8416)^2 = 7.849$.

1. Size for QALYs. The sample is $n = 2 \times 7.849 \times 0.04 / 0.0025 = 251.2$, so 252 per arm.

2. Use the baseline measurement. If baseline utility and the one-year QALY outcome correlate at 0.5, the factor $(1 - 0.5^2) = 0.75$ gives $n = 0.75 \times 251.2 = 188.4$, so 189 per arm.

3. Value the QALY gain. The NICE manual (PMG36, sections 6.3.4 to 6.3.8) judges ICERs against a range of GBP 25,000 to GBP 35,000 per QALY gained, and the example uses the lower end. Expected incremental net monetary benefit is $\text{INMB} = 25{,}000 \times 0.05 - 500 = 750$, so GBP 750 per patient.

4. Variance of net benefit with no correlation. The QALY term is $(\lambda\sigma_E)^2 = (25{,}000 \times 0.20)^2 = 25{,}000{,}000$ and the cost term is $\sigma_C^2 = 3{,}000^2 = 9{,}000{,}000$, giving 34,000,000 and a standard deviation of GBP 5,831. The sample is $n = 2 \times 7.849 \times 34{,}000{,}000 / 562{,}500 = 948.8$, so 949 per arm.

5. Allow a positive correlation. With a correlation of 0.3 between QALYs and costs, the covariance term is $(2\lambda\rho_{EC}\,\sigma_E\sigma_C) = 2 \times 25{,}000 \times 0.3 \times 0.20 \times 3{,}000 = 9{,}000{,}000$. The variance falls to 25,000,000, the standard deviation to GBP 5,000, and $n = 2 \times 7.849 \times 25{,}000{,}000 / 562{,}500 = 697.7$, so 698 per arm.

6. Power of the clinically sized trial. With 252 per arm and a standard deviation of GBP 5,000, the standard error of the difference is $\text{SE} = 5{,}000 \times \sqrt{2/252} = 445.4$, and power is $\Phi(750/445.4 - 1.960) = \Phi(-0.276) = 0.39$.

7. Randomise by practice. With 20 patients per practice and an ICC of 0.05, the design effect is $\text{DE} = 1 + 19 \times 0.05 = 1.95$, and the QALY sample becomes $n = 251.2 \times 1.95 = 489.8$, or 490 patients, so 25 practices per arm.

8. Interpretation. If the true net benefit is GBP 750, the trial sized for QALYs has about a 39% chance of a statistically significant result. Claxton argued that decisions should rest on mean net benefit whatever its significance, with its spread relevant only to whether more information is needed. On that view the smaller trial still informs adoption, but its imprecision becomes decision uncertainty.

Experimental design in discrete choice experiments

In a choice experiment the design allocates attribute levels to choice questions, not treatments to patients. The ISPOR task force report by Reed Johnson and colleagues, published in 2013, sets out the requirements for designs that identify the choice-model preference parameters and compares construction approaches without endorsing one. An accompanying editorial by Whitty and Kauf quotes the report's description of experimental design as generating the specific combinations of attributes and levels that respondents evaluate, and notes that every health DCE published between 2001 and 2008 in the review it cited used a fractional factorial design.

Attribute levels are combined so that no two effects are confounded and each can be estimated separately. The editorial notes the report's focus on the trade-off between statistical efficiency and response efficiency.

Where experimental design stops and other designs begin

Randomisation makes the groups differ, apart from chance, only in the intervention received, which is the basis of causal inference in a trial. Generalisability is a separate matter, and the ISPOR trial report prefers effectiveness designs where possible. An observational study does not assign the exposure, so protection against confounding has to be argued from assumptions instead. Blinding and allocation concealment protect a randomised comparison once the sequence exists.

Common errors in designing and reading experiments

Most errors treat a design as simpler or more informative than it is. Each carries into the economic evaluation built on the study.

1. Analysing a structured design as a simple one. Ignoring stratification or clustering misstates standard errors.
2. Reading a non-significant interaction as none. Factorial trials powered for main effects rarely rule out a sizeable interaction.
3. Assuming the clinical sample covers the economics. The example shows a sample adequate for QALYs falling well short for net benefit.

Sources

• Borm GF, Fransen J, Lemmens WA. A simple sample size formula for analysis of covariance in randomized clinical trials. Journal of Clinical Epidemiology. 2007;60(12):1234-1238. https://doi.org/10.1016/j.jclinepi.2007.02.006
• Claxton K. The irrelevance of inference: a decision-making approach to the stochastic evaluation of health care technologies. Journal of Health Economics. 1999;18(3):341-364. https://doi.org/10.1016/S0167-6296(98)00039-3
• Fisher RA. The Design of Experiments. Edinburgh: Oliver and Boyd; 1935. Read in the 1971 ninth edition. https://archive.org/details/in.ernet.dli.2015.199572
• Higgins JPT, Eldridge S, Li T. Chapter 23: Including variants on randomized trials. In Higgins JPT, Thomas J, Chandler J, et al, eds. Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane; last updated October 2019. https://www.cochrane.org/authors/handbooks-and-manuals/handbook/current/chapter-23
• International Council for Harmonisation. ICH E9 Statistical principles for clinical trials. Step 4 version, 5 February 1998. https://database.ich.org/sites/default/files/E9_Guideline.pdf
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). Published 31 January 2022, last updated 13 October 2026. Sections 3.3.2 and 6.3.4 to 6.3.8. https://www.nice.org.uk/process/pmg36
• Ramsey SD, Willke RJ, Glick H, et al. Cost-effectiveness analysis alongside clinical trials II: an ISPOR Good Research Practices Task Force report. Value in Health. 2015;18(2):161-172. https://doi.org/10.1016/j.jval.2015.02.001
• Reed Johnson F, Lancsar E, Marshall D, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health. 2013;16(1):3-13. https://doi.org/10.1016/j.jval.2012.08.2223
• Whitty JA, Kauf TL. Turning the spotlight on experimental design in discrete choice experiments: a focus on pragmatic approaches in health. Value in Health. 2013;16(1):1-2. https://doi.org/10.1016/j.jval.2012.10.015
• Willan AR. Analysis, sample size, and power for estimating incremental net health benefit from clinical trial data. Controlled Clinical Trials. 2001;22(3):228-237. https://doi.org/10.1016/S0197-2456(01)00110-6