Factorial Design

Factorial Design in Randomised Trials and Economic Evaluation

A factorial design crosses two or more interventions inside one randomised trial, so each participant is randomised to every factor at once and the trial can answer a separate question about each intervention. In the simplest 2 × 2 version, participants receive neither treatment, one of the two, or both. The main attraction is efficiency, because each participant contributes to the comparison for every factor, and the design is the only one that lets an interaction between treatments be estimated from randomised comparisons. For health economists the design raises its own problem, since costs and QALYs can interact even when the clinical effects add up neatly, and a decision made one factor at a time can then recommend the wrong treatment combination. The general principles of planning a controlled study belong to experimental design, and crossing the factors adds its own consequences for sample size, analysis and cost-effectiveness decisions.

Cells, margins and the 2 × 2 layout

Each factor has levels, such as active drug and placebo, and each combination forms a cell. ICH E9, the ICH guideline on statistical principles for clinical trials, describes the 2 × 2 design as subjects randomly allocated to one of the four combinations of two treatments, A and B, namely A alone, B alone, both A and B, and neither. The CONSORT extension for factorial trials, published by Kahan and colleagues in 2023, calls the usual version a full factorial design, in which every participant is eligible for every combination. It names two variants that need different methods, a fractional design in which some combinations are omitted and a partial design in which some participants are randomised to only some factors.

The row and column totals are the margins. Montgomery, Peters and Little explain that the usual primary analysis of a 2 × 2 trial compares the margins, so the effect of A comes from everyone allocated A against everyone not allocated A, whatever they received for B. Dakin and Gray call this the at-the-margins analysis and contrast it with an inside-the-table analysis, which treats each cell as a separate treatment arm. ICH E9 also describes crossing $m$ doses of one drug with $n$ doses of another, giving $m \times n$ groups, to estimate a dose-response surface for a combination.

Main effects, interaction and the regression model

A regression model with a product term gives both analyses in one equation. Montgomery and colleagues recommend multivariable regression, especially when cell sizes are unequal.

$$Y_i = \beta_0 + \beta_A x_{Ai} + \beta_B x_{Bi} + \beta_{AB} x_{Ai} x_{Bi} + \varepsilon_i$$

where $Y_i$ is the outcome for participant $i$, $x_{Ai}$ and $x_{Bi}$ equal 1 if the participant was allocated active A or active B and 0 otherwise, $\beta_0$ is the mean in cell 0, $\beta_A$ is the effect of A when B is absent, $\beta_B$ is the effect of B when A is absent, $\beta_{AB}$ is the interaction, and $\varepsilon_i$ is random error. With four cells the model is saturated, so the interaction equals a contrast of cell means.

$$\beta_{AB} = \mu_{ab} - \mu_a - \mu_b + \mu_0$$

where $\mu_k$ is the mean outcome in cell $k$, the definition Dakin and Gray use. A value of zero means the effects are additive on the scale analysed. Dakin and Gray call an interaction super-additive when the effect of the combination is greater than the separate effects added together and sub-additive when it is smaller, and qualitative when the effect of one intervention changes sign as well as size depending on whether the other is given, and the interaction effect page covers scale dependence in depth. Montgomery and colleagues note that for binary outcomes the presence or absence of an interaction depends on the model, because logistic regression assumes effects multiply on the odds scale.

With equal cell sizes, the at-the-margins estimate of A averages its effect across the two levels of B.

$$\Delta_A = \frac{\mu_a + \mu_{ab}}{2} - \frac{\mu_0 + \mu_b}{2} = \frac{(\mu_a - \mu_0) + (\mu_{ab} - \mu_b)}{2}$$

where $\Delta_A$ is the marginal effect of A. It equals the effect of A alone only when $\beta_{AB}$ is zero. The CONSORT extension makes the same point in terms of estimands, noting that a factorial analysis estimates the effect of A in the absence of B only if A and B do not interact, and otherwise estimates the mean effect of A across the strata of B.

Why two questions can share one sample

Montgomery and colleagues give a worked sample size in which intervention A has a target difference of 0.35 standard deviations and B a target of 0.30, with 90% power and a two-sided alpha of 1%. A factorial trial of 664 participants gives 90% power for B and 97% power for A, while a three-arm parallel trial would need 907 participants for the same targets. ICH E9 adds that the 2 × 2 design can evaluate two treatments with the same number of subjects needed for one alone, and has proved particularly valuable for very large mortality trials.

ISIS-2 randomised 17,187 patients with suspected acute myocardial infarction to streptokinase, aspirin, both or neither, and reported that each agent reduced five-week vascular mortality. Vascular deaths were 8.0% among those allocated both agents against 13.2% among those allocated neither, and the separate effects appeared to be additive.

The saving is conditional. ICH E9 states that the efficiency and validity of this use depend on the absence of interaction, and that evidence that this condition is likely to be met should be established in advance from prior information and data. Montgomery and colleagues show that detecting an interaction as large as the main effects, with the same power, needs a fourfold increase in sample size, and with no increase the interaction must be at least twice the size of the main effects.

Choosing and reporting the analysis

In a systematic review of factorial trials with clinically important binary outcomes, McAlister and colleagues found 44 trials, of which 36 (82%) were done for efficiency and 8 (18%) to assess the incremental benefit of combining treatments. Only 2 of 31 comparisons showed a statistically significant interaction, which the authors read as investigators restricting the design to treatments unlikely to interact substantively. The CONSORT extension lists three aims, evaluating more than one intervention without increasing the sample size, testing whether interventions interact, and identifying the best combination.

A common shortcut is two-stage testing, in which a test of interaction decides between the marginal and the cell-by-cell analysis. The CONSORT extension states that using a test of interaction to guide the choice of analysis can introduce bias and is not recommended. Montgomery and colleagues warn that a high p-value for the interaction most likely reflects low power and cannot be taken as evidence of no interaction. The CONSORT extension asks reports to give the estimated interaction and its precision for each primary outcome, and outcomes, harms and adherence for each treatment group as well as for the main comparisons.

Factorial trials in economic evaluation

Dakin and Gray explain why factorial trials are harder to evaluate economically. Interactions may occur more often for costs and QALYs than for clinical endpoints, partly because many utility measures, such as EQ-5D, have diminishing marginal effects built in. Their simulated example has zero interaction on the log-odds scale for clinical events, yet shows a super-additive interaction for cost, a sub-additive interaction for QALYs and a qualitative interaction in net monetary benefit. They add that every study that finds non-zero interactions for costs or QALYs, with at least one treatment more costly and more effective than its comparator, will show a qualitative interaction for net monetary benefit at some ceiling ratio.

The decision rule follows from this. Dakin and Gray argue that when treatments interact, the cells should be treated as mutually exclusive options, compared incrementally against the cost-effectiveness frontier, with the single strategy of highest net monetary benefit chosen. Separate ICER decisions on each factor, which the at-the-margins analysis implies, give the same answer when effects on costs and QALYs are additive, but can pick the wrong combination when net monetary benefit has a qualitative interaction. They propose the opportunity cost of ignoring interactions, the net monetary benefit lost by adopting the combination chosen at the margins instead of the one chosen inside the table. Their recommendations favour including interactions in an economic evaluation unless they are shown to be negligible, and carrying interactions into any extrapolation or multiple imputation model. They note that the at-the-margins analysis can bias the expected value of perfect information, and that Bayesian analyses with informative priors on the interaction terms offer a compromise between including and excluding them.

Worked example with two drugs and a hidden interaction

All numbers are illustrative. A 2 × 2 factorial trial with 250 participants per cell compares drug A and drug B with placebo, and the economic evaluation uses a ceiling ratio of GBP 25,000 per QALY. Net monetary benefit for each cell is $NMB = \lambda Q - C$, where $\lambda$ is the ceiling ratio, $Q$ is mean QALYs per patient and $C$ is mean cost per patient.

Step 1, interactions. For cost, 3,400 minus 2,600 minus 2,800 plus 2,000 gives 0, so costs are additive. For QALYs, 5.07 minus 5.04 minus 5.06 plus 5.00 gives minus 0.03, a sub-additive interaction. For net monetary benefit, 123,350 minus 123,400 minus 123,700 plus 123,000 gives minus GBP 750, which equals 25,000 × (minus 0.03).

Step 2, analysis at the margins. Allocated A, mean cost is (2,600 + 3,400) / 2 = GBP 3,000 and mean QALYs (5.04 + 5.07) / 2 = 5.055. Not allocated A, the figures are (2,000 + 2,800) / 2 = GBP 2,400 and (5.00 + 5.06) / 2 = 5.030. The ICER for A is 600 / 0.025 = GBP 24,000 per QALY, below the ceiling. For B, the marginal cost difference is (2,800 + 3,400) / 2 minus (2,000 + 2,600) / 2 = 3,100 minus 2,300 = GBP 800, and mean QALYs of 5.065 against 5.020 give a difference of 0.045, an ICER of about GBP 17,778. Both pass, so separate decisions recommend giving both drugs.

Step 3, analysis inside the table. Ordered by cost, a against 0 has an ICER of 600 / 0.04 = GBP 15,000, and b against a has 200 / 0.02 = GBP 10,000. The lower ICER further along the frontier means A alone is ruled out by extended dominance. B against 0 gives 800 / 0.06 = GBP 13,333, and ab against b gives 600 / 0.01 = GBP 60,000, above the ceiling. The preferred cell is b, which has the highest NMB.

Step 4, the cost of ignoring the interaction. Adopting both drugs yields GBP 123,350 per patient against GBP 123,700 for B alone, a loss of GBP 350, or 350 / 25,000 = 0.014 QALYs at the ceiling ratio. The marginal result for A hides two different answers. Without B, A gains 0.04 QALYs for GBP 600 and adds GBP 400 of NMB. With B, it gains 0.01 QALYs for the same cost and loses GBP 350. The marginal NMB of GBP 25 is the average of the two.

A trial powered for main effects would estimate a QALY interaction of minus 0.03 imprecisely, so a non-significant test would not have justified the marginal analysis.

Factorial, crossover and multi-arm designs

A factorial trial randomises each participant to a combination of treatments given together, while a crossover design gives the treatments one after another to the same participant to compare them within the person. Dakin and Gray note that three-arm studies of A, B and a control neither offer the factorial efficiency gain nor give a direct estimate of the interaction. Montgomery and colleagues add that a factorial design is contraindicated when the main interest is a head-to-head comparison of A alone against B alone, and unsuitable for interventions that cannot be used together. Dakin and Gray liken factorial trials to subgroup analysis, with stronger inference because patients are randomised to both factors.

Common misreadings of factorial results

Four errors recur. A marginal effect is often reported as if it were the effect of the treatment used alone, although it is an average over the other factor and differs from the effect alone whenever there is an interaction. A non-significant interaction test is read as proof of additivity, despite the low power Montgomery and colleagues describe. Clinical additivity is taken to imply economic additivity, which Dakin and Gray show need not hold. Factors are reported in separate papers, which hides the cell means an inside-the-table decision needs, and Dakin and Gray prefer one paper for all factors unless effects are clearly additive.

Sources

• Dakin H, Gray A. Economic evaluation of factorial randomised controlled trials: challenges, methods and recommendations. Statistics in Medicine. 2017;36(18):2814-2830. https://doi.org/10.1002/sim.7322
• ICH. ICH Topic E9: Statistical Principles for Clinical Trials (CPMP/ICH/363/96). London: European Medicines Agency; September 1998. Section 3.1.3. https://www.ema.europa.eu/en/documents/scientific-guideline/ich-e-9-statistical-principles-clinical-trials-step-5_en.pdf
• ISIS-2 (Second International Study of Infarct Survival) Collaborative Group. Randomised trial of intravenous streptokinase, oral aspirin, both, or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet. 1988;2(8607):349-360. https://pubmed.ncbi.nlm.nih.gov/2899772/
• Kahan BC, Hall SS, Beller EM, et al. Reporting of factorial randomized trials: extension of the CONSORT 2010 statement. JAMA. 2023;330(21):2106-2114. https://doi.org/10.1001/jama.2023.19793
• McAlister FA, Straus SE, Sackett DL, Altman DG. Analysis and reporting of factorial trials: a systematic review. JAMA. 2003;289(19):2545-2553. https://doi.org/10.1001/jama.289.19.2545
• Montgomery AA, Peters TJ, Little P. Design, analysis and presentation of factorial randomised controlled trials. BMC Medical Research Methodology. 2003;3:26. https://doi.org/10.1186/1471-2288-3-26