Efficient Design

Efficient Design in Discrete Choice Experiments

An efficient design chooses the choice tasks of a discrete choice experiment so that the preference parameters are estimated as precisely as possible for a given number of tasks and respondents. In health this matters because samples of patients or clinicians are often small and costly to recruit, and because the estimates feed willingness-to-pay figures, value sets and benefit-risk weights used in decisions. Efficiency is always defined for a stated model and stated parameter values, which is both its strength over an orthogonal design and its main risk.

From information to D-error

For a multinomial logit model the precision of the estimates is described by the asymptotic variance matrix, the inverse of the Fisher information matrix. Unlike in linear regression, that information depends on the parameters themselves, because it depends on the choice probabilities.

$$I(\beta) = \sum_{s=1}^{S} Z_s'\left(P_s - p_s p_s'\right) Z_s, \qquad \Omega(\beta) = I(\beta)^{-1}$$

where $Z_s$ holds the attribute levels of the alternatives in choice set $s$, $p_s$ is the vector of their choice probabilities at parameters $\beta$, $P_s$ is a diagonal matrix with $p_s$ on its diagonal, and $\Omega(\beta)$ is the asymptotic variance matrix of the estimates. The most common summary of $\Omega$ is the D-error.

$$\text{D-error} = \left|\Omega(\beta)\right|^{1/K}$$

where $K$ is the number of parameters and the vertical bars denote the determinant. Lower D-error means more precise estimates, and a D-efficient design is one with the lowest D-error that can be found. A-error uses the average of the variances instead of the determinant. D-efficiency, a related measure reported by some software, rises as D-error falls and is often scaled from 0 to 100 against an ideal design, so the two should not be confused.

Priors and Bayesian efficient designs

Because D-error depends on $\beta$, the designer must assume parameter values before data are collected. Setting $\beta = 0$ gives a design efficient only if respondents are indifferent to every attribute, which reproduces the logic of an orthogonal design. Point priors from a pilot or the literature give a design tuned to those values, and a Bayesian efficient design averages D-error over a prior distribution, which protects against poorly chosen point priors. Rose and Bliemer and the ISPOR task force reported by Reed Johnson and colleagues describe these options and the software that searches for designs.

Worked example comparing two sets of four choice pairs

The figures in this example are illustrative. Two attributes, effectiveness and side effects, are coded plus or minus 1, and the prior is $\beta = (1, 1)$. An orthogonal set of four pairs includes two pairs in which one alternative is better on both attributes.

$$I_{\text{orth}}(\beta) = \begin{pmatrix} 2.141 & -1.859 \\ -1.859 & 2.141 \end{pmatrix}, \qquad \text{D-error} = \left(\frac{1}{2.141^2 - 1.859^2}\right)^{1/2} = 0.94$$

where the determinant of the information matrix is 1.130 and the determinant of its inverse is 0.885. A second set uses the pairs (1, 1) against (−1, 1), (1, 1) against (1, −1), (1, −1) against (−1, 1), and (−1, −1) against (1, −1), none of which offers an alternative better on every attribute.

$$I_{\text{eff}}(\beta) = \begin{pmatrix} 1.840 & -1.000 \\ -1.000 & 1.420 \end{pmatrix}, \qquad \text{D-error} = 0.79$$

where the determinant of the information matrix is 1.613. The second set gives about 16% lower D-error at the prior, so it estimates the parameters more precisely for the same number of tasks. If the true parameters were zero, the ranking would reverse, with D-error of 0.25 for the orthogonal set and 0.45 for the second set, which is why the choice of prior matters.

Practical choices in an efficient design

Efficiency is computed for one model, so a design optimised for a simple multinomial logit may not suit a mixed logit with preference heterogeneity or a model with interactions. Designers also restrict implausible combinations, block the design across respondents, and decide whether to include an opt-out, and each choice changes the design matrix and its efficiency. The ISPOR task force notes that statistical efficiency can conflict with how easy the tasks are to answer, since the most informative sets often ask respondents to trade off close alternatives.

Interpretation and use in health preference studies

A lower D-error at the prior means smaller expected standard errors, so a study can reach a given precision with fewer respondents or tasks. That gain is real only if the priors are near the truth and the analysis model matches the design model. Reporting the criterion, the priors and their source, the software and the final design allows others to judge both conditions.

Common misreadings of efficiency

Labelling a D-error as D-efficiency, or reading a higher D-error as better, inverts the criterion. Comparing D-errors computed under different priors, numbers of parameters or coding schemes is not meaningful. Treating an efficient design as efficient for any model ignores that efficiency was defined for one specification, and treating it as free of bias confuses precision with validity, which depends on the model and on respondents' behaviour.

Sources

• Huber J, Zwerina K. The importance of utility balance in efficient choice designs. Journal of Marketing Research. 1996;33(3):307-317.
• Rose JM, Bliemer MCJ. Constructing efficient stated choice experimental designs. Transport Reviews. 2009;29(5):587-617.
• Reed Johnson F, Lancsar E, Marshall D, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health. 2013;16(1):3-13.
• Louviere JJ, Hensher DA, Swait JD. Stated Choice Methods: Analysis and Applications. Cambridge University Press; 2000.
• Train KE. Discrete Choice Methods with Simulation. 2nd ed. Cambridge University Press; 2009.