Choice Experiment Design

Choice Experiment Design in Health Preference Studies

Every estimate that a discrete choice experiment reports, whether a marginal willingness to pay for shorter waits or the weight patients place on a risk of side effects, can only be as good as the plan that decided what respondents were asked. That plan fixes the attributes and their levels, how many alternatives appear together, whether an opt-out is offered, how many tasks each person answers and how the tasks are shared out across versions of the questionnaire. Once data are collected, those decisions cannot be undone, so a preference parameter left out of the design cannot be recovered by a cleverer model. The ISPOR task force report on experimental design by Johnson and colleagues illustrates the scale of the problem with four attributes at three levels each, which give 81 possible profiles and 3,240 possible two-alternative choice questions, far more than any respondent can answer.

Attributes and levels as the first design decision

Attributes are the features that describe each alternative, such as waiting time, chance of improvement, mode of delivery or cost to the patient, and levels are the values each attribute can take. The list of attributes decides what trade-offs can be observed at all, and the levels decide the range over which each preference is estimated. A preference estimated for waits of 2 to 12 weeks says nothing reliable about a wait of a year.

Attribute development is usually qualitative work before any statistics. Coast and colleagues observed in 2012 that attribute generation for DCEs is often poorly reported, so its rigour is hard to judge. Reviewing the health literature, Helter and Boehler drew out four stages of a general attribute development process, namely raw data collection, data reduction, removing inappropriate attributes and wording. The ISPOR checklist for conjoint analysis applications in health, reported by Bridges and colleagues, gives attributes and levels their own item, separate from the construction of tasks and the experimental design.

Two choices at this stage shape the economics later. Including a cost attribute is what allows a marginal rate of substitution to be expressed as willingness to pay, so a design without one can rank features but cannot price them. Treating an attribute as categorical or as linear changes the number of parameters to be estimated, which in turn changes how many choice tasks the design needs.

Full factorial and fractional factorial plans

A full factorial lists every combination of attribute levels. Its size is the product of the numbers of levels, so the number of profiles multiplies with each attribute added.

$$N = \prod_{k=1}^{K} L_k$$

where $N$ is the number of profiles in the full factorial, $K$ is the number of attributes and $L_k$ is the number of levels of attribute $k$.

Choice questions multiply the problem again, because profiles must be combined into sets of alternatives. With two alternatives per task, the number of distinct unordered pairs of different profiles is $N(N-1)/2$, which is how the ISPOR example reaches 3,240 pairs from 81 profiles. Johnson and colleagues note that the feasible number of choice questions is less than the full factorial of all possible combinations of attribute levels.

A fractional factorial keeps a subset of the full factorial chosen so that the effects of interest can still be estimated. The price of a fraction is aliasing, in which some effects can no longer be separated from others. Regular fractions are classed by resolution. In a resolution III plan, main effects are clear of one another but may be aliased with two-way interactions, while a resolution V plan keeps main effects and two-way interactions clear of each other, assuming three-way and higher interactions are negligible. Resolution is defined for regular fractions, so for mixed-level plans such as the 3 by 3 by 2 by 4 example below the precise description is an orthogonal array of strength 2 for main effects or strength 4 for main effects and two-way interactions. The classical factorial design of a clinical trial uses the same logic for treatments, with patients in place of choice tasks.

Which preference effects a design can identify

The design decides which terms the utility function may contain. A general specification with main effects and two-way interactions can be written as follows.

$$V_{nj} = \sum_{k=1}^{K} \beta_k x_{njk} + \sum_{k<l} \gamma_{kl} x_{njk} x_{njl}$$

where $V_{nj}$ is the systematic utility of alternative $j$ for respondent $n$, $x_{njk}$ is the coded level of attribute $k$ in that alternative, $\beta_k$ is the main effect of attribute $k$ and $\gamma_{kl}$ is the interaction effect between attributes $k$ and $l$.

A main-effects design sets every $\gamma_{kl}$ to zero before the study starts. Johnson and colleagues point out that leaving interaction terms out assumes in advance that the interactions are zero or, if not, independent of the other attribute effects. If a patient values a faster appointment more when the chance of improvement is high, a main-effects design forces that preference into the separate main effects, where it cannot be seen.

Counting parameters gives a first check. With categorical coding, each attribute with $L_k$ levels needs $L_k-1$ parameters, and an interaction between two attributes needs the product of their parameter counts. Each choice task with $J$ alternatives supplies at most $J-1$ independent comparisons.

$$S(J-1) \geq P$$

where $S$ is the number of distinct choice tasks in the design, $J$ is the number of alternatives in each task and $P$ is the number of parameters to be estimated. Meeting the count is necessary but not sufficient, because the tasks must also vary the attributes in ways that separate each parameter from the others.

Orthogonal and efficient construction

The two main families of construction differ in what they optimise. An orthogonal design varies attribute levels independently so that the columns of the design are uncorrelated, which removes multicollinearity among attributes. Johnson and colleagues treat a design as strictly orthogonal only when it is also balanced, with each level of an attribute appearing equally often. For a nonlinear model such as the multinomial logit model, the information a task provides depends on the parameter values, so an orthogonal and balanced design need not be the most efficient one.

An efficient design chooses tasks to make the variance of the parameter estimates small under an assumed model and prior values, and the D-efficient design is one type, built on the determinant of the parameter covariance matrix. The ISPOR report discusses Bayesian approaches, in which the design is built around a prior distribution of plausible parameter values instead of fixed guesses. In their 2012 review of health DCEs published from 2001 to 2008, de Bekker-Grob, Ryan and Gerard identified 114 DCEs and reported a move towards fewer attributes, more choice sets and designs that were statistically more efficient. The ISPOR report does not endorse any one approach and gives guidance for choosing a design suited to the study.

Johnson and colleagues set statistical efficiency against what they call response efficiency, the measurement error that comes from respondents' inattention and other unobserved contextual influences. Statistical efficiency improves with many difficult trade-off questions, while response efficiency improves with fewer and easier ones. Overall design efficiency depends on both.

Labelled and unlabelled alternatives, opt-out and status quo

The alternatives in a choice set can be generic, such as treatment A and treatment B, or labelled with meaningful names, such as particular screening tests. A labelled design allows alternative-specific constants and coefficients, which adds parameters, and the label brings in whatever respondents associate with the name. De Bekker-Grob and colleagues ran labelled and unlabelled versions of a colorectal cancer screening DCE in the Netherlands and reported that the labels influenced choices, with about 22% of respondents showing dominant preferences for the test labels.

An opt-out or status quo option lets respondents decline every offered alternative, which mirrors real decisions where doing nothing is possible and is needed to predict uptake. The opt-out also adds an alternative-specific constant to the model and changes the information each task provides. Veldwijk and colleagues gave 781 Dutch patients with type 2 diabetes nine tasks with an opt-out and nine forced-choice tasks. About a fifth always opted out, respondents who met the opt-out in the first part of the questionnaire and those with lower education opted out significantly more often, and the forced and unforced models found different significant attributes but the same relative importance. The authors suggested that a dual response design, which asks a forced choice first and then whether the respondent would take it up, might give better data than a direct opt-out.

Number of choice tasks and blocking

When a design has more tasks than one respondent should answer, it is split into blocks, and each respondent is randomly assigned one block. Blocking keeps the burden per person manageable while the sample as a whole covers every task, as long as each block is answered by enough respondents.

The number of tasks per respondent trades information against fatigue. In a web survey about dental services, Bech, Kjaer and Lauridsen presented 1,053 respondents with 5, 9 or 17 choice sets. Response rates did not differ and self-reported certainty showed no systematic difference, but some willingness-to-pay estimates differed with the number of sets, and respondents who answered 17 sets had somewhat higher response variance than those who answered 5. They concluded that respondents could handle 17 choice sets without problems in that setting. The total sample is a separate decision, covered by sample size calculation, because repeated tasks from one respondent are not independent observations.

Dominance checks and pilot testing

An alternative dominates another when it is at least as good on every attribute and better on at least one. Johnson and colleagues note that dominated alternatives can arise in orthogonal designs when attribute levels are naturally ordered a priori, as with cost or risk. A task with a dominant option tells the analyst little about trade-offs, so designs usually screen such tasks out, and some studies deliberately keep one as a test of attention or understanding.

Failing a dominance test does not make a response irrational. Lancsar and Louviere argued in 2006 that deleting so-called irrational responses can remove valid preferences, introduce sample selection bias and reduce the statistical efficiency and power of the estimated models, and they advised caution about deleting responses from stated preference experiments.

Piloting checks that respondents read the attributes as intended and can cope with the number of tasks. For efficient designs a pilot can also supply the prior parameter values for the final design, linking this step back to the construction step. The Bridges checklist gives instrument design and data collection their own items alongside the experimental design.

Illustrative worked example of profiles, choice sets and parameters

All numbers are illustrative. A health service plans a DCE on physiotherapy for back pain with four attributes. Waiting time has 3 levels (2, 6 or 12 weeks), chance of improvement has 3 levels (40, 60 or 80 per cent), appointment mode has 2 levels (face to face or video) and cost to the patient has 4 levels (GBP 0, GBP 10, GBP 25 or GBP 50).

Step 1. Full factorial. The number of profiles is 3 × 3 × 2 × 4 = 72.

Step 2. Possible paired tasks. The number of unordered pairs of different profiles is

$$\frac{72 \times 71}{2} = 2556$$

where 72 is the number of profiles and 71 is the number of other profiles each can be paired with. No respondent can answer 2,556 questions, so a fraction is needed.

Step 3. Main-effects parameters. With categorical coding the parameters are 2 for waiting time, 2 for chance of improvement, 1 for mode and 3 for cost, so 2 + 2 + 1 + 3 = 8.

Step 4. One planned interaction. Waiting time by chance of improvement adds 2 × 2 = 4 parameters, so the model has 12. All six two-way interactions would add 4 + 2 + 6 + 2 + 6 + 3 = 23 parameters, for 31 in total.

Step 5. Checking the design size. The team chooses 24 paired tasks. Each supplies one comparison, so 24 × 1 = 24, which exceeds 12 but falls short of 31. The design can support main effects and the one planned interaction, but it cannot identify every two-way interaction, and any omitted interaction is assumed to be zero.

Step 6. Blocking and sample. The 24 tasks are split into 2 blocks of 12. With 400 respondents, 200 answer each block, giving 400 × 12 = 4,800 choices in total, and each task is answered by 200 people.

Step 7. Adding an opt-out. An opt-out adds one constant, so the model has 13 parameters. Each task now has three alternatives, so the count becomes 24 × 2 = 48, well above 13. A choice against the opt-out still carries attribute information, because it compares the utility of an offered option with a constant, but the count of 48 overstates the information on attribute trade-offs, since the opt-out itself has no attribute levels to vary.

The arithmetic shows why the planned model has to be written down before the design is built. If the team later wanted to test whether cost sensitivity varies with appointment mode, an interaction left out of the plan, the data could not support that test without stronger assumptions.

How design choices reach economic evaluation

DCE results enter health economics as willingness-to-pay values for cost-benefit analysis, as weights in benefit-risk assessment and as evidence on likely uptake of a service. Each use depends on the design. A willingness-to-pay figure rests on the cost levels chosen and on the cost coefficient being identified cleanly, which a design with correlated cost and quality attributes can undermine. Uptake predictions need an opt-out or status quo, because a forced-choice design only shows which of the offered options respondents prefer.

Preference heterogeneity raises the bar further. A random parameter model adds a spread parameter for each random coefficient, and a latent class model repeats the coefficients for every class, so both need more information than a simple logit model with the same attributes. A design built only for mean main effects may leave little precision for the heterogeneity analyses that a decision maker most wants.

Boundaries with neighbouring design concepts

Choice experiment design is the whole plan, orthogonal, efficient and D-efficient designs are ways of building its matrix of tasks, and a choice set is one task within it. Experimental design in the wider sense includes trials with randomised groups, while choice experiment design allocates attribute levels to hypothetical alternatives. Conjoint analysis designs for ratings or rankings share the factorial foundations, and best-worst scaling uses designs of its own.

Common misreadings of choice experiment design

A design that is orthogonal is not automatically efficient for a logit model, and a design with a low D-error is not automatically easy to answer. Both measures describe the design matrix, not the respondent's experience of it.

A main-effects design does not show that interactions are absent. It assumes them away, and a later finding of no interaction from such data reflects the design, not the preferences. Similarly, a block that looks balanced in the full design may be unbalanced on its own, so results that differ by block may reflect the design as much as the respondents.

Sources

• Bech M, Kjaer T, Lauridsen J. Does the number of choice sets matter? Results from a web survey applying a discrete choice experiment. Health Economics. 2011;20(3):273-286. https://doi.org/10.1002/hec.1587
• Bridges JFP, Hauber AB, Marshall D, et al. Conjoint analysis applications in health, a checklist. A report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health. 2011;14(4):403-413. https://doi.org/10.1016/j.jval.2010.11.013
• Coast J, Al-Janabi H, Sutton EJ, et al. Using qualitative methods for attribute development for discrete choice experiments: issues and recommendations. Health Economics. 2012;21(6):730-741. https://doi.org/10.1002/hec.1739
• de Bekker-Grob EW, et al. Labeled versus unlabeled discrete choice experiments in health economics: an application to colorectal cancer screening. Value in Health. 2010;13(2):315-323. https://doi.org/10.1111/j.1524-4733.2009.00670.x
• de Bekker-Grob EW, Ryan M, Gerard K. Discrete choice experiments in health economics: a review of the literature. Health Economics. 2012;21(2):145-172. https://doi.org/10.1002/hec.1697
• Helter TM, Boehler CEH. Developing attributes for discrete choice experiments in health: a systematic literature review and case study of alcohol misuse interventions. Journal of Substance Use. 2016;21(6):662-668. https://doi.org/10.3109/14659891.2015.1118563
• Johnson FR, Lancsar E, Marshall D, et al. Constructing experimental designs for discrete-choice experiments. Report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health. 2013;16(1):3-13. https://doi.org/10.1016/j.jval.2012.08.2223
• Lancsar E, Louviere J. Deleting 'irrational' responses from discrete choice experiments: a case of investigating or imposing preferences? Health Economics. 2006;15(8):797-811. https://doi.org/10.1002/hec.1104
• Louviere JJ, Hensher DA, Swait JD. Stated Choice Methods: Analysis and Applications. Cambridge: Cambridge University Press; 2000.
• Veldwijk J, Lambooij MS, de Bekker-Grob EW, Smit HA, de Wit GA. The effect of including an opt-out option in discrete choice experiments. PLoS ONE. 2014;9(11):e111805. https://doi.org/10.1371/journal.pone.0111805