Orthogonal Design

Orthogonal Design in Discrete Choice Experiments

Orthogonal designs were the standard way to build the choice tasks of a discrete choice experiment in health, and many published valuation and preference studies still use them. Their appeal is that each attribute varies independently of the others, so its effect can be separated from theirs. That property carries over fully to linear models, but a choice experiment is analysed with a logit model, and for logit models orthogonality of the attribute levels does not make the estimates uncorrelated and does not make the design efficient. Knowing where the property holds explains why health preference research has largely moved to efficient designs, and when an orthogonal design is still a reasonable choice.

What orthogonality means in a design matrix

A design is orthogonal when the columns of its coded design matrix are uncorrelated, so that every pair of levels of two attributes appears together equally often. Orthogonal main-effects plans are usually fractions of a full factorial, chosen from catalogued orthogonal arrays so that the number of profiles stays manageable. Louviere, Hensher and Swait set out how such fractions are built and how profiles are then grouped into choice sets.

$$X'X = \text{diag}(k_1, k_2, \ldots, k_p)$$

where $X$ is the coded design matrix with one column per parameter and $k_j$ is the sum of squares of column $j$. All off-diagonal terms are zero, so the columns share no information.

Why orthogonality works for linear models

In a linear regression the variance of the least squares estimates is proportional to the inverse of $X'X$. When that matrix is diagonal, its inverse is diagonal too, so the estimates are uncorrelated, and for a given number of runs an orthogonal and balanced design minimises their variance. This is the setting in which orthogonal arrays were developed, and it is why rating and ranking conjoint studies analysed by regression benefited from them.

Why orthogonality is not enough for logit models

A choice experiment is analysed with a multinomial logit or a related model, where the information a choice set provides depends on the choice probabilities, and those depend on the parameters being estimated. Huber and Zwerina showed that an efficient choice design needs level balance, minimal overlap, utility balance and orthogonality together, and that orthogonality of the attribute levels alone does not guarantee efficiency once the parameters are not zero.

$$I(\beta) = \sum_{s=1}^{S} Z_s'\left(P_s - p_s p_s'\right) Z_s$$

where $I(\beta)$ is the Fisher information matrix of the multinomial logit model, $Z_s$ holds the attribute levels of the alternatives in choice set $s$, $p_s$ is the vector of their choice probabilities at parameters $\beta$, and $P_s$ is a diagonal matrix with $p_s$ on the diagonal. Only when every alternative in a set is equally likely, which happens at $\beta = 0$, does the information matrix of an orthogonal design keep the diagonal form of $X'X$. At realistic parameter values the estimates become correlated and less precise, and choice sets in which one alternative is better on every attribute add almost no information.

Worked example with two attributes and four choice pairs

The figures in this example are illustrative. Two attributes, effectiveness and side effects, are each coded plus or minus 1, and four pairs are built from an orthogonal plan, namely (1, 1) against (−1, −1), (1, −1) against (−1, 1), (−1, 1) against (1, −1), and (−1, −1) against (1, 1). At $\beta = (0, 0)$ every choice probability is 0.5.

$$I(0) = \begin{pmatrix} 4 & 0 \\ 0 & 4 \end{pmatrix}, \qquad \text{D-error} = \left|I(0)^{-1}\right|^{1/2} = 0.25$$

where the zero off-diagonal terms mean uncorrelated estimates, and D-error, the square root of the determinant of the variance matrix for two parameters, summarises their joint imprecision. If respondents in fact value both attributes with $\beta = (1, 1)$, the first and last pairs offer an alternative that is better on both attributes, chosen with probability 0.982.

$$I(\beta) = \begin{pmatrix} 2.141 & -1.859 \\ -1.859 & 2.141 \end{pmatrix}, \qquad \text{D-error} = 0.94$$

where the large off-diagonal terms show strongly correlated estimates. D-error rises almost fourfold, and a non-orthogonal set of four pairs without dominated alternatives gives a lower D-error of 0.79 at the same parameters. The example shows the general point that the same orthogonal plan can be efficient or poor depending on the true preferences.

When an orthogonal design is still a reasonable choice

Orthogonal designs need no assumptions about parameter values, which suits exploratory studies with new attributes and no pilot data. They are transparent, easy to reproduce from published arrays, and robust in the sense that they are never tuned to wrong priors. The ISPOR task force on experimental design for discrete choice experiments, reported by Reed Johnson and colleagues, describes orthogonal and efficient approaches as both acceptable, with the choice depending on prior information, the model to be estimated and practical constraints.

Relation to efficient designs and other choices

An efficient design chooses choice sets to minimise D-error or a similar criterion at assumed parameter values, often averaged over a prior distribution in a Bayesian efficient design. It gives up orthogonality where that buys precision. Orthogonality also says nothing about removing dominated or implausible combinations, about blocking the design across respondents, or about whether interactions can be estimated, all of which need separate decisions.

Common misreadings of orthogonality

The most frequent error is to claim that an orthogonal design gives uncorrelated or unbiased estimates in a logit analysis. Orthogonality holds for the attribute levels shown, not for the information in the choices respondents make, and bias is a property of the model and data, not of orthogonality. Assuming that an orthogonal plan remains orthogonal after dominated pairs are deleted, or after an opt-out alternative is added, is another, since both changes alter the design matrix. Reporting a design as orthogonal without saying how profiles were paired into choice sets leaves its efficiency unknown.

Sources

• Louviere JJ, Hensher DA, Swait JD. Stated Choice Methods: Analysis and Applications. Cambridge University Press; 2000.
• Huber J, Zwerina K. The importance of utility balance in efficient choice designs. Journal of Marketing Research. 1996;33(3):307-317.
• Reed Johnson F, Lancsar E, Marshall D, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health. 2013;16(1):3-13.
• Rose JM, Bliemer MCJ. Constructing efficient stated choice experimental designs. Transport Reviews. 2009;29(5):587-617.
• Kuhfeld WF. Marketing Research Methods in SAS: Experimental Design, Choice, Conjoint, and Graphical Techniques. SAS Institute; 2010.