Best-Worst Scaling

Best-Worst Scaling in Health Preference Research

Best-worst scaling builds a scale of preference or priority from a planned series of small tasks, each answered with two extreme choices. In a set of four, those two choices fix five of the six possible paired comparisons, so the method gathers ordering information quickly and avoids the scale-use problems of ratings. In health economics it is used to rank what patients and the public prioritise and to value the levels of capability and quality of life instruments used in economic evaluation. The method has three cases that differ in what respondents compare, is usually built on balanced incomplete block designs, and is analysed by best-minus-worst counts or by the maxdiff and sequential models, which a worked example with seven illustrative service priorities compares.

Origins and place among stated preference methods

Best-worst scaling is a stated preference method grounded in random utility theory, the theory behind the discrete choice experiment. Finn and Louviere devised it in a 1992 study of public concern about food safety, and Louviere, Flynn and Marley describe it as an extension of paired comparison to choices among more than two options. The first health application called it maximum difference conjoint analysis, and McIntosh and Louviere introduced it formally to health care research.

The gain over pairs is efficiency. Ranking seven objects through pairs needs 21 tasks. In a set of four, the best choice places one option above three others and the worst choice places one below three others, so only the middle pair stays unknown.

The three cases of objects, profiles and multiple profiles

Flynn separates three types of best-worst scaling, known as cases, and the chapter on best-worst scaling theory and methods in the 2024 edition of the Handbook of Choice Modelling, by Marley, describes the same three. They differ in what the respondent compares, which decides what the results can support.

Flynn, Louviere, Peters and Coast set out why best-worst tasks help health research. Comparing utility parameters from a conventional discrete choice experiment cannot show whether one attribute, such as quality of care, is valued more than another, such as waiting time, while a best-worst task can. That comparison holds for the levels shown in the task, so in profile and multi-profile tasks the measured importance of an attribute also depends on the range of levels the researcher chose, the confounding that Hollin and colleagues describe and that the object case avoids. Cheung and colleagues found 62 health studies published up to April 2016 and classed 26 as object case, 29 as profile case and 7 as multi-profile case. Hollin and colleagues later found 165 object case studies of health priorities up to the end of 2021.

Building choice sets with balanced incomplete block designs

The object case is usually built from a balanced incomplete block design (BIBD). Every choice set has the same size, every object appears the same number of times, and every pair appears together the same number of times, so no object gains from the way sets were drawn. Two counting identities must hold, as follows.

$$b \, k = v \, r \qquad \lambda (v - 1) = r (k - 1)$$

where $v$ is the number of objects, $b$ the number of sets, $k$ the objects per set, $r$ the appearances of each object, and $\lambda$ the number of times each pair appears together. The first counts object appearances two ways; the second counts one object's pairings two ways.

Cheung and colleagues found the BIBD was the most common design for object case studies (54%), while an orthogonal design was most common for profile case (66%) and multi-profile case (43%) studies. Hollin and colleagues report that orthogonal designs are now seen as inappropriate for the object case because they give sets of unequal size. The ICECAP-A valuation used an orthogonal main effects plan of 16 profiles, plus its foldover, from 1,024 possible states, which estimates main effects but not interactions.

From choices to scores with counts, maxdiff and sequential models

Analysis ranges from simple counts to formal choice models. The simplest approach counts how often each item was chosen best and worst. The best-minus-worst score is the difference, and dividing by the opportunities to be chosen puts items on a common footing.

$$S_i = \frac{B_i - W_i}{N \, r_i}$$

where $S_i$ is the standardised score of item $i$, $B_i$ and $W_i$ are its best and worst counts across the sample, $N$ is the number of respondents, and $r_i$ is the number of times the item appears in each respondent's design. Scores run from $-1$, always worst, to $+1$, always best. Some studies also report the square root of best over worst counts, which is undefined for an item never chosen worst.

Choice models place items on a latent utility scale. The maxdiff model assumes the respondent picks, from all ordered pairs of distinct items, the pair with the largest utility difference, giving the probability below.

$$P(b, w \mid C) = \frac{\exp(V_b - V_w)}{\sum_{j \in C} \sum_{k \in C, \, k \neq j} \exp(V_j - V_k)}$$

where $P(b, w \mid C)$ is the probability that item $b$ is chosen best and item $w$ worst from set $C$, and $V_b$, $V_w$, $V_j$ and $V_k$ are systematic utilities. The sequential model treats the worst choice as made from the items left once the best is removed, as follows.

$$P(b, w \mid C) = \frac{\exp(V_b)}{\sum_{j \in C} \exp(V_j)} \times \frac{\exp(-V_w)}{\sum_{k \in C_{-b}} \exp(-V_k)}$$

where $C_{-b}$ is the set without item $b$ and other terms are as above. In practice the worst data are often appended to the best data with every explanatory variable multiplied by minus one, then estimated by conditional logit, mixed logit, latent class or hierarchical Bayes methods. Cheung and colleagues found object case studies most often used best-minus-worst summary statistics (32%) or hierarchical Bayes (21%), and profile and multi-profile studies most often multinomial logit. Hollin and colleagues found most object case studies did not state their choice model, and call the label MaxDiff a potential misnomer because it implies the maxdiff assumption.

Worked example of seven service priorities in a balanced design

All objects, responses and utilities here are illustrative. An outpatient service asks patients to prioritise seven changes, labelled A to G. A is a shorter wait for a first appointment, B the same clinician each visit, C a clinic nearer home, D clear written information, E evening appointments, F online booking and G free parking. The design is the seven-set BIBD used as an example by Hollin and colleagues.

1. Check the design. With $v = 7$, $b = 7$, $k = 4$, $r = 4$ and $\lambda = 2$, both identities hold.

$$7 \times 4 = 7 \times 4 = 28 \qquad 2 \times (7 - 1) = 4 \times (4 - 1) = 12$$

where 28 is the number of object appearances and 12 the pairings of any one object across the design.

2. Record the choices. A hypothetical respondent's true order is A, B, C, D, E, F, G, answered without error.

3. Count and standardise. With $N = 1$ and $r = 4$, each score is the best-minus-worst count divided by 4.

Best and worst counts each sum to 7, one per set.

4. Interpret the counts. The extremes are clear, but C, third in the true order, scores below D, because every set with C also held A or B, while D was the strongest object in set 4. One respondent's counts can therefore misorder middle items, as they reverse C and D here; a sample, a model, or both are needed to separate them.

5. Compare the two models for set 1. Take illustrative utilities $V_A = 1.0$, $V_C = 0.5$, $V_E = 0$ and $V_G = -0.5$. The twelve ordered pairs give three differences of 0.5, three of $-0.5$, two of 1.0, two of $-1.0$ and one each of 1.5 and $-1.5$, which combine as follows.

$$D = 3 \times 1.649 + 3 \times 0.607 + 2 \times 2.718 + 2 \times 0.368 + 4.482 + 0.223 = 17.645$$

where $D$ is the maxdiff denominator, each exponential rounded to three decimals. The maxdiff probability of A best and G worst is $P_{\max} = 4.482 / 17.645 = 0.254$. Under the sequential model, A is best with probability $P_A = 2.718 / 5.974 = 0.455$, where 5.974 sums $\exp(V)$ over the four objects, and G is then worst among C, E and G with probability $P_G = 1.649 / 3.256 = 0.506$, where 3.256 sums $\exp(-V)$ over those three. The joint probability is $P_{\text{seq}} = 0.455 \times 0.506 = 0.230$. The models disagree from the same utilities, so a study should state which it assumed.

Valuing capability and health states

The profile case has produced tariffs for outcome measures used in economic evaluation. Coast and colleagues describe the ICECAP index for older people as the first quality of life measure for economic evaluation valued by best-worst scaling, using a general population sample aged 65 and over, with values anchored at one for full capability and zero for no capability. ICECAP-A was valued in a UK sample of 413 adults, using scale-adjusted latent class models to separate preferences from choice consistency. The Carer Experience Scale was valued by 162 unpaid carers of older people.

For EQ-5D instruments the evidence on health state valuation is mixed. Profile case tasks were used to collect EQ-5D-Y preferences from adolescents and adults in Australia and Spain, and the authors concluded that adolescents' best-worst preferences were valid. For EQ-5D-5L states, one comparison found a binary choice experiment more feasible and reliable than the multi-profile case.

Evidence on health utility indices in general is also cautious. Comparing best-worst scaling with a discrete choice experiment for valuing the dimensions of such indices, Krucien, Watson and Ryan found low concordance between the methods and lower quality best-worst data, and advised caution about using indices based on best-worst data. Best-worst values are relative, so they are not on the QALY scale unless anchored; the ICECAP-A tariff paper notes two routes, rescaling with time trade-off tasks or including length of life as an attribute.

Good practice standards and reporting

ISPOR good practice reports were written for conjoint analysis in general or for discrete choice experiments. The 2011 checklist of the ISPOR Good Research Practices for Conjoint Analysis Task Force covers conjoint analysis, including discrete choice experiments and other multiattribute stated preference methods. Later ISPOR task force reports on experimental design (2013) and statistical analysis (2016) address discrete choice experiments, so they carry over most directly to the multi-profile case.

Reviews show where practice falls short. Cheung and colleagues found that 90% of studies did not report whether responders resembled non-responders. Hollin and colleagues found average samples rising from 228 to 472 respondents, formal sample size justification in only 5.3% of studies, and no reported experimental design in 20%.

Best-worst scaling compared with choice experiments, ranking and rating

The object case attaches no levels to items, so its results show priority but not trade-offs, and cannot give a marginal rate of substitution or willingness to pay. The matching advantage, noted above, is that object importance is not confounded with the range of levels the researcher chose. The multi-profile case is closest to conjoint analysis in the choice experiment tradition, and Mühlbacher and colleagues regard it as a refinement of the conventional discrete choice experiment. Against a full ranking, best-worst tasks ask only for extremes; against ratings, they force a choice, so respondents cannot rate every item as highly important.

Limitations and common misreadings

Best-minus-worst scores cannot be read as ratios. They can be negative, and because every set yields one best and one worst choice they sum to zero across items, as in the worked example, so a statement that one item is twice as important as another has no meaning on that scale. The square root of best over worst counts is sometimes reported as a ratio-type measure, but ratio-scale properties come from a choice model, such as the maxdiff model, and the measure is undefined for items never chosen worst, as for A, B, C and D in the worked example. Worst choices may follow a different mental process from best choices, and comparisons with choice experiments have given mixed results, so Cheung and colleagues advise care in directly comparing results from the two methods. Differences in choice consistency can look like differences in preference unless scale is modelled. Profile case designs cannot estimate interactions between attributes. As the worked example shows, middle-ranked items get less information than extreme ones. Results also depend on the item list, and an omitted object cannot be ranked.

Sources

• Al-Janabi H, Flynn TN, Coast J. Estimation of a preference-based carer experience scale. Medical Decision Making. 2011;31(3):458-468.
• Bridges JFP, Hauber AB, Marshall D, et al. Conjoint analysis applications in health: a checklist. A report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value in Health. 2011;14(4):403-413.
• Cheung KL, Wijnen BFM, Hollin IL, et al. Using best-worst scaling to investigate preferences in health care. PharmacoEconomics. 2016;34(12):1195-1209.
• Coast J, Flynn TN, Natarajan L, et al. Valuing the ICECAP capability index for older people. Social Science and Medicine. 2008;67(5):874-882.
• Dalziel K, Catchpool M, García-Lorenzo B, et al. Feasibility, validity and differences in adolescent and adult EQ-5D-Y health state valuation in Australia and Spain: an application of best-worst scaling. PharmacoEconomics. 2020;38(5):499-513.
• Finn A, Louviere JJ. Determining the appropriate response to evidence of public concern: the case of food safety. Journal of Public Policy and Marketing. 1992;11(2):12-25.
• Flynn TN, Louviere JJ, Peters TJ, Coast J. Best-worst scaling: what it can do for health care research and how to do it. Journal of Health Economics. 2007;26(1):171-189.
• Flynn TN. Valuing citizen and patient preferences in health: recent developments in three types of best-worst scaling. Expert Review of Pharmacoeconomics and Outcomes Research. 2010;10(3):259-267.
• Flynn TN, Marley AAJ. Best-worst scaling: theory and methods. In: Hess S, Daly A, eds. Handbook of Choice Modelling. Cheltenham: Edward Elgar; 2014. p. 178-201.
• Flynn TN, Huynh E, Peters TJ, et al. Scoring the ICECAP-A capability instrument. Estimation of a UK general population tariff. Health Economics. 2015;24(3):258-269.
• Hauber AB, González JM, Groothuis-Oudshoorn CGM, et al. Statistical methods for the analysis of discrete choice experiments: a report of the ISPOR Conjoint Analysis Good Research Practices Task Force. Value in Health. 2016;19(4):300-315.
• Hollin IL, Paskett J, Schuster ALR, Crossnohere NL, Bridges JFP. Best-worst scaling and the prioritization of objects in health: a systematic review. PharmacoEconomics. 2022;40(9):883-899.
• Johnson FR, Lancsar E, Marshall D, et al. Constructing experimental designs for discrete-choice experiments: report of the ISPOR Conjoint Analysis Experimental Design Good Research Practices Task Force. Value in Health. 2013;16(1):3-13.
• Krucien N, Watson V, Ryan M. Is best-worst scaling suitable for health state valuation? A comparison with discrete choice experiments. Health Economics. 2017;26(12):e1-e16.
• Louviere JJ, Flynn TN, Marley AAJ. Best-Worst Scaling: Theory, Methods and Applications. Cambridge: Cambridge University Press; 2015.
• Marley AAJ. Best-worst scaling: theory and methods. In: Hess S, Daly A, eds. Handbook of Choice Modelling. 2nd ed. Cheltenham: Edward Elgar; 2024. p. 206-245.
• Marley AAJ, Louviere JJ. Some probabilistic models of best, worst, and best-worst choices. Journal of Mathematical Psychology. 2005;49(6):464-480.
• Mühlbacher AC, Kaczynski A, Zweifel P, Johnson FR. Experimental measurement of preferences in health and healthcare using best-worst scaling: an overview. Health Economics Review. 2016;6:2.
• Mühlbacher AC, Zweifel P, Kaczynski A, Johnson FR. Experimental measurement of preferences in health care using best-worst scaling (BWS): theoretical and statistical issues. Health Economics Review. 2016;6:5.
• Xie F, Pullenayegum E, Gaebel K, et al. Eliciting preferences to the EQ-5D-5L health states: discrete choice experiment or multiprofile case of best-worst scaling? European Journal of Health Economics. 2014;15(3):281-288.