Ordinal Scaling

Ordinal Scaling of Health Outcomes and Preference Data

An ordinal scale tells an analyst which of two patients or health states is better off and says nothing about by how much. Much of the raw material of health economics is ordinal in this sense, including EQ-5D response levels, clinical grades such as the modified Rankin Scale after stroke, and rankings of health states in valuation studies. Cost-utility analysis needs more than order, so ordered categories reach a QALY only after preference weights or a statistical model supply the distances. The four levels of measurement are compared under measurement scale type, and the level with equal differences under interval scaling.

Order-preserving transformations and the statistics they allow

Stevens, writing on scales of measurement in 1946, said that the ordinal scale arises from the operation of rank-ordering and that any order-preserving transformation leaves its form invariant. The codes attached to the categories can therefore be replaced by any other numbers in the same order without losing information.

$$x' = f(x), \quad f(x_1) > f(x_2) \text{ whenever } x_1 > x_2$$

where $x$ is the code given to a category, $x'$ is the code after relabelling and $f$ is any strictly increasing function. A statement about ordinal data is meaningful when its truth survives every such $f$. A comparison of median levels between two arms passes this test, because the case at the median keeps its position under every order-preserving change, while a difference in mean codes does not, since another spacing of the codes can shrink or reverse it.

Stevens listed the median and percentiles as the statistics suited to ordinal data. He wrote that, in the strictest propriety, means and standard deviations ought not to be used with such scales, while granting a pragmatic sanction because the practice often leads to fruitful results, and he warned that they are in error to the extent that the successive intervals are unequal in size.

Where ordinal data arise in health economic work

Outcome instruments record ordered categories, such as the five response levels of each EQ-5D-5L dimension or the 5-point Glasgow Outcome Scale after traumatic brain injury, which Roozenbeek and colleagues report is usually collapsed into unfavourable and favourable outcome in trials. Valuation studies ask respondents to rank health states or choose between them, as in a discrete choice experiment or best-worst scaling, so each answer records an order of preference.

The categories themselves decide what an ordinal scale can show. Devlin, Parkin and Janssen report that in an English NHS PROMs pilot, 95.7% of patients reported a level 2 problem on EQ-5D-3L mobility before hip surgery and none reported level 3, because level 3 means being confined to bed. The only mobility gain the instrument could record was a move to no problems, and the EQ-5D-5L set its most severe mobility level at being unable to walk about.

EQ-5D profiles and the limits of a partial order

Within one EQ-5D dimension the levels form a complete order, but across a five-dimension profile they do not. Devlin, Parkin and Janssen describe the rule that an EQ-5D state is better than another if it is better on at least one dimension and no worse on any other, and worse if the reverse holds. Two profiles that differ in opposite directions on different dimensions, such as 21111 and 11112, cannot be ranked from the responses alone. Ordering every profile requires a scoring rule, and the same authors list a value set, equally weighted dimensions and levels, or scores predicted from the EQ VAS.

The Paretian Classification of Health Change

Devlin, Parkin and Browne introduced the Paretian Classification of Health Change (PCHC) for EQ-5D data from the NHS PROMs pilot. It borrows the welfare economics idea of a Pareto improvement, where utility can be measured only in ordinal terms, and places each patient's change between two time points in one of four groups, better, worse, the same, or mixed when one dimension improves and another worsens.

For hip replacement, Devlin, Parkin and Janssen report that 82.0% of patients improved, 4.2% worsened, 4.7% had no change and 9.0% had a mixed change. Cataract removal looked very different, with 21.0% improving and 26.4% worsening, although the authors caution that the EQ-5D may not capture the benefits that cataract surgery provides. The classification needs no external scoring system or preference weights, and its limitations follow from that ordinal basis. A one-level gain counts the same as a four-level gain, the importance of each dimension to people is ignored, and the method is uninformative when mixed changes dominate.

Ordinal outcomes in trial analysis

Collapsing ordered categories into two groups is the simplest analysis and discards the most information. Devlin, Parkin and Janssen note that a shift between no problems and some problems misses improvements such as extreme to moderate problems, losing the advantage of the more refined EQ-5D-5L. The Optimising Analysis of Stroke Trials Collaboration, using 55 datasets from 47 trials with 54,173 patients, found that approaches using the ordered nature of functional outcomes were statistically more efficient than those collapsing the data into two groups. Its list of suitable approaches included ordinal logistic regression, the t test and a rank-based test, and the t test on category codes is an instance of the pragmatic sanction Stevens described.

The proportional odds model is the usual regression for an ordered outcome. Roozenbeek and colleagues describe it as having the structure of binary logistic regression with an ordinal outcome of more than two categories, estimating a common odds ratio over all possible cut-offs.

$$\ln\left(\frac{P(Y \le j \mid x)}{P(Y > j \mid x)}\right) = \alpha_j + \beta x, \quad j = 1, \ldots, K-1$$

where $Y$ is the outcome category numbered 1 (best) to $K$ (worst), $j$ is a cut-point between categories, $x$ is 1 for the treatment arm and 0 for the comparator, $\alpha_j$ is the log odds of being at level $j$ or better in the comparator arm, and $\beta$ is the log of the common odds ratio, which is $e^{\beta}$. The model is formally valid when the odds ratio is the same at every cut-off, the proportional odds assumption, and Roozenbeek and colleagues add that the common odds ratio can still summarise the treatment effect, read as the average shift over the whole scale, when the odds ratios differ.

In the CRASH trial of corticosteroids after traumatic brain injury, with 9,554 patients, they found that a dichotomous analysis of the six-month Glasgow Outcome Scale gave an odds ratio of 1.09 (95% CI 0.98 to 1.21, P = 0.096), while proportional odds regression gave 1.15 (95% CI 1.06 to 1.25, P = 0.0007), where an odds ratio above 1 meant a worse outcome with corticosteroids. Bath and colleagues, for the European Stroke Organisation Outcomes Working Group, concluded that a fixed dichotomy is rarely the most efficient analysis and that, with no best approach for every trial, design should consider whether a treatment tends to move all patients somewhat or some patients a lot.

Worked example with one EQ-5D-5L dimension in two trial arms

All numbers in this example are illustrative. A trial records the EQ-5D-5L pain or discomfort dimension at 12 weeks for 200 patients in each arm, with level 1 meaning no pain and level 5 extreme pain.

Step 1 (cumulative proportions). The share of patients at each level or better is 0.300, 0.600, 0.800 and 0.925 in the treatment arm (60, 120, 160 and 185 of 200) and 0.200, 0.450, 0.700 and 0.875 in the comparator arm (40, 90, 140 and 175 of 200). The treatment arm has the larger share at every cut-point.

Step 2 (odds ratio at each cut-point). At level 1 or better the odds are 0.30 / 0.70 = 0.4286 against 0.20 / 0.80 = 0.2500, an odds ratio of 1.714. The same steps give 1.500 against 0.8182 (1.833) at level 2 or better, 4.000 against 2.333 (1.714) at level 3 or better and 12.33 against 7.000 (1.762) at level 4 or better. A single dichotomy would report just one of these.

Step 3 (common odds ratio). The four odds ratios are close, so the proportional odds assumption is reasonable here. Fitting the model above to these counts by maximum likelihood gives $\beta = 0.568$ and a common odds ratio of $e^{0.568} = 1.76$ across all cut-points.

Step 4 (two sets of utility decrements that keep the order). Weights A give decrements of 0, 0.05, 0.12, 0.30 and 0.40 for levels 1 to 5. The treatment arm's total decrement is 60 × 0.05 + 40 × 0.12 + 25 × 0.30 + 15 × 0.40 = 21.3, a mean of 0.1065 over 200 patients, and the comparator's is 50 × 0.05 + 50 × 0.12 + 35 × 0.30 + 25 × 0.40 = 29.0, a mean of 0.1450, so the gain is 0.0385. Weights B give 0, 0.01, 0.02, 0.03 and 0.04. The totals are 2.75 and 3.55, means of 0.01375 and 0.01775, a gain of 0.0040.

Interpretation. Because the treatment arm's cumulative proportion is higher at every cut-point, any set of decrements that rises with severity gives the treatment arm the lower mean decrement, so the direction of the benefit is an ordinal conclusion. The size is not. Held for one year with the other dimensions unchanged, the gain is 0.0385 QALYs under weights A and 0.0040 under weights B, nearly ten times smaller. When the cumulative proportions cross, even the direction depends on the weights.

Turning ordered categories into QALY weights

A QALY multiplies time by a utility and adds the results, which needs values whose differences carry meaning, the property set out under cardinal scaling. Ordinal responses reach a cost-utility analysis through added preference information, most often a value set that assigns a utility to every EQ-5D profile. Devlin, Parkin and Janssen note that summarising profiles by values brings possible information loss and bias, and treat profile analysis as the starting point even when QALYs are the goal.

Ordinal preference data can also yield cardinal values through a statistical model. Salomon showed that a conditional logit model fitted to rankings of EQ-5D states yields interval-scale values, whose anchors such as full health and dead still have to be chosen.

A third route predicts ordinal responses from another instrument. Gray, Rivero-Arias and Clarke used multinomial logit regression to predict each EQ-5D response level from SF-12 answers and then attached utility scores, a form of mapping function.

Choices that change ordinal results

Ordinal methods avoid assumptions about spacing. The analyst still makes choices that move the results.

1. Category definitions. The number and wording of levels fix which changes can be recorded.
2. Cut-point for a dichotomy. Each cut-point gives its own odds ratio.
3. The proportional odds assumption. When effects differ by cut-point, the common odds ratio is a summary measure and no longer a single shared effect.
4. The rule for ranking whole profiles. Different scoring rules can order the same profiles differently.

Ordinal scaling beside neighbouring scales and methods

A nominal scale records only which category a case falls in, so a trial arm is nominal while an EQ-5D level is ordinal, and a ratio scale adds equal differences and a true zero. Cardinal scaling, the contrasting property, requires differences in health state values to reflect strength of preference. The ranking method of health state valuation is a technique, and ordinal scaling is the property of the data it produces.

Misreadings of ordinal health data

Most errors with ordinal data come from reading the category codes as quantities. The four below recur in trial reports and economic models.

1. Level codes measure severity in equal steps. The first step up the scale need not equal the last, and a value set estimates a separate decrement for each step.
2. A level sum score orders profiles. Devlin, Parkin and Janssen show that the values of profiles sharing a level sum score spread widely and overlap with those of other scores, and conclude that it is wrong to treat the score as ordinal.
3. A common odds ratio is a health gain. It summarises a shift in category shares and carries no QALY meaning without utilities.
4. An improvement in the PCHC is a large benefit. The classification records direction only, so a one-level gain and a four-level gain count alike.

Sources

• Bath PM, Lees KR, Schellinger PD, Altman H, Bland M, Hogg C, Howard G, Saver JL; European Stroke Organisation Outcomes Working Group. Statistical analysis of the primary outcome in acute stroke trials. Stroke. 2012;43(4):1171-1178. https://doi.org/10.1161/STROKEAHA.111.641456
• Devlin N, Parkin D, Janssen B. Methods for Analysing and Reporting EQ-5D Data. Cham: Springer; 2020. Chapter 2, Analysis of EQ-5D profiles. https://doi.org/10.1007/978-3-030-47622-9_2
• Devlin NJ, Parkin D, Browne J. Patient-reported outcome measures in the NHS: new methods for analysing and reporting EQ-5D data. Health Economics. 2010;19(8):886-905. https://doi.org/10.1002/hec.1608
• Gray AM, Rivero-Arias O, Clarke PM. Estimating the association between SF-12 responses and EQ-5D utility values by response mapping. Medical Decision Making. 2006;26(1):18-29. https://doi.org/10.1177/0272989X05284108
• Optimising Analysis of Stroke Trials (OAST) Collaboration; Bath PM, Gray LJ, Collier T, Pocock S, Carpenter J. Can we improve the statistical analysis of stroke trials? Statistical reanalysis of functional outcomes in stroke trials. Stroke. 2007;38(6):1911-1915. https://doi.org/10.1161/STROKEAHA.106.474080
• Roozenbeek B, Lingsma HF, Perel P, Edwards P, Roberts I, Murray GD, Maas AIR, Steyerberg EW. The added value of ordinal analysis in clinical trials: an example in traumatic brain injury. Critical Care. 2011;15(3):R127. https://doi.org/10.1186/cc10240
• Salomon JA. Reconsidering the use of rankings in the valuation of health states: a model for estimating cardinal values from ordinal data. Population Health Metrics. 2003;1:12. https://doi.org/10.1186/1478-7954-1-12
• Stevens SS. On the theory of scales of measurement. Science. 1946;103(2684):677-680. https://doi.org/10.1126/science.103.2684.677