Discriminant Validity

Discriminant Validity in Health Measurement and Departures from the EQ-5D

Validating a health measure means showing what its scores leave out as well as what they capture, and a related term is used for whether a measure separates groups that should differ in health. Both matter in health economics because utility values from instruments such as the EQ-5D, SF-6D and HUI3 become the quality weights in a QALY, and a measure that blurs distinct aspects of health, or cannot tell milder from more severe patients apart, distorts QALY gains. The multitrait-multimethod matrix, COSMIN and FDA guidance treat the two meanings differently, and known-groups results bear on NICE's test for departing from the EQ-5D. An illustrative example reads a small correlation matrix and shows how a weak known-groups result changes an incremental cost-effectiveness ratio.

Two meanings of discriminant evidence in health measurement

The term is used in two senses that are easy to confuse, so a validation report should say which one it tests. In the first sense, which comes from psychology, discriminant validity means that scores correlate less with measures of different constructs than with measures of the same construct, so a mood subscale should relate more strongly to another mood measure than to a physical functioning scale. In the second sense, usually called known-groups or discriminative validity, scores should differ between groups expected to differ in health, such as patients with mild and moderate disease. The first compares measures, the second compares people.

The multitrait-multimethod origin

The first meaning comes from the multitrait-multimethod matrix that Donald Campbell and Donald Fiske proposed in 1959, whose convergent half is covered on the convergent validity page. The discriminant half concerns the cells that pair different traits.

In David Kenny's summary of the Campbell and Fiske approach, discriminant validity requires that different-trait, different-method correlations are not too high, especially relative to the same-trait, different-method correlations of the validity diagonal, and method variance shows up when different-trait, same-method correlations exceed different-trait, different-method ones. Kenny notes that the approach has no standard for what counts as a good result. A high correlation between a pain scale and a mood scale on one self-completed form can therefore reflect the shared method rather than overlapping constructs.

Where COSMIN and the FDA file discriminant evidence

Two frameworks for patient-reported outcome measures (PROMs) treat this evidence as part of construct validity, tested through hypotheses set in advance, but they label it differently. The COSMIN user manual for systematic reviews does not use the term discriminant validity at all.

The 2018 COSMIN manual splits its hypotheses-testing box into part 9a, comparison with other outcome measurement instruments, labelled convergent validity, and part 9b, comparison between subgroups, labelled discriminative or known-groups validity. Correlations with instruments measuring unrelated constructs, the evidence psychology calls discriminant, are rated under part 9a. In Box 9b, COSMIN's "discriminative" therefore points to the known-groups meaning.

The US Food and Drug Administration's 2009 guidance on patient-reported outcome measures keeps the meanings apart. Its table of measurement properties lists the strength of correlation testing prior hypotheses as discriminant and convergent validity, and the degree to which an instrument distinguishes groups hypothesised to differ as known-groups validity.

Writing discriminant hypotheses for utility instruments

A discriminant hypothesis states which correlation should be low, how low, and how much lower than a convergent comparison, with direction and magnitude fixed in advance as the COSMIN manual requires. The generic correlation bands that the manual reproduces from de Vet and colleagues are tabulated on the convergent validity page. For preference-based measures they need care, because the EQ-5D index is a preference-weighted aggregate across five dimensions (mobility, self-care, usual activities, pain or discomfort, and anxiety or depression), so it is expected to correlate with both physical and mental health scales.

The NICE Decision Support Unit's Technical Support Document 8 (TSD 8) states that construct validation is best performed at the dimension level, although evidence is often only available for the index. At that level a discriminant hypothesis is easy to state. Once score directions are aligned, the EQ-5D anxiety or depression dimension should correlate more strongly with a mental health scale than with a physical functioning scale, and the mobility dimension the other way round.

Known-groups discrimination and NICE's test for the EQ-5D

Known-groups evidence bears more directly on economic evaluation than correlational evidence, because it shows whether utility values differ between the health states of a model. The COSMIN standards for part 9b state that whether a difference is as large as expected matters more than whether it is statistically significant, which depends on sample size.

NICE's manual (PMG36, section 4.3) prefers the EQ-5D for adults, and paragraph 4.3.10 requires a case against it to rest on qualitative evidence of missing content, supported by poor performance on tests of construct validity and responsiveness drawn from a synthesis of peer-reviewed literature. Paragraph 4.3.11 allows evidence other than psychometric measures when evidence generation is difficult, for example in rare diseases.

TSD 8 asks whether the EQ-5D fails to reflect known differences between groups, preferably across the relevant dimensions. It warns that groups defined by clinical measures such as visual acuity or respiratory function may have only a weak relationship to health-related quality of life, and adds that the EQ-5D failing to reflect differences between groups found by the HUI3 or SF-6D may suggest it is not valid in that group.

Latent-variable criteria and the HTMT ratio

When an instrument has several multi-item subscales, discriminant evidence can come from item correlations or a latent-variable model. The COSMIN manual accepts confirmatory factor analysis or structural equation modelling over several scales to check whether subscales measuring similar constructs are more related than those measuring different ones.

Henseler, Ringle and Sarstedt showed by simulation that the two dominant criteria in variance-based structural equation modelling do not reliably detect a lack of discriminant validity in common research situations, and proposed the heterotrait-monotrait ratio (HTMT).

$$HTMT_{ij} = \frac{\bar{r}_{ij}}{\sqrt{\bar{r}_{ii} \times \bar{r}_{jj}}}$$

where $HTMT_{ij}$ is the ratio for constructs $i$ and $j$, $\bar{r}_{ij}$ is the average correlation between their indicators, and $\bar{r}_{ii}$ and $\bar{r}_{jj}$ are the average correlations among the indicators within each construct. The HTMT estimates the correlation between the constructs; the authors cite thresholds of 0.85 and 0.90, above which discriminant validity is lacking, and call the exact level debatable.

Worked example with a condition-specific questionnaire and the EQ-5D in 200 patients

The figures below are illustrative and do not come from any study. An analyst validating a new condition-specific questionnaire in 200 adults with a long-term neurological condition measures physical functioning (PF) and mood (MD) by two methods, the patient questionnaire (S) and a clinician rating (C), with all scores coded so that higher means better health. The EQ-5D-3L is collected alongside.

Reading the multitrait-multimethod table

The analyst prespecified that same-trait correlations would be 0.50 or higher and different-trait, different-method correlations below 0.30. Physical functioning and mood would count as related but dissimilar under the generic hypotheses, but the analyst expected separate methods to remove the shared reporting component and push their correlation below that band. The lower triangle of the correlation matrix is shown below.

1. Validity diagonal. The same-trait correlations are 0.64 for physical functioning and 0.52 for mood, both meeting the 0.50 hypothesis.

2. Discriminant check across methods. The different-trait, different-method correlations are 0.18 and 0.21, both below 0.30, and the smallest gap to a validity value, 0.52 − 0.21 = 0.31, exceeds 0.10.

3. Method variance. The different-trait, same-method correlations are 0.45 for the questionnaire and 0.30 for the clinician ratings. The questionnaire value is more than twice the different-method values, pointing to shared method variance in the self-reports, and the mood validity value exceeds it by only 0.52 − 0.45 = 0.07, less than the 0.10 gap that the generic hypotheses set between categories, applied here by analogy.

4. HTMT. Treating the two methods as two indicators of each trait, the between-trait correlations 0.45, 0.18, 0.21 and 0.30 average 1.14 / 4 = 0.285, and the within-trait correlations are 0.64 and 0.52.

$$HTMT_{PF,MD} = \frac{0.285}{\sqrt{0.64 \times 0.52}} = \frac{0.285}{0.5769} = 0.494$$

where 0.285 is the average between-trait correlation and 0.5769 is the geometric mean of the two within-trait correlations. A value of 0.49 is far below 0.85.

5. Interpretation. The summary ratio passes easily, yet the questionnaire's two subscales overlap more than the traits do, a problem confined to one method. If both subscales later enter a mapping function to EQ-5D as separate predictors, part of what each coefficient picks up is a shared reporting style.

Known-groups comparison and its effect on QALYs

Before the questionnaire is scored, the clinician rating of physical functioning is split at a prespecified cut-off into 100 patients with mild and 100 with moderate impairment. The analyst hypothesises that the questionnaire physical score (0 to 100) and the EQ-5D-3L index will both be lower in the moderate group, with a standardised difference of at least 0.50.

1. Pooled standard deviation. The spread within groups is combined as follows.

$$s_p = \sqrt{\frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2}}$$

where $s_p$ is the pooled standard deviation, $n_1$ and $n_2$ are the group sizes (100 each), and $s_1$ and $s_2$ are the group standard deviations. The EQ-5D result is shown first.

$$s_p = \sqrt{\frac{99 \times 0.0400 + 99 \times 0.0484}{198}} = \sqrt{0.0442} = 0.2102$$

where 0.0400 and 0.0484 are the squared standard deviations. For the questionnaire the same steps give $\sqrt{(99 \times 256 + 99 \times 324) / 198} = \sqrt{290} = 17.03$.

2. Standardised difference. The separation is expressed as Cohen's d.

$$d = \frac{\bar{x}_1 - \bar{x}_2}{s_p}$$

where $d$ is the standardised difference and $\bar{x}_1$ and $\bar{x}_2$ are the mild and moderate group means. The formula is applied to both measures below.

$$d_{Q} = \frac{68 - 52}{17.03} = 0.94 \qquad d_{EQ} = \frac{0.71 - 0.66}{0.2102} = 0.24$$

where $d_{Q}$ and $d_{EQ}$ are the standardised differences for the questionnaire and the EQ-5D. The hypothesis is confirmed for the questionnaire and not for the EQ-5D.

3. Effect on a model. Suppose a Markov model uses the EQ-5D means as utilities for mild and moderate states, and a treatment keeps patients in the mild state for 2 extra years at an extra cost of £3,000. Ignoring discounting, the results are as follows.

$$\Delta Q = 2 \times (0.71 - 0.66) = 0.10 \qquad ICER = \frac{3000}{0.10} = 30000$$

where $\Delta Q$ is the QALY gain and $ICER$ is the incremental cost-effectiveness ratio in pounds per QALY. If a preference-based measure valid in this group, such as the HUI3, showed a difference of 0.12 between the states, the gain would be 2 × 0.12 = 0.24 QALYs and the ratio 3,000 / 0.24 = £12,500 per QALY.

4. Interpretation. If the small EQ-5D difference reflects insensitivity, the ratio is overstated by a factor of 30,000 / 12,500 = 2.4; if the clinical grouping has little effect on health-related quality of life, the alternative TSD 8 warns about, the ratio is right. The questionnaire separates the groups partly because its physical score shares a trait with the grouping variable, so the next step is to examine the EQ-5D mobility and usual activities dimensions. One study cannot meet the PMG36 test, but it can feed the literature synthesis the test requires.

Boundaries with convergent, criterion and responsiveness evidence

Discriminant and convergent evidence are most informative as a pair, since a low correlation means little unless a correlation with a similar measure is clearly higher. Criterion validity needs a gold standard, which COSMIN holds a PROM in principle lacks. Responsiveness concerns change scores, which the COSMIN manual treats as the only difference from cross-sectional validity, so known-groups comparisons of change belong to responsiveness.

Common misreadings of discriminant results

Most errors come from reading a correlation or group difference without the comparison it should be judged against. Four recur in health economic work.

A low correlation is weak evidence on its own, since measurement error or a narrow spread of health depresses every correlation, convergent ones included. A statistically significant group difference can be too small to matter for utilities, and a non-significant one can reflect a small sample. Groups defined with the instrument's own items, or with a measure close to the comparator, make known-groups tests circular. In evidence reviews, COSMIN's discriminative or known-groups validity and the multitrait-multimethod sense of discriminant validity are different results, since the first compares people and the second compares measures.

Sources

• Brazier J, Longworth L. NICE DSU Technical Support Document 8: An introduction to the measurement and valuation of health for NICE submissions. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2011. Section 3.3 and Table 1. https://doi.org/10.15131/shef.data.33738301
• Campbell DT, Fiske DW. Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin. 1959;56(2):81-105. https://doi.org/10.1037/h0046016
• Henseler J, Ringle CM, Sarstedt M. A new criterion for assessing discriminant validity in variance-based structural equation modeling. Journal of the Academy of Marketing Science. 2015;43(1):115-135. https://doi.org/10.1007/s11747-014-0403-8
• Kenny DA. Multitrait-multimethod matrix. Structural equation modeling notes; revised 2012. https://davidakenny.net/cm/mtmm.htm
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0. Amsterdam: COSMIN; February 2018. Sections 3.4, 5.7 and 5.8, Box 9 and Table 8. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Section 4.3. https://www.nice.org.uk/process/pmg36
• US Food and Drug Administration. Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Silver Spring, MD: FDA; December 2009. Table 2 and section III.E.2. https://www.fda.gov/media/77832/download