Construct Validity

Construct Validity in Testing Health Measures by Hypothesis When No Gold Standard Exists

Health economists work with quantities that nobody can observe directly. Health-related quality of life, and the utility values that turn it into QALYs, are known only through questionnaires and valuations, so the case that an instrument captures them has to be built indirectly, by checking that its scores behave as theory predicts. The idea goes back to Cronbach and Meehl's 1955 paper, and COSMIN now divides construct validity into structural validity, hypotheses testing and cross-cultural validity, with hypotheses scored and summarised by its 75% rule. Construct validity also enters NICE's test for departing from the EQ-5D, and an illustrative study of eight hypotheses is worked through below. The pages on convergent validity, discriminant validity and criterion validity cover those tests in detail.

From Cronbach and Meehl to the modern view of validity

The term came from an American Psychological Association committee that, in the early 1950s, distinguished four types of validity, namely predictive, concurrent, content and construct. In 1955 Lee Cronbach and Paul Meehl explained it. Construct validation, they wrote, is involved whenever a test is interpreted as a measure of an attribute that is not operationally defined, and it must be investigated whenever no criterion or universe of content is accepted as entirely adequate to define the quality to be measured.

Their account rests on a nomological network, the interlocking system of laws that makes up a theory and links constructs to one another and to observable quantities. A proposed interpretation of a score generates testable hypotheses, and evidence can come from group differences, correlation matrices and factor analysis, internal structure and change over occasions. They warned that rejecting a null hypothesis does not finish the job, since the task is to state as definitely as possible the degree of validity a test is presumed to have.

The joint Standards for Educational and Psychological Testing of 2014 call validity a unitary concept, refer to types of validity evidence rather than distinct types of validity, and state that interpretations of test scores for proposed uses are evaluated, not the test itself. Health measurement, following COSMIN, still names construct validity as a separate measurement property, and that convention is followed here.

Why construct validity carries the weight in health measurement

Criterion validity needs a reference measurement of the same quantity, and for health-related quality of life there is rarely one. The COSMIN user manual for systematic reviews states that, in principle, a patient-reported outcome measure (PROM) has no gold standard, the long version of a shortened instrument being the only exception, and that in that case all validity evidence counts as evidence for construct validity.

The NICE Decision Support Unit's Technical Support Document 8 (TSD 8) makes the same point for economic evaluation. It records that the lack of a gold standard in health measurement and valuation has left some health economists sceptical about trying to prove validity, and that psychometric tests can be adapted with care for the EQ-5D. Such tests can never prove validity, it adds, but they provide evidence on the degree to which an instrument validly measures the concept tested.

COSMIN's three parts of construct validity

The COSMIN taxonomy of measurement properties for health-related patient-reported outcomes, agreed by international Delphi consensus and published by Mokkink and colleagues in 2010, places construct validity in the validity domain alongside content and criterion validity. It splits construct validity into three aspects, treated as measurement properties in the 2018 manual.

Measurement invariance means that respondents from different groups with the same level of the latent trait respond similarly to an item. The manual also sets an order of evaluation, with content validity first, as the most important property, then internal structure, then the remaining properties, hypotheses testing among them. A PROM with high-quality evidence of inadequate content validity can be excluded from a review before its construct validity is examined.

Hypotheses testing and COSMIN's 75% rule

Hypotheses testing covers the two tests TSD 8 names for utility instruments, known-group differences and convergent validity. The COSMIN manual calls it an ongoing, iterative process in which more specific hypotheses, and more of them, gather more evidence. Each hypothesis states the expected direction and magnitude, absolute or relative, of a correlation or difference, because otherwise it is hard to decide afterwards whether a result agrees.

The manual's examples include a correlation of at least 0.50 with an instrument measuring the same construct, and a correlation with one instrument at least 0.10 higher than with another; its generic bands are given under convergent validity. Cronbach and Meehl had set out the known-groups logic. Failing to find an expected group difference would seriously challenge a test, while too close a match with group membership would suggest partial invalidity, since group members are expected to overlap. COSMIN adds that p-values should not be used, and that a group difference is judged on whether it is as large as expected rather than on statistical significance, which depends on sample size.

Each result is rated sufficient (+) when it agrees with its hypothesis and insufficient (−) when it does not, and the results of all studies are taken together as follows.

$$P = \frac{n_{+}}{n_{+} + n_{-}}$$

where $P$ is the proportion of results in accordance with the hypotheses, $n_{+}$ is the number in accordance and $n_{-}$ the number not in accordance. The summary is sufficient when at least 75% of results are in accordance, insufficient when at least 75% are not, and inconsistent (±) otherwise. The rule was written for reviews that summarise several studies, so applying it to one study is an analogy rather than a COSMIN rating.

Structural validity and the dimensions of a score

Structural validity asks whether items group into the dimensions an instrument claims, which is what justifies summing them into a total score or subscale scores. The COSMIN manual notes that it is usually assessed by factor analysis, with confirmatory factor analysis testing a prespecified structure against the fit criteria above, while exploratory factor analysis can propose one. Across studies, a sufficient rating needs at least 75% of confirmatory studies to find the same factor structure.

It applies only to reflective models, in which every item manifests one underlying construct; in a formative model the items together form the construct, need not correlate, and give factor analyses that cannot be interpreted. Cronbach and Meehl warned that high internal consistency supports construct validity only when the theory of the trait calls for high item intercorrelations.

Construct validity of preference-based measures in NICE appraisals

In health technology assessment the practical question is whether a preference-based measure captures what matters to a patient group, because its values become health state utility values and so QALY weights. Under paragraph 4.3.10 of NICE's manual (PMG36), a case that the EQ-5D is inappropriate needs qualitative empirical evidence of its lack of content validity, supported by evidence that it performs poorly on tests of construct validity, meaning it does not perform as would be expected, and responsiveness in a particular patient population, drawn from a synthesis of peer-reviewed literature.

TSD 8 prefers validation at dimension level, and it notes that the appropriateness of the EQ-5D concerns its descriptive system rather than its values, though most evidence is concerned with the index. For populating cost-effectiveness models, it adds, responsiveness is less important than validity because models require mean health state values.

Worked example of eight hypotheses for the EQ-5D in rheumatoid arthritis

The figures below are illustrative and do not come from any study. An analyst testing whether the EQ-5D-5L index behaves as expected in 300 adults with rheumatoid arthritis writes eight hypotheses before seeing the data. All scores are coded so that higher means better health, and two disease activity groups are defined beforehand from a clinical activity score.

1. Count the results. The H6 gap is 0.62 − 0.41 = 0.21 and the H8 gap is 0.48 − 0.43 = 0.05, so six hypotheses are confirmed and two are not.

$$P = \frac{6}{6 + 2} = 0.75$$

where 6 is the number confirmed and 2 the number not confirmed. By analogy with COSMIN's review-level rule, the study just reaches the 75% needed for a sufficient rating.

2. Look at which hypotheses failed. H7 is the known-groups test closest to a model, since it asks whether the index separates the disease activity states a model would use. H8 is a dimension-level discriminant test, which fails even though the dimension-level convergent test H3 passed. H5, the near-certain prediction about education, lifts the count; without it the study scores 5 of 7, or 0.714, below 75%.

3. Combine with earlier studies. Suppose two earlier illustrative studies in the same condition, each testing five of the same hypotheses, had 4 of 5 and 2 of 5 results in accordance. Taken together, the counts are as follows.

$$P = \frac{6 + 4 + 2}{8 + 5 + 5} = \frac{12}{18} = 0.667$$

where the numerator counts results in accordance and the denominator all results. Results not in accordance make up 6 / 18 = 0.333, so neither threshold is met and the summary is inconsistent (±). A review team would then look for an explanation, such as different disease severity across the samples.

4. Consider the effect on a model. A utility gap smaller than the true difference between health states shrinks the QALY gain and raises the incremental cost-effectiveness ratio, as the worked example on the discriminant validity page shows, so the H7 failure matters more for a submission than the count suggests.

Construct validity alongside content, criterion and responsiveness evidence

Construct validity sits among several measurement properties, and confusion usually comes from treating one as a substitute for another. The comparison follows the COSMIN manual.

Common misreadings of construct validity evidence

Most misreadings treat construct validity as a fixed attribute of an instrument rather than an accumulating argument about one use. The Standards evaluate interpretations of scores for proposed uses, and PMG36 refers to a particular patient population, so evidence from one condition transfers to another only with care.

A count of confirmed hypotheses depends on which hypotheses were written, and near-certain predictions, such as a weak correlation with an unrelated variable, raise it without testing much; COSMIN asks the review team to formulate one set of hypotheses and compare every study against it. The 75% rule summarises a body of evidence and is not a pass mark for one study. A correlation can differ significantly from zero and still be far smaller than predicted. High internal consistency is not evidence that the intended construct is measured.

Sources

• American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: AERA; 2014. Chapter 1, Validity. https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf
• Brazier J, Longworth L. NICE DSU Technical Support Document 8: An introduction to the measurement and valuation of health for NICE submissions. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; 2011. Sections 3.3 and 3.4. https://doi.org/10.15131/shef.data.33738301
• Cronbach LJ, Meehl PE. Construct validity in psychological tests. Psychological Bulletin. 1955;52(4):281-302. https://doi.org/10.1037/h0040957. Full text as reproduced in Classics in the History of Psychology: http://psychclassics.yorku.ca/Cronbach/construct.htm
• Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology. 2010;63(7):737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006. Taxonomy as presented by the COSMIN initiative: https://www.cosmin.nl/tools/cosmin-taxonomy-measurement-properties/
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0. Amsterdam: COSMIN; February 2018. Tables 1, 2, 4 and 8; chapter 3, pp. 32 and 40 to 41; sections 5.1, 5.3, 5.7 and 5.8. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Paragraph 4.3.10. https://www.nice.org.uk/process/pmg36