Structural Validity (Measurement Instrument)

Structural Validity (Measurement Instrument) and the Dimensions Behind a Health Score

Before the items of a quality-of-life questionnaire are summed into a total, or split into subscale scores, someone has to show that the answers really fall into the dimensions the scoring assumes. Structural validity is the measurement property that carries this burden, and COSMIN, the consensus initiative that defines measurement properties for patient-reported outcome measures (PROMs), treats it as one part of construct validity. The term is used here in its instrument sense. Structural validity in model validation concerns whether a decision model's architecture, such as its health states and time horizon, represents the disease process, which is a question about models and not about questionnaires. For a health economist the instrument sense matters whenever a score is used as a trial outcome, as a predictor in a mapping function, or as the descriptive base of a new preference-based measure, because a score built on the wrong dimensions weights the items in a way nobody intended.

The COSMIN definition and why its name was contested

COSMIN defines structural validity as the degree to which the scores of a PROM are an adequate reflection of the dimensionality of the construct to be measured. The 2018 COSMIN guideline for systematic reviews, reported by Prinsen and colleagues, adds that it is usually assessed by factor analysis or by item response theory or Rasch analysis.

The term itself was the least settled part of the taxonomy. In the international Delphi study behind it, consensus required agreement of at least 67 per cent of the panel, and agreement on terminology ranged from 74 to 88 per cent except for structural validity, which reached 56 per cent. The panel discussed the term at length before settling on it.

Reflective models as the condition for testing structure

Factor analysis assumes that the items are caused by the construct, so that a person with more of the latent variable tends to score higher on every item. COSMIN states that structural validity is only relevant for PROMs based on such a reflective model, and its checklist asks first whether the scale consists of effect indicators. In a formative model the items together make up the construct, need not correlate, and give factor solutions with no clear meaning.

The distinction is live for instruments used in economic evaluation. Gamst-Klaussen, Gudex and Olsen combined confirmatory tetrad analysis with confirmatory factor analysis on EQ-5D-5L data from 7,933 respondents in six countries, 1,760 of them healthy and 6,173 in one of seven disease groups. Their best-fitting model treated self-care and usual activities as effect indicators, and pain or discomfort, anxiety or depression, and mobility as causal indicators, with a CFI of 0.992, TLI of 0.972, RMSEA of 0.075 and SRMR of 0.012. They concluded in 2018 that the EQ-5D contains both causal and effect indicators. Under COSMIN's own condition, a single reflective factor model is therefore the wrong test of the EQ-5D's structure. The good fit indices above belong to the authors' mixed causal and effect model, not to a reflective one.

Methods for assessing structural validity

Classical test theory and item response theory both offer tests of structure. Exploratory factor analysis lets the data suggest how many factors there are and which items load on each, while confirmatory factor analysis tests a structure stated in advance. Item response theory and Rasch models check whether the items form a single dimension and order people and items along it.

COSMIN rates a confirmatory factor analysis result against fit indices. Under the criteria in the 2018 guideline, structural validity is sufficient when the comparative fit index (CFI), the Tucker-Lewis index (TLI) or a comparable measure exceeds 0.95, or the root mean square error of approximation (RMSEA) is below 0.06, or the standardised root mean square residual (SRMR) is below 0.08. The result is indeterminate when not all the information needed for a sufficient rating is reported, and insufficient when the criteria are not met.

The criteria are joined by "or", so one index meeting its threshold is enough. A model can clear the rule on CFI while its RMSEA signals poor fit, and a careful review reports all the indices.

Study quality and sample size under the COSMIN Risk of Bias checklist

A fit statistic says little if the study behind it was weak. The COSMIN Risk of Bias checklist, published by Mokkink and colleagues in 2018, adapted the original COSMIN checklist and its four-point rating system for systematic reviews of PROMs. Its structural validity box asks whether the scale is based on a reflective model, whether the study concerns unidimensionality or structural validity, whether exploratory or confirmatory factor analysis was performed, and whether the sample size was adequate.

In the December 2017 version of the checklist the top rating for a factor analysis needs at least seven times the number of items and at least 100 respondents. The sample size standard was kept for single studies of structural validity because their results cannot be pooled. The Risk of Bias checklist became version 3.0 within the COSMIN guideline version 2.0 reported by Mokkink, Elsman and Terwee in 2024, so a review should state which version it used.

Structural validity as the gate for internal consistency

COSMIN orders the properties so that structure is examined before internal consistency. Its criteria rate internal consistency reliability sufficient only when Cronbach's alpha is at least 0.70 for each unidimensional scale or subscale and there is at least low evidence for sufficient structural validity. The COSMIN user manual explains that evidence for structural validity or unidimensionality is a prerequisite for interpreting Cronbach's alpha, and recommends evaluating structural validity first. The version 2.0 criteria state the same condition as at least low evidence for sufficient unidimensionality.

The order matters because alpha is computed for a set of items assumed to measure one thing. Alpha for a total score that spans two dimensions can look respectable and still describe nothing coherent, while the alphas for the correct subscales are the ones that answer the reliability question.

Dimensionality in preference-based instruments

Building a preference-based measure from a long questionnaire depends on a defensible dimensional structure, because each dimension becomes an attribute of the health states that are later valued. Mavranezouli and colleagues faced this with the CORE-OM, a 34-item measure of common mental health problems whose domains are highly correlated. They used Rasch analysis to reduce the number of items and response levels and so produce a unidimensional measure. The result, reported in 2011, was CORE-6D, a two-dimensional descriptive system with a unidimensional five-item emotional component and one physical symptom item. Eleven plausible emotional states combined with three levels of the physical item give 11 × 3 = 33 plausible health states.

In this setting structural evidence decides which health states exist to be valued with methods such as the time trade-off. For an established generic measure scored with a value set, the role is different. The utility attached to a state comes from preferences over whole states, not from adding items that share a factor. The causal-indicator evidence for the EQ-5D does warn against reading a factor score from its five dimensions as a measure of health-related quality of life.

Worked example with a two-subscale questionnaire (illustrative)

All numbers are illustrative. A 12-item disease-specific questionnaire is scored as two subscales, physical function with six items and emotional function with six items, and a trial team wants to use both subscale scores as secondary outcomes and as candidate predictors in a mapping function.

1. Sample size. Seven times the number of items is 7 × 12 = 84, and the December 2017 checklist also needs at least 100 respondents, so the top rating needs 100 respondents. A study of 90 patients falls short of that rating even though it exceeds 84.

2. Fit of the two-factor model. A confirmatory factor analysis in 300 patients gives CFI 0.96, TLI 0.94, RMSEA 0.08 and SRMR 0.09. CFI is above 0.95, so the result is sufficient under the COSMIN rule, while RMSEA and SRMR both miss their thresholds. The review records the rating with all four indices, and the report notes that support rests on one index.

3. Internal consistency of each subscale. Cronbach's alpha is computed per subscale.

$$\alpha = \frac{k}{k-1}\left(1-\frac{\sum_{i=1}^{k} \sigma_i^2}{\sigma_X^2}\right)$$

where $\alpha$ is Cronbach's alpha, $k$ is the number of items in the subscale, $\sigma_i^2$ is the variance of item $i$ and $\sigma_X^2$ is the variance of the subscale total. For physical function the item variances sum to 6.0 and the total variance is 20.0, so alpha is 6/5 × (1 − 6/20), which is 1.2 × 0.7 = 0.84. For emotional function the item variances sum to 6.0 and the total variance is 15.0, so alpha is 1.2 × (1 − 6/15), which is 1.2 × 0.6 = 0.72.

4. Interpretation. Both alphas are at least 0.70 and structural validity is rated sufficient, so internal consistency is sufficient for each subscale under COSMIN. The emotional subscale sits close to the threshold. In a COSMIN review the structural validity condition refers to at least low evidence across the body of studies, so this one study's sufficient rating is a single input and not the summary judgement. Had the factor analysis supported only one factor, the two subscale scores would have no structural basis as separate outcomes, and entering both into a mapping function would rest on a split the data did not show.

How the instrument sense differs from neighbouring properties

Structural validity shares a name with a model validation concept and shares methods with other measurement properties, which invites confusion. Structural validity in model validation asks whether a decision model's architecture, including its health states and time horizon, represents the real-world process being modelled. The instrument sense asks whether questionnaire scores reflect the dimensions of a construct, and evidence for one says nothing about the other.

Multiple-group factor analysis tests whether the structure found in one population holds in another, which is how structural validity and cross-cultural validity connect.

Common misreadings of structural validity evidence

Good fit is often read as proof that the instrument measures the intended construct. Fit shows only that the correlations are consistent with the proposed structure, and construct validity needs further hypotheses tested against other evidence. A single index that clears the COSMIN threshold can sit beside others that do not, and the "or" in the rule is a minimum standard, not a summary of fit.

Applying factor analysis to items that are formative breaks COSMIN's own condition, as the EQ-5D work on causal indicators shows. A model modified repeatedly until it fits the data in hand is exploratory, whatever software routine produced it. A dimensional structure shown in one population may not hold in another, which is a question for cross-cultural validity, and a structure that supports a disease-specific score does not turn that score into a utility, which still needs a value set or a justified mapping function before such a score can enter a cost-utility analysis.

Sources

• Gamst-Klaussen T, Gudex C, Olsen JA. Exploring the causal and effect nature of EQ-5D dimensions: an application of confirmatory tetrad analysis and confirmatory factor analysis. Health and Quality of Life Outcomes. 2018;16:153. https://doi.org/10.1186/s12955-018-0975-y
• Mavranezouli I, Brazier JE, Young TA, Barkham M. Using Rasch analysis to form plausible health states amenable to valuation: the development of CORE-6D from a measure of common mental health problems (CORE-OM). Quality of Life Research. 2011;20:321-333. https://doi.org/10.1007/s11136-010-9768-4
• Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology. 2010;63(7):737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006
• Mokkink LB, de Vet HCW, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, Terwee CB. COSMIN Risk of Bias checklist for systematic reviews of Patient-Reported Outcome Measures. Quality of Life Research. 2018;27(5):1171-1179. https://doi.org/10.1007/s11136-017-1765-4. Checklist version of December 2017: https://cosmin.nl/wp-content/uploads/COSMIN_risk-of-bias-checklist_dec-2017.pdf
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0. Amsterdam: COSMIN; February 2018. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• Mokkink LB, Elsman EBM, Terwee CB. COSMIN guideline for systematic reviews of patient-reported outcome measures version 2.0. Quality of Life Research. 2024;33(11):2929-2939. https://doi.org/10.1007/s11136-024-03761-6
• Prinsen CAC, Mokkink LB, Bouter LM, Alonso J, Patrick DL, de Vet HCW, Terwee CB. COSMIN guideline for systematic reviews of patient-reported outcome measures. Quality of Life Research. 2018;27(5):1147-1157. https://doi.org/10.1007/s11136-018-1798-3