Criterion Validity

Criterion Validity Against a Gold Standard

Criterion validity is the test applied when a reference measurement of the same quantity already exists and the question is how closely a shorter, cheaper or more available measure reproduces it. In health economics it is rarer for quality of life instruments than many validation papers imply, since the COSMIN consensus holds that patient-reported outcome measures have no gold standard, short forms apart. It is common for the data behind costs and cohorts, such as claims algorithms checked against medical records, self-reported service use and sick leave checked against records, and mapping predictions checked against observed EQ-5D values. Such studies use concurrent or predictive designs and agreement statistics, and the illustrative claims algorithm below shows how a small false positive rate distorts a cost estimate.

Concurrent and predictive designs within the validity domain

The COSMIN taxonomy, agreed through an international Delphi study and published by Mokkink and colleagues in 2010, places criterion validity in the validity domain alongside content and construct validity. The COSMIN user manual defines it as the degree to which the scores of a patient-reported outcome measure (PROM) are an adequate reflection of a gold standard. The US Food and Drug Administration's 2009 guidance on patient-reported outcome measures defines it similarly.

The joint Standards for Educational and Psychological Testing (2014) describe two designs historically distinguished for gathering test-criterion evidence. In a concurrent study the measure and the criterion are obtained at about the same time; in a predictive study the criterion is obtained later. The Standards find concurrent evidence particularly useful for alternative measures of a construct with an accepted measurement procedure, as with a short questionnaire against its parent or a coded diagnosis against the medical record. A predictive design keeps the time lag of the real use, as when a baseline score is used to anticipate later admissions or costs.

Why patient-reported and preference-based measures rarely have a gold standard

Health-related quality of life and its utility values are known only through reports and valuations, so no external measurement holds the true value. The COSMIN user manual records that its panel reached consensus that no gold standard exists for PROMs, with one exception. When a shortened instrument is compared with the original long version, the long version can be treated as the gold standard. It adds that authors often wrongly treat a widely used and well-known comparator as a gold standard, and that such a study should be assessed as a study of construct validity.

The FDA guidance agrees that for most PROs criterion validity testing is not possible, for instance for any symptom known only to the patient, and cites a new sleep scale compared with polysomnography as a rare exception. A comparison of the EQ-5D with the SF-6D or HUI3 therefore belongs to convergent validity, however established the comparator.

Where a criterion exists in health economic work

A criterion is available when the reference is a fuller or more direct record of the same quantity rather than another view of it. Four situations recur in economic evaluation, each with its own design risks.

Short forms against their parent instrument

Ware, Kosinski and Keller used regression to select and score 12 items from the SF-36 to reproduce its Physical and Mental Component Summary scales in the general US population (2,333 people). The SF-12 achieved multiple $R^2$ values of 0.911 and 0.918. Within the estimation sample $R^2$ is the squared correlation between predicted and observed scores, so 0.911 corresponds to a correlation of about $\sqrt{0.911} = 0.95$. Brazier and Roberts later derived an SF-6D from the SF-12 for economic evaluation.

The design needs care. The SF-12 items were scored from SF-36 responses, and the COSMIN manual warns that a short-versus-long comparison can be biased when the short-version scores are computed from answers given to the long version. Ware and colleagues also reported that standard errors were nearly always larger for the SF-12, so close agreement in average scores did not mean equal precision.

Mapping predictions against observed utilities

A mapping function predicts EQ-5D utilities from a clinical or condition-specific measure, and the observed EQ-5D values act as the criterion. The NICE Decision Support Unit's Technical Support Document 22 states that for most economic models predictions should be close to the sample means for patients with the same covariates, and recommends plots of predicted against observed group means by a conditioning variable. Predictions always vary less than the data because they abstract from random variation, which the document says does not matter for cost-effectiveness analysis, so a modest patient-level correlation is not by itself poor criterion validity. TSD 22 adds that a purely random split for out-of-sample testing has very little power to detect systematic errors from misspecification.

Administrative-data algorithms against medical records

Studies built on real-world data such as insurance claims identify patients through code-based algorithms, checked against a reference standard such as medical record review and reported as diagnostic accuracy statistics. In a review of 271 such validation articles, Benchimol and colleagues found that only 36.9% reported four or more accuracy estimates, and that in 65.9% of studies reporting predictive values the prevalence in the validation cohort was higher than in the administrative data, potentially inflating those values. They warn of misclassification bias and incorrect estimates of incidence and health services utilisation, which costing studies and budget impact analysis rely on.

Resource use and absence data against records and registries

Criterion studies of cost data also show that the record is not always the better criterion. In the POPMACT trial of interventions for deliberate self-harm, Byford and colleagues compared patient self-report with general practice records. Agreement on overall costs was relatively high but hid substantial variation, since GP records were more accurate on general practice contacts, yet less reliable on other health services.

For productivity costs, Munk Killingmo and colleagues tested the iMTA Productivity Cost Questionnaire (iPCQ) against Norwegian registry data in 144 people on sick leave for at least four weeks with musculoskeletal disorders. Agreement on whether long-term absenteeism occurred was 100%, but since everyone was recruited while on such leave, false reports of absence could not be detected. The two-way random, average-measures agreement intraclass correlation was 0.97 for duration and 0.86 for days of complete absence, with a median difference of 17 days on the latter; an average-measures coefficient describes the mean of the two sources and so flatters the questionnaire used alone. The authors note that the iPCQ omits part-time sick leave and may overestimate absence.

Statistics for agreement with a criterion

The statistic follows the scale of the measure and of the criterion. The COSMIN manual prefers a correlation when both are continuous, the area under the receiver operating characteristic curve when the measure is continuous and the criterion dichotomous, and sensitivity and specificity when both are dichotomous; its criteria rate criterion validity as sufficient when the correlation or the area is at least 0.70. For a dichotomous measure against a dichotomous criterion, the measures are as follows.

$$Se = \frac{TP}{TP + FN} \qquad Sp = \frac{TN}{TN + FP}$$

where $Se$ is sensitivity and $Sp$ is specificity, $TP$ and $FN$ are criterion-positive cases that the measure flags and misses, and $TN$ and $FP$ are criterion-negative cases that it clears and wrongly flags. The positive predictive value, the share of flagged cases that are true cases, also depends on prevalence, as follows.

$$PPV = \frac{Se \times p}{Se \times p + (1 - Sp) \times (1 - p)}$$

where $PPV$ is the positive predictive value and $p$ is the prevalence where the measure is used, which need not equal the prevalence in the validation sample. The positive predictive value falls as prevalence falls, while the negative predictive value rises.

For a continuous measure against a yes or no criterion, the area under the curve summarises the ROC curve and equals the probability that a randomly chosen diseased subject is ranked above a randomly chosen non-diseased one. When both are continuous and the measure is to replace the criterion in a cost or QALY calculation, level matters as well as ordering. A Pearson correlation is unchanged by any positive linear transformation of either score, so an agreement form of the intraclass correlation is more informative, reported with the size of the systematic difference; the iPCQ study reported a median difference and a Wilcoxon signed-rank test.

Worked example of a heart failure algorithm in claims data

The figures below are illustrative and do not come from any study. An analyst wants the mean annual healthcare cost per heart failure patient from a claims database, using a code-based algorithm to identify patients. The example assumes that cost depends only on true disease status, so false positives cost the same as the average person without heart failure. The algorithm is first checked against medical record review in 1,000 patients, 200 of whom have heart failure according to the records.

1. Accuracy in the validation sample. Sensitivity and specificity come from the columns, as follows.

$$Se = \frac{180}{180 + 20} = 0.90 \qquad Sp = \frac{760}{760 + 40} = 0.95$$

where 180 and 20 are the true positives and false negatives, and 760 and 40 the true negatives and false positives. From the rows, 180 / 220 = 0.818 of flagged patients have heart failure and 760 / 780 = 0.974 of unflagged patients do not.

2. Effect on a cost estimate. Suppose patients with heart failure cost £6,000 a year and patients without it £1,000. The mean annual cost of the flagged cohort is as follows.

$$\bar{C} = \frac{180 \times 6000 + 40 \times 1000}{220} = \frac{1120000}{220} = 5091$$

where $\bar{C}$ is the mean annual cost in pounds per flagged patient. The 40 false positives pull the estimate £909 below the true £6,000, about 15.2% too low.

3. Moving to the full database. The example assumes that sensitivity and specificity transfer unchanged from the validation sample to the database, which should itself be checked. The algorithm flags 6.7% of the database, and the true prevalence can be recovered from that share, as follows.

$$p = \frac{0.067 - (1 - 0.95)}{0.90 + 0.95 - 1} = \frac{0.017}{0.85} = 0.02$$

where $p$ is the true prevalence, 0.067 is the flagged share and 0.05 is the false positive rate. A prevalence of 2%, against 20% in the validation sample, gives the following.

$$PPV = \frac{0.90 \times 0.02}{0.90 \times 0.02 + 0.05 \times 0.98} = \frac{0.018}{0.067} = 0.269$$

where 0.98 is one minus the prevalence. In 100,000 people the algorithm flags 1,800 of the 2,000 true cases and 4,900 of the 98,000 without heart failure, 6,700 in all, with a mean cost calculated as follows.

$$\bar{C} = \frac{1800 \times 6000 + 4900 \times 1000}{6700} = \frac{15700000}{6700} = 2343$$

where $\bar{C}$ is again the mean annual cost in pounds per flagged patient.

4. Interpretation. Sensitivity and specificity are unchanged, but the share of flagged patients with heart failure falls from 82% to 27%. The flagged cohort is 3.35 times the true number of cases and its total cost of £15.7 million is about 31% above the £12 million spent on the true cases, while its mean cost of £2,343 is less than half the true £6,000. Quoting the predictive value from a validation sample enriched with cases, as Benchimol and colleagues found in most studies reporting predictive values, would have overstated accuracy.

Using criterion evidence in economic evaluation and HTA

Criterion evidence shows how closely one measure reproduces a reference in a particular population. The Standards note that observed test-criterion correlations vary substantially between settings, in some domains largely through sampling fluctuation, score ranges and criterion reliability, so an observed coefficient depends on the score range and criterion reliability in the sample.

In a decision model, criterion evidence is most useful as parameters. The accuracy of a cohort-defining algorithm can correct prevalence and cost inputs, and the analysis should vary the algorithm's accuracy in the model's sensitivity analysis. A mapping function is judged on whether predicted means match observed means across the severity range in the model. A short form that meets COSMIN's 0.70 receives its sufficiency rating, a rule for systematic reviews of measurement properties; a QALY or cost calculation also needs the mean difference and the precision.

In choosing a utility instrument for health technology assessment, criterion validity has little role, because NICE's manual (PMG36, section 4.3) frames the evidence needed to argue that the EQ-5D is inappropriate in terms of content validity, construct validity and responsiveness, not criterion validity.

Boundaries with construct validity, responsiveness and model validation

Construct validity, with its convergent and discriminant parts, applies when no gold standard exists, and COSMIN splits responsiveness the same way into a criterion approach and construct approaches based on hypotheses. Model validation compares a model's outputs with observed data, and the predictive validity of a model concerns its forecasts rather than the scores of a measure.

Sources

• American Educational Research Association, American Psychological Association, National Council on Measurement in Education. Standards for Educational and Psychological Testing. Washington, DC: AERA; 2014. Chapter 1, Validity. https://www.testingstandards.net/uploads/7/6/6/4/76643089/standards_2014edition.pdf
• Benchimol EI, Manuel DG, To T, Griffiths AM, Rabeneck L, Guttmann A. Development and use of reporting guidelines for assessing the quality of validation studies of health administrative data. Journal of Clinical Epidemiology. 2011;64(8):821-829. https://doi.org/10.1016/j.jclinepi.2010.10.006
• Brazier JE, Roberts J. The estimation of a preference-based measure of health from the SF-12. Medical Care. 2004;42(9):851-859. https://doi.org/10.1097/01.mlr.0000135827.18610.0d
• Byford S, Leese M, Knapp M, Seivewright H, Cameron S, Jones V, et al. Comparison of alternative methods of collection of service use data for the economic evaluation of health care interventions. Health Economics. 2007;16(5):531-536. https://doi.org/10.1002/hec.1175
• Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29-36. https://doi.org/10.1148/radiology.143.1.7063747
• Mokkink LB, Terwee CB, Patrick DL, Alonso J, Stratford PW, Knol DL, et al. The COSMIN study reached international consensus on taxonomy, terminology, and definitions of measurement properties for health-related patient-reported outcomes. Journal of Clinical Epidemiology. 2010;63(7):737-745. https://doi.org/10.1016/j.jclinepi.2010.02.006. Taxonomy as presented by the COSMIN initiative: https://www.cosmin.nl/tools/cosmin-taxonomy-measurement-properties/
• Mokkink LB, Prinsen CAC, Patrick DL, Alonso J, Bouter LM, de Vet HCW, Terwee CB. COSMIN methodology for systematic reviews of Patient-Reported Outcome Measures (PROMs): user manual. Version 1.0. Amsterdam: COSMIN; February 2018. Table 4 and sections 5.6 and 5.8. https://www.cosmin.nl/wp-content/uploads/COSMIN-syst-review-for-PROMs-manual_version-1_feb-2018.pdf
• Munk Killingmo R, Tveter AT, Småstuen MC, Storheim K, Grotle M. Comparison of self-reported and public registered absenteeism among people on long-term sick leave due to musculoskeletal disorders: criterion validity of the iMTA Productivity Cost Questionnaire. European Journal of Health Economics. 2021;22(6):865-872. https://doi.org/10.1007/s10198-021-01294-0
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Section 4.3. https://www.nice.org.uk/process/pmg36
• US Food and Drug Administration. Guidance for Industry. Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims. Silver Spring, MD: FDA; December 2009. Section III.E.2. https://www.fda.gov/media/77832/download
• Wailoo A, Hernández Alava M, Pudney S. NICE DSU Technical Support Document 22: Mapping to estimate health state utilities (updates and replaces TSD 10). Sheffield: Decision Support Unit, ScHARR, University of Sheffield; June 2023, updated September 2026. Sections 4.5 and 4.6. https://doi.org/10.15131/shef.data.33849340
• Ware J Jr, Kosinski M, Keller SD. A 12-Item Short-Form Health Survey: construction of scales and preliminary tests of reliability and validity. Medical Care. 1996;34(3):220-233. https://doi.org/10.1097/00005650-199603000-00003