Reference Standard

Reference Standard in Diagnostic Test Evaluation

In a diagnostic accuracy study each participant's index test result is cross-classified against a reference standard, and the resulting two-by-two table supplies the sensitivity and specificity used in health technology assessment. The older name, gold standard, implied an error-free classification, which almost no test achieves. An imperfect reference standard biases accuracy because every disagreement is counted as an error of the index test, and that bias passes into the decision models that value tests.

From gold standard to best available method

STARD 2015 defines the clinical reference standard as the best available method for establishing the presence or absence of the target condition, and keeps the term gold standard for an error-free reference standard. Rutjes and colleagues, in a methods review in the NIHR HTA series, judge a perfect reference standard unlikely to exist in practice and trace the change of terminology to that observation.

Errors arise in several ways. A radiological reference can miss tumours below its level of detection, a biopsy can be taken outside the tumour, and conditions such as migraine have no histological or biochemical reference, only criteria based on symptoms and signs, some agreed at consensus meetings. Even a sound reference embodies a threshold for disease. In appendicitis, histology of the removed appendix and clinical follow-up classify some patients differently, because some inflamed appendices would have recovered without intervention.

The reference standard therefore defines the target condition in practice. QUADAS-2 asks whether there is concern that the target condition as defined by the reference standard does not match the review question, which matters to an economist because a model's treatment effects apply to a particular definition of disease. The classical accuracy design is a form of criterion validity, with the reference standard as the criterion.

How reference errors become index test errors

Within the accuracy framework, any disagreement between the reference standard and the index test is labelled a false result for the index test. Rutjes and colleagues explain that the net bias can run upwards or downwards depending on whether the errors of the two tests are correlated. When both tend to err in the same patients, as an index test and a reference may with early cancers, apparent accuracy is inflated.

With errors independent given true disease status, observed sensitivity follows from the true accuracy of both tests, as below.

$$Se_{obs} = \frac{p \, Se_T \, Se_R + (1 - p)(1 - Sp_T)(1 - Sp_R)}{p \, Se_R + (1 - p)(1 - Sp_R)}$$

where $Se_{obs}$ is the sensitivity observed against the reference standard, $Se_T$ and $Sp_T$ are the true sensitivity and specificity of the index test, $Se_R$ and $Sp_R$ are those of the reference standard, and $p$ is the true prevalence. Observed specificity follows in the same way from the people both tests call negative. The evidence review that informed QUADAS-2 summarised theoretical work showing that an inaccurate reference standard leads to underestimation of test performance when errors are independent and overestimation when they are dependent, and underestimation when the index test is more accurate than the reference.

Appraising the reference standard in accuracy studies

STARD 2015 asks reports to describe it in enough detail to allow replication, give the rationale for choosing it where alternatives exist, and state whether its assessors knew the index test results. Its authors note that accuracy estimates can differ with the definition of the reference standard.

QUADAS-2 asks whether the reference standard is likely to classify the target condition correctly and whether its results were interpreted without knowledge of the index test. QUADAS-3, published in 2026 and now recommended by its authors, has a Target Condition domain that also asks whether the reference standard avoided incorporating the index test and whether the interval between the two tests was appropriate. The same domain asks whether the target condition was assessed in all participants and in the same way, which are questions of verification rather than of the reference standard's own accuracy.

Building or replacing a reference standard when none is adequate

Rutjes and colleagues, and Reitsma and colleagues, group the solutions into four families. Missing reference results can be imputed or adjusted for, an imperfect reference can be corrected, a reference can be constructed from several tests, or the index test can be validated against clinical data instead.

Correction inverts the formula above, or a version of it that allows correlated errors, and needs reliable external values for the reference standard's error rates and any correlation of errors; if those values are wrong, the corrected estimates can be more biased than the unadjusted ones. A composite reference standard combines several imperfect tests through a rule fixed in advance, such as any test positive, and the index test plays no part in the classification. Discrepant analysis retests only patients in whom the index test and the first reference disagree, so the index test decides who is verified and its accuracy tends to be biased upwards. In a panel diagnosis, experts classify each patient from all available information, with a risk of incorporation bias if they see the index test result.

Latent class analysis treats true disease status as an unobserved class and estimates prevalence and the accuracy of every test from the pattern of their results, so its target condition is a mathematical entity rather than a clinical definition. Simulation studies cited by Rutjes and colleagues show that positively dependent errors make its estimates too high, especially for the more accurate test. Reviewing 64 diagnostic studies using latent class models, van Smeden and colleagues found that 61% relied on conditional independence and 28% reported nothing that would allow the assumptions to be checked.

Clinical follow-up often serves as the reference, or as a second reference for some patients. Its risks are that the condition changes during follow-up, that a condition present at testing never produces detectable events, or that the period is poorly chosen.

The Chlamydia trachomatis data analysed by Alonzo and Pepe, and reworked by Rutjes and colleagues, show how much the choice matters. An enzyme immunoassay was the index test in 324 specimens, with culture as the first reference and PCR as the second.

The same immunoassay results give sensitivities from 0.83 to 0.92, depending only on how the target condition is classified.

Consequences for test-treatment models and NICE diagnostics assessment

A test-treatment model turns sensitivity and specificity into numbers of people correctly and wrongly treated, so bias in accuracy becomes bias in costs and QALYs. NICE's Diagnostics Assessment Programme manual of 2011, replaced in 2022 by the health technology evaluations manual (PMG36), stated that, depending on the correlation between the technologies evaluated and the reference standard, sensitivity and specificity can be unduly high or low, that they can be adjusted if the correlations and the reference standard's error rate are known, and that otherwise sensitivity analysis is needed to explore the uncertainty.

In July 2025 the programme was folded into the NICE HealthTech programme, whose manual (PMG48) has no equivalent passage on imperfect reference standards. PMG48 lists the reference standard among the sources of heterogeneity in meta-analysis of test accuracy, and asks for a linked evidence model whose links between test results, care decisions and final outcomes are specified and justified. A reference standard that defines disease differently from the trials that measured treatment benefit weakens one of those links, and Rutjes and colleagues note that results from current intervention studies may not apply to the extra cases a more sensitive test detects.

Worked example with a new test judged against an imperfect reference

The figures below are illustrative. A new test with true sensitivity 0.90 and specificity 0.95 is evaluated in 10,000 people with a true prevalence of 10%. The reference standard has sensitivity 0.85 and specificity 0.98, with errors independent of the new test's errors given true status.

1. Expected two-by-two table. Of 1,000 people with disease, 900 test positive and the reference misses 135 of them; of 9,000 without it, the reference wrongly calls 9 of 450 test positives and 171 of 8,550 test negatives positive.

2. Observed accuracy. Sensitivity and specificity are read from the columns, as follows.

$$Se_{obs} = \frac{774}{774 + 256} = 0.751 \qquad Sp_{obs} = \frac{8394}{8394 + 576} = 0.936$$

where 774 and 256 are reference-positive people the new test calls positive and negative, and 8,394 and 576 are reference-negative people it calls negative and positive. The test appears to miss a quarter of cases although it misses a tenth, because 135 of its apparent false positives are cases the reference missed and 171 of its apparent false negatives are reference errors.

3. Value to the decision maker. Each true positive is taken to be worth a net £5,000 (health gain valued at the threshold, less treatment cost), each false positive costs £600 in unnecessary work-up, and the 10% prevalence comes from separate epidemiological data. The highest price at which testing beats not testing is as follows.

$$P_{max} = p \times Se \times G - (1 - p)(1 - Sp) \times L$$

where $P_{max}$ is the maximum price per person tested, $G$ is the net value of a true positive, $L$ is the cost of a false positive, and $Se$ and $Sp$ are the accuracy values the model uses. With the true accuracy the result is as follows.

$$P_{max} = 0.10 \times 0.90 \times 5000 - 0.90 \times 0.05 \times 600 = 450 - 27 = 423$$

where the result is in pounds. With the observed accuracy the result is as follows.

$$P_{max} = 0.10 \times 0.751 \times 5000 - 0.90 \times 0.064 \times 600 = 375.50 - 34.56 = 340.94$$

where the result is in pounds.

4. Interpretation. The imperfect reference understates the test's value by about £82 per person, a fifth of its true value. At a price of £380 the test has a net monetary benefit of £43 per person against no testing on its true accuracy, but a loss of about £39 on its observed accuracy, which reverses a sound adoption decision. Had the two tests' errors been positively correlated, the same study could instead have overstated the test's value.

Misreadings of what a reference standard can show

A study that finds a new test less sensitive than the reference cannot by itself separate a worse test from a better one, because every disagreement is charged to the index test. Correction and latent class methods replace the imperfection with assumptions about error rates and dependence that the data often cannot test, and wrong assumptions can enlarge the bias.

Where no acceptable reference standard exists, Rutjes and colleagues conclude that holding on to the accuracy paradigm may be fruitless. Clinical validation, relating index test results to other clinical characteristics and future events, is then the better route, and for an HTA it points towards evidence on outcomes rather than accuracy alone.

Sources

• Alonzo TA, Pepe MS. Using a combination of reference tests to assess the accuracy of a new diagnostic test. Statistics in Medicine. 1999;18(22):2987-3003. https://pubmed.ncbi.nlm.nih.gov/10544302/
• Bossuyt PM, Reitsma JB, Bruns DE, et al; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. https://doi.org/10.1136/bmj.h5527
• National Institute for Health and Clinical Excellence. Diagnostics Assessment Programme manual. Manchester: NICE; December 2011. Section 14.3.2. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-diagnostics-guidance/Diagnostics-assessment-programme-manual.pdf
• National Institute for Health and Care Excellence. NICE health technology evaluations: the manual (PMG36). London: NICE; 31 January 2022, version replaced on 14 July 2025. https://www.nice.org.uk/process/pmg36/update/pmg36/documents/supporting-documentation
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48). London: NICE; 14 July 2025, last updated 17 December 2025. Sections 2.2.15 and 2.3.14. https://www.nice.org.uk/process/pmg48
• Reitsma JB, Rutjes AW, Khan KS, Coomarasamy A, Bossuyt PM. A review of solutions for diagnostic accuracy studies with an imperfect or missing reference standard. Journal of Clinical Epidemiology. 2009;62(8):797-806. https://doi.org/10.1016/j.jclinepi.2009.02.005
• Rutjes AWS, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PMM. Evaluation of diagnostic tests when there is no gold standard. A review of methods. Health Technology Assessment. 2007;11(50). https://doi.org/10.3310/hta11500
• van Smeden M, Naaktgeboren CA, Reitsma JB, Moons KG, de Groot JA. Latent class models in diagnostic studies when there is no reference standard, a systematic review. American Journal of Epidemiology. 2014;179(4):423-431. https://doi.org/10.1093/aje/kwt286
• Whiting P, Rutjes A, Westwood M, et al. Updating QUADAS: evidence to inform the development of QUADAS-2. University of Bristol; 2010. https://www.bristol.ac.uk/media-library/sites/quadas/migrated/documents/quadas2reportv4.pdf
• Whiting PF, Rutjes AW, Westwood ME, et al; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529-536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
• Whiting PF, Tomlinson E, Rutjes AWS, et al. QUADAS-3: a revised tool for the quality assessment of diagnostic test accuracy studies. Annals of Internal Medicine. 2026;179(4):548-555. https://doi.org/10.7326/ANNALS-25-02104