Verification Bias

Verification Bias in Diagnostic Accuracy Studies

Verification bias arises in a diagnostic accuracy study when the index test result decides who goes on to the reference standard, or which reference standard they get. Patients testing positive are commonly sent for the definitive test, often invasive or costly, while many with a negative result are never checked or are checked by a weaker method. The two-by-two table then no longer represents the people tested, so the sensitivity and specificity it yields can be wrong however large the study. In health technology assessment these two figures set the numbers of missed cases and false alarms in linked-evidence models, so the error carries through to costs and QALYs. Two forms are distinguished, partial verification and differential verification, and the partial form is also called work-up bias.

Partial and differential verification

The Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy describes partial verification bias as arising when not all study patients are verified by the reference standard and the choice of whom to verify is not random, particularly when the index test result influences it. Begg and Greenes described the same problem in 1983 as estimates taken from a sample restricted to verified cases, and noted that adjusting them requires assumptions about how cases were selected for verification.

Differential verification occurs when some patients are verified by one reference standard and others by another, which the Handbook says is of most concern when index test positives receive one method and negatives a second. Its example is imaging for solid tumours, where a tumour that is seen is biopsied and a patient with no visible tumour is followed up, because a tumour that cannot be found cannot be biopsied. Such designs often reflect the difficulty of justifying an invasive test in people at low risk, but the bias remains.

Which way the estimates move

The direction of partial verification bias depends on who goes unverified and how they are analysed. When only verified patients are analysed and test negatives are verified less often, Kosinski and Barnhart note that sensitivity would often be higher and specificity lower than the true values. Rutjes and colleagues add that this pattern may leave the diagnostic odds ratio unchanged, which is one reason summary accuracy measures can hide the problem.

Predictive values behave differently. When verification depends only on the test result, the verified patients within each test-result group are a random sample of that group, so the positive predictive value and negative predictive value among verified patients are unbiased; Arifin and Yusof note that the Begg and Greenes and imputation corrections rely on this. If unverified test negatives are simply counted as disease-free, which the Cochrane Handbook treats as an extreme form of differential verification, both sensitivity and specificity are overestimated.

Differential verification matters, the Handbook explains, when the reference standards differ in accuracy and the choice between them follows the index test result, and accuracy is overestimated when index test errors are correlated with errors of the weaker standard. The QUADAS-2 background document describes a D-dimer study for pulmonary embolism with scans for test positives and clinical follow-up for test negatives, where an embolism missed by the test may also be missed at follow-up, so a false negative is recorded as a true negative and both sensitivity and specificity are overestimated.

Evidence from meta-epidemiological studies

Meta-epidemiological studies compare flawed and sound studies across many meta-analyses. Lijmer and colleagues found that studies using different reference tests for positive and negative index results overestimated diagnostic performance compared with studies using a single reference test, with a relative diagnostic odds ratio of 2.2 (95% CI 1.5 to 3.3).

Rutjes and colleagues analysed 487 primary studies from 31 meta-analyses. Studies relying on two or more reference standards reported diagnostic odds ratios on average 60% higher than studies with a single standard, although the effect was not statistically significant and varied a good deal between meta-analyses. Like Lijmer, they found no consistent effect of partial verification and suggested that its direction and size are hard to predict. Of the 487 studies, 453 (93%) verified all index test results with a reference standard.

Assessing verification in a systematic review

A systematic review of test accuracy assesses verification as part of risk of bias. QUADAS-2 places it in the flow and timing domain, whose signalling questions ask whether all patients received a reference standard and whether they received the same one. QUADAS-3, published in 2026 and now recommended by the QUADAS team at the University of Bristol, moves these items into a Target Condition domain, asking whether the target condition was assessed in all participants and in the same way. Its new Analysis domain asks how missing reference standard data were handled.

Chapter 10 of the Cochrane Handbook (version 1.0) states that corrections for verification bias have to be made within individual studies before meta-analysis, and suggests investigating verification as a source of heterogeneity. Chapter 9 adds that random sampling of patients for verification, even at different rates by test result, still allows unbiased estimates when the analysis allows for the sampling plan.

Correcting partial verification under a missing at random assumption

Correction treats unverified patients as missing data. The Begg and Greenes method, as written by Arifin and Yusof, uses Bayes' theorem to combine predictive values in verified patients with the test-result distribution in everyone tested.

$$\widehat{Se} = \frac{P(T^+)\,P(D^+ \mid T^+, V)}{P(T^+)\,P(D^+ \mid T^+, V) + P(T^-)\,P(D^+ \mid T^-, V)}$$

where $\widehat{Se}$ is the corrected sensitivity, $P(T^+)$ and $P(T^-)$ are the proportions of all tested patients with positive and negative index results, and $P(D^+ \mid T^+, V)$ and $P(D^+ \mid T^-, V)$ are the proportions with the target condition among verified patients in each result group. Corrected specificity has the same structure with $D^-$ in place of $D^+$ and the $T^-$ term in the numerator.

The method is valid only if the data are missing at random, meaning that nothing besides the test result, or other recorded variables, influenced who was verified. Arifin and Yusof group the alternatives into inverse probability weighting by the estimated verification probability, as proposed by Alonzo and Pepe, and multiple imputation, all resting on the same assumption. When verification also depends on unrecorded risk factors the bias is nonignorable, and Kosinski and Barnhart illustrate a sensitivity analysis of how far selection depends on disease.

Cronin and Vickers showed that standard corrections are inadequate when there are few false negatives, as is common in screening. In one simulated data set, a corrected area under the curve ranged from 0.550 to 0.852 as false negatives were varied from 0 to 4, and they suggest designing the study so that a large enough random sample of test negatives is verified.

Worked example with a triage test verified mainly in test positives

The numbers in this example are illustrative. A triage test with true sensitivity 0.80 and specificity 0.90 is used in 1,000 people, 200 of whom have the condition. It gives 160 true positives, 40 false negatives, 80 false positives and 720 true negatives, so 240 people test positive and 760 negative. All 240 positives are biopsied, but only a random 25% of negatives are, which is 190 people, made up of 10 false negatives and 180 true negatives.

1. Analyse verified patients only. Sensitivity is 160/170 = 0.941 and specificity is 180/260 = 0.692, so sensitivity is overstated and specificity understated. The diagnostic odds ratio is (160 × 180)/(10 × 80) = 36, exactly the true value of (160 × 720)/(40 × 80) = 36. Among verified negatives 180/190 = 0.947 are disease-free, the same as the true 720/760 = 0.947.

2. Apply the Begg and Greenes correction. With $P(T^+)$ = 0.24 and $P(T^-)$ = 0.76, the corrected estimates recover the true values.

$$\widehat{Se} = \frac{0.24 \times \frac{160}{240}}{0.24 \times \frac{160}{240} + 0.76 \times \frac{10}{190}} = \frac{0.16}{0.16 + 0.04} = 0.80$$

where 160/240 and 10/190 are the shares with the condition among verified positives and negatives.

$$\widehat{Sp} = \frac{0.76 \times \frac{180}{190}}{0.24 \times \frac{80}{240} + 0.76 \times \frac{180}{190}} = \frac{0.72}{0.08 + 0.72} = 0.90$$

where $\widehat{Sp}$ is the corrected specificity and 80/240 and 180/190 are the shares without the condition. The correction works here because test negatives were verified at random; had clinicians chosen the more worrying negatives, it would not.

3. Pass the estimates into a model. A decision tree for 1,000 people tested attaches illustrative consequences of GBP 6,000 and 0.4 QALYs lost to each missed case and GBP 500 to each false positive.

The biased inputs count about three times the true number of false positives yet still understate cost, because they hide about 28 of every 40 missed cases. A model built on them would credit the test with 11.3 fewer QALYs lost per 1,000 tested than it causes, and the overstated sensitivity would make the test look safer for ruling disease out than it is.

Consequences for test evaluation in HTA

Because trials that follow patients from test to outcome are rare, test evaluations rely on linked evidence. The NICE HealthTech programme manual (PMG48), which combines the former Diagnostics Assessment Programme with other NICE programmes, states that the links between test results, care decisions and final outcomes should be specified and justified. Accuracy is the first link, so verification bias enters a health technology assessment at its start. PMG48 also lists differences in reference standard among the likely sources of heterogeneity when test accuracy studies are pooled.

Punglia and colleagues found that in 6,691 men screened with prostate-specific antigen only 705 (11%) had a biopsy, and that correcting for verification raised the area under the ROC curve from 0.69 to 0.86 in men under 60. At a biopsy threshold of 4.1 ng per millilitre they estimated that 82% of cancers in younger men would be missed.

Boundaries with neighbouring biases

Verification bias concerns who receives the reference standard after enrolment, whereas spectrum effects and selection bias concern who enters the study; Rutjes and colleagues reported the highest, though imprecise, estimates in studies of severe cases and healthy controls, which is a selection problem. Error in a reference standard applied to everyone is a separate issue, although the Cochrane Handbook also uses the words verification bias for it, so the term needs reading in context. Reading the reference standard with knowledge of the index result is a form of information bias, which QUADAS-2 assesses with a separate question in its reference standard domain.

Sources

• Arifin WN, Yusof UK. Partial verification bias correction using inverse probability bootstrap sampling for binary diagnostic tests. Diagnostics (Basel). 2022;12(11):2839. https://doi.org/10.3390/diagnostics12112839
• Begg CB, Greenes RA. Assessment of diagnostic tests when disease verification is subject to selection bias. Biometrics. 1983;39(1):207-215. https://doi.org/10.2307/2530820
• Cronin AM, Vickers AJ. Statistical methods to correct for verification bias in diagnostic studies are inadequate when there are few false negatives: a simulation study. BMC Medical Research Methodology. 2008;8:75. https://doi.org/10.1186/1471-2288-8-75
• Kosinski AS, Barnhart HX. Accounting for nonignorable verification bias in assessment of diagnostic tests. Biometrics. 2003;59(1):163-171. https://doi.org/10.1111/1541-0420.00019
• Lijmer JG, Mol BW, Heisterkamp S, et al. Empirical evidence of design-related bias in studies of diagnostic tests. JAMA. 1999;282(11):1061-1066. https://doi.org/10.1001/jama.282.11.1061
• Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy, version 1.0. The Cochrane Collaboration; 2010. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/uploads/Chapter%2010%20-%20Version%201.0.pdf
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48). London: NICE; 2025, last updated 17 December 2025. https://www.nice.org.uk/process/pmg48
• Punglia RS, D'Amico AV, Catalona WJ, Roehl KA, Kuntz KM. Effect of verification bias on screening for prostate cancer by measurement of prostate-specific antigen. New England Journal of Medicine. 2003;349(4):335-342. https://doi.org/10.1056/NEJMoa021659
• Reitsma JB, Rutjes AWS, Whiting P, Vlassov VV, Leeflang MMG, Deeks JJ. Chapter 9: Assessing methodological quality. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy, version 1.0.0. The Cochrane Collaboration; 2009. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/uploads/ch09_Oct09.pdf
• Rutjes AWS, Reitsma JB, Di Nisio M, Smidt N, van Rijn JC, Bossuyt PMM. Evidence of bias and variation in diagnostic accuracy studies. CMAJ. 2006;174(4):469-476. https://doi.org/10.1503/cmaj.050090
• Whiting PF, Rutjes AWS, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529-536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009. Tool and background document: University of Bristol QUADAS team, 2012. https://www.bristol.ac.uk/media-library/sites/quadas/migrated/documents/background-doc.pdf
• Whiting PF, Tomlinson E, Rutjes AWS, et al. QUADAS-3: a revised tool for the quality assessment of diagnostic test accuracy studies. Annals of Internal Medicine. 2026;179(4):548-555. https://doi.org/10.7326/ANNALS-25-02104. Tool version 1.2: https://www.bristol.ac.uk/population-health-sciences/projects/quadas/quadas-3/