Diagnostic Accuracy

Diagnostic Accuracy and the Value of a Test

Most economic evaluations of a diagnostic, screening or triage test start from how often the test is right and in which direction it is wrong, because true and false results send people down care pathways with different costs and health outcomes. The measures of accuracy answer different questions, some conditional on whether a person has the target condition and others on what the test said. Some of these measures depend on prevalence, accuracy studies are reported and appraised with STARD and QUADAS, and their results are pooled across studies with hierarchical models. Accuracy alone does not establish value, as two triage tests show, the test with fewer correct results delivering more net monetary benefit in one setting and less in another.

The two-by-two table and what each measure conditions on

An accuracy study cross-classifies each person's index test result against a reference standard, which STARD 2015 defines as the best available method for establishing the presence or absence of the target condition. The Cochrane Handbook for Diagnostic Test Accuracy Reviews separates measures conditional on disease status, sensitivity and specificity, from measures conditional on the test result, the positive predictive value and negative predictive value. Each is a proportion of one row or one column of the table, as follows.

$$Se = \frac{TP}{TP + FN}, \quad Sp = \frac{TN}{TN + FP}, \quad PPV = \frac{TP}{TP + FP}, \quad NPV = \frac{TN}{TN + FN}$$

where $TP$ and $FN$ are people with the target condition whom the index test calls positive and negative, $TN$ and $FP$ are people without it whom the test calls negative and positive, $Se$ is sensitivity, $Sp$ is specificity, and $PPV$ and $NPV$ are the predictive values.

A likelihood ratio says how much more likely a result is in people with the condition than in people without it. Multiplying pre-test odds by it gives post-test odds, and so converts a pre-test probability into a post-test probability. The diagnostic odds ratio is the ratio of the two likelihood ratios, as below.

$$LR^{+} = \frac{Se}{1 - Sp}, \quad LR^{-} = \frac{1 - Se}{Sp}, \quad DOR = \frac{LR^{+}}{LR^{-}} = \frac{TP \times TN}{FP \times FN}$$

where $LR^{+}$ and $LR^{-}$ are the positive and negative likelihood ratios and $DOR$ is the diagnostic odds ratio. The Handbook notes that it has little direct clinical relevance, since a clinician is usually interested in the sum of false negatives and false positives while it reflects their product, but it remains important in meta-analytic models.

Why predictive values and the proportion correct move with prevalence

Sensitivity and specificity are defined within the diseased and non-diseased groups, so prevalence does not enter them algebraically. Predictive values mix the two groups. Bayes' theorem gives the positive predictive value as follows.

$$PPV = \frac{Se \times p}{Se \times p + (1 - Sp)(1 - p)}$$

where $p$ is the prevalence where the test is used and the other terms are as above. As prevalence falls, false positives from the larger disease-free group make up more of the positive results. The criterion validity page shows what this does to a cost estimate built on a claims algorithm validated in a sample enriched with cases.

The share of results that are correct, often reported simply as "accuracy", is a prevalence-weighted average, as below.

$$Acc = p \times Se + (1 - p) \times Sp$$

where $Acc$ is the proportion of correct results. At low prevalence it is dominated by specificity. At 5% prevalence a rule that calls everyone negative scores 0.95. NICE's Diagnostics Assessment Programme manual (2011, since replaced by the NICE HealthTech programme manual, PMG48, 2025) set this definition aside because a test may be incorrect in more than one way and because it depends on prevalence, and used "test accuracy" for any measure of correctness, including sensitivity, specificity and predictive values.

Sensitivity and specificity are not constants either. STARD 2015 states that they are not fixed test properties, and that the relative numbers of false positive and false negative results vary across settings with how patients present and which tests they have already had, and with the definition of the reference standard. An estimate used in a model therefore needs to match the population and the position in the pathway that the model represents.

Thresholds, ROC curves and the numbers a model needs

Most tests produce a measurement or graded reading that becomes positive or negative at a chosen threshold. The Handbook calls this dependence a fundamental aspect of diagnostic test evaluation, since moving the threshold raises one of sensitivity and specificity and lowers the other. The ROC curve traces the pair across all thresholds, and the area under the curve summarises how well the test ranks people with the condition above people without it.

A decision model follows people through the result they receive, so it needs the sensitivity and specificity at the threshold used in practice; the 2011 NICE manual called the area under the ROC curve of little use for computing test accuracy in practice or cost effectiveness. Choosing the threshold trades the consequences of the two errors, the question decision curve analysis and cost-effectiveness analysis address.

Reporting and appraising accuracy studies with STARD 2015 and QUADAS

STARD 2015 lists 30 items that should appear in every report of a diagnostic accuracy study and expects a participant flow diagram in all reports. For a modeller the most useful items are the intended use and clinical role of the index test, which places the estimate in a pathway, and the cross tabulation of index test results by reference standard results, which allows every measure to be recomputed.

Risk of bias in primary studies is judged with the QUADAS tools. QUADAS-2, published in 2011, has four domains, covering patient selection, index test, reference standard, and flow and timing. Each is judged for risk of bias and the first three also for applicability. The QUADAS team at the University of Bristol now recommends QUADAS-3, published in 2026, whose domains are Participants, Index Test, Target Condition and Analysis, the last replacing Flow and Timing. It makes its judgements for individual accuracy estimates rather than whole studies and introduces an ideal test accuracy trial to aid the assessment.

Pooling accuracy across studies with bivariate and HSROC models

Chapter 10 of the Cochrane Handbook (version 1.0) explains that meta-analysis of accuracy must handle two statistics at once, allow for the trade-off between them when studies use different thresholds, and expect heterogeneity, and it recommends hierarchical random effects models. The bivariate model of Reitsma and colleagues, in the form the Handbook describes, treats each study's counts as binomial and gives the logit sensitivities and specificities a bivariate normal distribution across studies, with means, between-study variances and a correlation; Reitsma originally approximated the binomial within-study distributions by normal ones. The hierarchical summary ROC (HSROC) model of Rutter and Gatsonis lets test stringency and accuracy vary across studies and yields a summary ROC curve, and the Handbook notes that the two models are mathematically equivalent without covariates, as Harbord and colleagues showed. A single summary point pooled across mixed thresholds refers to no threshold in particular, which the Handbook says must be avoided.

The Handbook advises against shortcuts. Pooling sensitivity and specificity separately may underestimate accuracy, pooling likelihood ratios separately can produce impossible estimates, and predictive values should not be pooled because they depend on prevalence. PMG48 points to the NICE Decision Support Unit's technical support document 25 for these methods. For the model, PMG36, to which PMG48 refers for economic methods, states that because sensitivity and specificity are usually correlated and may vary with how a test is used, point estimates with distributions are not usually appropriate, and that correlation should be reflected in the probabilistic sensitivity analysis. Joint draws of the summary logit sensitivity and specificity, with their correlation, represent uncertainty about average accuracy; when the decision setting resembles a new study and between-study heterogeneity matters, the predictive distribution applies instead, shown by the Handbook's 95% prediction region for a future study.

Why accuracy alone does not establish the value of a test

Accuracy is an intermediate step towards health outcomes. Fryback and Thornbury's hierarchy of diagnostic efficacy places accuracy at level 2 of 6, below effects on diagnostic thinking, management, patient outcomes and societal costs and benefits, and holds that each lower level is necessary but not sufficient for the next. PMG48 notes that most outcomes follow from treatment started, modified or stopped, that trials of tests are unusual, and that a linked evidence model should specify and justify the links between test results, care decisions and final outcomes.

Vickers, Van Calster and Steyerberg note that sensitivity, specificity and the area under the curve do not answer whether a test should be used in practice. Ferrante di Ruffano and colleagues place testing at the start of a test-treatment pathway, argue that better accuracy helps patients only if it changes diagnoses and management, and caution against using accuracy as a surrogate for patient health. Lord, Irwig and Simes argue that accuracy studies suffice when a new test is safer or more specific with similar sensitivity, but that a more sensitive test finds extra cases to which trials of treatment in cases found by the old test may not apply.

Worked example with two triage tests in two settings

The figures below are illustrative. Two triage tests, each costing £40, are compared for a condition in which every positive result goes on to a confirmatory test costing £400, assumed perfect. Confirmed cases are treated at £3,000 and gain 1 QALY, net of treatment harms, with no discounting or time horizon modelled; missed cases and true negatives fare as they would without testing, and a false positive costs the £400 only, with no health loss or anxiety. Health is valued at £25,000 per QALY, the lower end of the range in NICE's PMG36. The table covers 1,000 people at 5% prevalence.

1. Read the accuracy measures. Test B leads on every summary in the table, although its 0.952 barely beats the 0.950 of calling everyone negative.

2. Value each result. Against no testing, a true positive is worth $G$ = 25,000 − 3,000 − 400 = £21,600 and a false positive costs $L$ = £400. The net monetary benefit per person tested is given by the expression below.

$$NMB = p \times Se \times G - (1 - p)(1 - Sp) \times L - c$$

where $NMB$ is net monetary benefit per person against no testing, $c$ is the cost of the triage test and the other terms are as defined above. For test A at 5% prevalence the result is as follows.

$$NMB_A = 0.05 \times 0.96 \times 21600 - 0.95 \times 0.24 \times 400 - 40 = 1036.8 - 91.2 - 40 = 905.6$$

where $NMB_A$ is in pounds per person. For test B the result is as follows.

$$NMB_B = 0.05 \times 0.80 \times 21600 - 0.95 \times 0.04 \times 400 - 40 = 864 - 15.2 - 40 = 808.8$$

where $NMB_B$ is in pounds per person. Test A is worth £96.80 more per person. Per 1,000 people its 8 extra cases are worth 8 × 21,600 = £172,800 and its 190 extra false positives cost 190 × 400 = £76,000.

3. Move to a lower-prevalence setting. Sensitivity and specificity are held fixed here purely for illustration, although the STARD point above warns that they may change between settings. At 2% prevalence the same formula gives £280.64 for test A (414.72 − 94.08 − 40) and £289.92 for test B (345.6 − 15.68 − 40), so test B is now preferred.

4. Find the break-even prevalence. With equal test costs, the difference between the tests is as follows.

$$\Delta NMB = p (Se_A - Se_B) G - (1 - p)(Sp_B - Sp_A) L = 3456p - 80(1 - p)$$

where $\Delta NMB$ is the net monetary benefit of test A minus that of test B per person. It is zero at $p = 80 / 3536 \approx 0.023$, so test A is preferred above a prevalence of about 2.3%.

5. Interpret. The test with fewer correct results and the lower diagnostic odds ratio is the better buy where the condition is more common, because finding a case is worth 54 times the cost of an unnecessary confirmation. Written as a risk, that exchange rate is 400 / 22,000, about 1.8%, which plays the role of the threshold probability in decision curve analysis, with the £40 triage cost outside it as a test harm. The two numbers answer different questions. About 1.8% is the risk at which referral for confirmation is worthwhile, while about 2.3% is the prevalence at which the two tests tie. The result also assumes, as Lord and colleagues warn may not hold, that the extra cases test A finds gain as much from treatment as the others.

Misreadings that distort economic models

A few errors recur when accuracy evidence enters a model or an HTA submission. Each distorts the costs and QALYs that follow.

• Ranking tests by the proportion correct or a single summary. Neither the proportion correct, the diagnostic odds ratio nor an overall score such as the Brier score carries the relative cost of the two errors, as the example shows.
• Treating accuracy as the outcome. Accuracy affects health only through decisions and treatment, so the link to final outcomes has to be modelled or measured.
• Ignoring the reference standard. Accuracy is agreement with the reference standard, and the 2011 NICE manual notes that this standard is often imperfect, so estimates can differ from true accuracy.

Sources

• Bossuyt PM, Reitsma JB, Bruns DE, Gatsonis CA, Glasziou PP, Irwig L, et al; STARD Group. STARD 2015: an updated list of essential items for reporting diagnostic accuracy studies. BMJ. 2015;351:h5527. https://pmc.ncbi.nlm.nih.gov/articles/PMC4623764/
• Ferrante di Ruffano L, Hyde CJ, McCaffery KJ, Bossuyt PM, Deeks JJ. Assessing the value of diagnostic tests: a framework for designing and evaluating trials. BMJ. 2012;344:e686. https://www.bmj.com/content/344/bmj.e686
• Fryback DG, Thornbury JR. The efficacy of diagnostic imaging. Medical Decision Making. 1991;11(2):88-94. https://doi.org/10.1177/0272989X9101100203
• Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007;8(2):239-251. https://doi.org/10.1093/biostatistics/kxl004
• Lord SJ, Irwig L, Simes RJ. When is measuring sensitivity and specificity sufficient to evaluate a diagnostic test, and when do we need randomized trials? Annals of Internal Medicine. 2006;144(11):850-855. https://doi.org/10.7326/0003-4819-144-11-200606060-00011
• Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 1.0. The Cochrane Collaboration; 2010. Sections 10.1.3, 10.2.3, 10.2.4, 10.4.1, 10.4.2 and 10.5.2. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/uploads/Chapter%2010%20-%20Version%201.0.pdf
• National Institute for Health and Clinical Excellence. Diagnostics Assessment Programme manual. Manchester: NICE; December 2011. Sections 2.1, 14.2.1. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-diagnostics-guidance/Diagnostics-assessment-programme-manual.pdf
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48). London: NICE; 14 July 2025, updated 17 December 2025. Sections 2.2.15, 2.3.1 and 2.3.14. https://www.nice.org.uk/process/pmg48
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 2022, last updated 13 October 2026. Sections 4.7.18, 4.7.19 and 6.3.4 to 6.3.7. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2
• Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology. 2005;58(10):982-990. https://doi.org/10.1016/j.jclinepi.2005.02.022
• Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine. 2001;20(19):2865-2884. https://doi.org/10.1002/sim.942
• Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. https://pmc.ncbi.nlm.nih.gov/articles/PMC4724785/
• Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, Leeflang MM, Sterne JA, Bossuyt PM; QUADAS-2 Group. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Annals of Internal Medicine. 2011;155(8):529-536. https://doi.org/10.7326/0003-4819-155-8-201110180-00009
• Whiting PF, Tomlinson E, Rutjes AWS, Davenport CF, Yang B, Westwood ME, et al. QUADAS-3: a revised tool for the quality assessment of diagnostic test accuracy studies. Annals of Internal Medicine. 2026;179(4):548-555. https://doi.org/10.7326/ANNALS-25-02104. Tool and recommendation: University of Bristol QUADAS team, https://www.bristol.ac.uk/population-health-sciences/projects/quadas/quadas-3/