ROC Curve

ROC Curve for Choosing Test Thresholds in Economic Evaluation

A receiver operating characteristic (ROC) curve shows every combination of true positive rate and false positive rate that a diagnostic test, screening tool or risk score can deliver as its positivity threshold is moved. Each point on it is in effect a different version of the test, with its own mix of cases found and false alarms, and so its own costs and health effects in an economic evaluation. The curve describes what the test can do, while prevalence and the values placed on correct and incorrect results decide which point is worth using. The area under curve condenses the curve into one measure of ranking and a summary ROC curve pools studies, but a decision model needs a single point.

Sensitivity plotted against the false positive rate

The Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy (chapter 10) defines the ROC curve as the graph of the values of sensitivity and specificity obtained by varying the positivity threshold across all possible values, plotted as sensitivity against one minus specificity. For a test in which higher values point to disease, each threshold gives one point, as follows.

$$TPR(c) = \Pr(X \ge c \mid D^{+}), \quad FPR(c) = \Pr(X \ge c \mid D^{-})$$

where $X$ is the test result, $c$ is the positivity threshold, $D^{+}$ and $D^{-}$ denote people with and without the target condition, $TPR(c)$ is the true positive rate or sensitivity, and $FPR(c)$ is the false positive rate, equal to one minus specificity.

The Handbook notes that the shape of the curve between the all-negative and all-positive corners depends on how much the distributions of results in the two groups overlap. A test that separates them well approaches the upper left corner, and a test whose distributions coincide lies on the rising diagonal. Metz described the curve as a complete empirical description of the decision threshold effect, covering every combination of correct and incorrect decisions a test can make.

Building an empirical curve from study data

With data, the curve is built by sorting people by score and lowering the threshold one value at a time, adding to the true positive count when the next person has the condition and to the false positive count when they do not. Fawcett notes that any curve from a finite sample is a step function, and that tied scores should be processed together, which draws a diagonal segment.

Only the order of the scores matters. Fawcett points out that ROC graphs measure the ability to produce good relative scores, so predicted risks need not be calibrated for the curve to look good, which leaves calibration to be checked separately, and that scores from different model types should not be compared at a common threshold.

Slopes of the curve as likelihood ratios

The slope of the curve carries the information used to revise a probability. Choi showed that the tangent at a point equals the likelihood ratio for the test value at that point, that the slope of the line joining the origin and a point equals the positive likelihood ratio at that cut-off, and that the slope between two points equals the likelihood ratio for a result between the two thresholds. On an empirical curve, the band between thresholds $c_{k}$ and $c_{k+1}$ has the slope below.

$$LR_{k} = \frac{TPR(c_{k}) - TPR(c_{k+1})}{FPR(c_{k}) - FPR(c_{k+1})}$$

where $LR_{k}$ is the likelihood ratio for a result in band $k$, $c_{k}$ is the lower threshold of the band and $c_{k+1}$ the next higher threshold. Multiplying the pre-test odds by $LR_{k}$ gives the post-test odds for a result in that band.

If the categories are ordered by strength of evidence for disease, these slopes fall as the threshold is lowered and the curve bows upwards. Fawcett shows that only points on the convex hull of a set of ROC points can be optimal under any combination of costs and class proportions, so a point lying beneath the hull is never the best choice.

Choosing an operating point with costs and prevalence

The 2011 NICE Diagnostics Assessment Programme manual, since replaced by the NICE HealthTech programme manual (PMG48), stated that each point on the ROC curve could be considered a different version of the test and evaluated separately, and that the optimal cut-off point is usually one of the decision problems explored in an assessment. PMG48 (section 2.1.15) counts using a different threshold for established tests among the ways current care can be delivered differently. Metz linked ROC analysis to cost and benefit analysis, using average net benefit to identify the optimal compromise among kinds of diagnostic error.

A simple version uses net monetary benefit per person tested, compared with testing and treating nobody, as follows.

$$NMB(c) = p \times G \times TPR(c) - (1 - p) \times L \times FPR(c) - k$$

where $p$ is the prevalence in the tested population, $G$ is the net monetary gain from finding and treating a true positive (health gain valued at the threshold, less treatment and confirmation costs), $L$ is the net monetary loss from a false positive, and $k$ is the cost of the test per person, the same at every cut-off. Points with equal net benefit lie on straight lines with the slope below.

$$m = \frac{(1 - p) \times L}{p \times G}$$

where $m$ is the slope of a line of equal net monetary benefit in ROC space. The best cut-off is where the highest such line touches the curve. On an empirical curve, a band of results is worth calling positive when its likelihood ratio exceeds $m$, which is the same as its post-test probability exceeding $L / (G + L)$, the threshold probability of decision curve analysis. Fawcett derives the same slope for iso-performance lines from error costs and class proportions, as do Perkins and Schisterman for a cost-weighted Youden index.

The Youden index, $J$, is the maximum over cut-offs of sensitivity plus specificity minus one, the greatest vertical distance between the curve and the diagonal. Perkins and Schisterman show that it minimises the total of the two misclassification rates when they are weighted equally, and that the point closest to the upper left corner can pick a different cut-off. Equal weighting corresponds to $m = 1$, which holds only when $(1 - p) L = p G$. In screening, with low prevalence and false positives that cost far less than a missed case, that balance is unlikely.

Worked example with a five-band score in two settings

All numbers in this example are illustrative. A triage score places each person in one of five bands, and a validation sample contains 100 people with the target condition and 900 without it. A positive result leads to a confirmatory test costing GBP 500, assumed perfect. A confirmed case gains 0.5 QALYs from treatment costing GBP 3,000, and health is valued at GBP 25,000 per QALY, the lower end of the range in NICE's PMG36. A false positive costs the confirmatory test only.

1. Plot the points. Each TPR is a cumulative count divided by 100, for example (40 + 25) / 100 = 0.65, and each FPR a cumulative count divided by 900, for example (9 + 36) / 900 = 0.05.

2. Read the slopes. Band 4 adds 0.25 to the TPR and 0.04 to the FPR, a slope of 0.25 / 0.04 = 6.25, and band 3 gives 0.20 / 0.10 = 2.0. The slopes fall steadily, so every point lies on the convex hull.

3. Apply the Youden index. Sensitivity plus specificity minus one is 0.39, 0.60, 0.70 and 0.55 for the first four cut-offs and 0 for calling everyone positive, so $J$ = 0.70 picks 3 or above in any setting.

4. Value the results. $G$ = 0.5 × 25,000 − 3,000 − 500 = GBP 9,000 and $L$ = GBP 500. In a specialist clinic with prevalence 0.10, $m$ = (0.90 × 500) / (0.10 × 9,000) = 450 / 900 = 0.5. Bands 5, 4 and 3 have likelihood ratios above 0.5 and band 2 does not, so the best cut-off is 3 or above.

5. Move to primary care. At prevalence 0.02, $m$ = (0.98 × 500) / (0.02 × 9,000) = 490 / 180 = 2.72. Band 3, with a likelihood ratio of 2.0, now falls below the line and the best cut-off is 4 or above. The table gives net monetary benefit per 1,000 people tested, before the test cost.

For example, the clinic figure at 3 or above sets 85 × 9,000 = 765,000 against 135 × 500 = 67,500, leaving 697,500, and the primary care figure at 4 or above sets 13 × 9,000 = 117,000 against 49 × 500 = 24,500, leaving 92,500. Using the Youden cut-off in primary care gives up 92,500 − 79,500 = GBP 13,000 per 1,000 people tested.

6. Check with the threshold probability. Here $L / (G + L)$ = 500 / 9,500 = 0.053. In primary care a band 3 result raises the odds from 0.02 / 0.98 = 0.0204 to 0.0204 × 2.0 = 0.0408, a probability of 0.039, below 0.053, so confirming it is not worthwhile.

7. Interpret. The curve was held fixed between the two settings by assumption, yet the best point moved, because prevalence enters the slope of the decision line and not the curve.

Comparing tests whose curves cross

If one curve lies above another everywhere, the first test offers more true positives at every false positive rate. Curves often cross, and Fawcett notes that a test with the larger area under the curve can perform worse in a particular region of ROC space. Vickers and Elkin note that when a false negative is much more harmful than a false positive, a model with much greater specificity but slightly lower sensitivity could have a higher area yet be a poorer choice for clinical use.

The 2011 NICE manual judged the area under the curve of little use for computing test accuracy in practice or cost effectiveness. A decision model needs the sensitivity and specificity at each test's operating point, so the curve, more than its area, is the useful form of diagnostic accuracy evidence.

Uncertainty and optimism in the chosen cut-off

A cut-off chosen in the same data used to estimate its accuracy flatters the test. Leeflang and colleagues simulated cut-offs chosen by the Youden index and found that the bias was inversely related to sample size, with true sensitivity and specificity of 84% estimated at about 90% in studies of 40 people and 86% in studies of 200.

Sensitivity and specificity at points along one curve are linked, since a cut-off that raises one lowers the other. The NICE manual (PMG36, sections 4.7.18 and 4.7.19) states that because they are usually correlated and may vary with how a test is used or interpreted, point estimates with distributions are not usually appropriate, and asks for correlation between parameters to be reflected in the probabilistic sensitivity analysis.

Misreadings of the ROC curve

Several errors recur when a curve supports a cost-effectiveness case. Most come from treating the curve as a property of the test alone, detached from the setting and the decision.

1. Treating the curve as fixed across settings. Fawcett notes that ROC curves do not change with the proportion of positive and negative cases. The 2011 NICE manual, however, warns that laboratory, operator and machine differences can shift the effective cut-off or move a test below its ROC curve.
2. Reading accuracy as usefulness. Zweig and Campbell describe the ROC plot as a pure index of accuracy, to be distinguished from the practical value of the information.

Sources

• Choi BC. Slopes of a receiver operating characteristic curve and likelihood ratios for a diagnostic test. American Journal of Epidemiology. 1998;148(11):1127-1132. https://doi.org/10.1093/oxfordjournals.aje.a009592
• Fawcett T. An introduction to ROC analysis. Pattern Recognition Letters. 2006;27(8):861-874. https://doi.org/10.1016/j.patrec.2005.10.010
• Leeflang MM, Moons KG, Reitsma JB, Zwinderman AH. Bias in sensitivity and specificity caused by data-driven selection of optimal cutoff values: mechanisms, magnitude, and solutions. Clinical Chemistry. 2008;54(4):729-737. https://doi.org/10.1373/clinchem.2007.096032
• Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 1.0. The Cochrane Collaboration; 2010. Section 10.2.5. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/uploads/Chapter%2010%20-%20Version%201.0.pdf
• Metz CE. Basic principles of ROC analysis. Seminars in Nuclear Medicine. 1978;8(4):283-298. https://doi.org/10.1016/s0001-2998(78)80014-2
• National Institute for Health and Clinical Excellence. Diagnostics Assessment Programme manual. Manchester: NICE; December 2011. Sections 14.2.1 and 14.3.2. https://www.nice.org.uk/Media/Default/About/what-we-do/NICE-guidance/NICE-diagnostics-guidance/Diagnostics-assessment-programme-manual.pdf
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48). London: NICE; 14 July 2025, last updated 17 December 2025. Section 2.1.15. https://www.nice.org.uk/process/pmg48/chapter/methods-for-guidance-produced-in-the-nice-healthtech-programme
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 31 January 2022, last updated 13 October 2026. Sections 4.7.15, 4.7.18 and 4.7.19. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation-2
• Perkins NJ, Schisterman EF. The inconsistency of "optimal" cutpoints obtained using two criteria based on the receiver operating characteristic curve. American Journal of Epidemiology. 2006;163(7):670-675. https://pmc.ncbi.nlm.nih.gov/articles/PMC1444894/
• Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. https://pmc.ncbi.nlm.nih.gov/articles/PMC2577036/
• Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clinical Chemistry. 1993;39(4):561-577. https://doi.org/10.1093/clinchem/39.4.561