Summary ROC Curve

Summary ROC Curve in Diagnostic Test Accuracy Meta-Analysis

Studies in a diagnostic accuracy review rarely sit on one point in ROC space, partly because they used different positivity thresholds and partly because accuracy varies between settings. A summary curve fitted through those points describes the sensitivity the test achieves at each level of specificity, on average across studies. In health technology assessment the curve is rarely the model input itself, because a decision model follows people through test results at one operating point, so the analyst must choose which output to carry forward. Hierarchical models have replaced the older Moses-Littenberg regression, and the NICE HealthTech programme manual (PMG48) points to the Decision Support Unit's Technical Support Document 25 (TSD 25) for these methods.

Study points, threshold effects and the trade-off across studies

A single-study ROC curve traces sensitivity against the false positive fraction (one minus specificity) as one test's threshold moves. TSD 25 explains that, for a test where higher values indicate disease, raising the cut-off increases specificity at the cost of sensitivity, and that tests read by people, such as imaging or lateral flow devices, have implicit thresholds because readers differ in how strict they are. Across studies this produces a threshold effect, and TSD 25 notes that sensitivity and specificity can typically be expected to be negatively correlated across studies.

Chapter 10 of the Cochrane Handbook for diagnostic test accuracy reviews describes the summary ROC plot, in which each study appears as a single sensitivity and specificity point. A summary curve, or a summary point with confidence and prediction regions, can be added. The Handbook frames the choice as one between an average operating point at a common threshold and an expected ROC curve across many thresholds.

The Moses-Littenberg regression and why it was replaced

One of the earliest and most widely used methods was published by Moses, Shapiro and Littenberg in 1993. It turns each study's pair of rates into two logit-scale quantities and fits a straight line between them.

$$D_i = \operatorname{logit}(TPR_i) - \operatorname{logit}(FPR_i), \qquad S_i = \operatorname{logit}(TPR_i) + \operatorname{logit}(FPR_i), \qquad D_i = a + b S_i$$

where $TPR_i$ and $FPR_i$ are the true and false positive rates of study $i$, $D_i$ is its log diagnostic odds ratio, $S_i$ is a proxy for its threshold that rises as more people test positive, and $a$ and $b$ are the intercept and slope of the regression. Moses and colleagues added one half to each cell count, fitted a resistant or equally weighted least-squares line, and back-transformed it to ROC space. Rearranging the line gives the curve in ROC coordinates.

$$\operatorname{logit}(TPR) = \frac{a}{1 - b} + \frac{1 + b}{1 - b}\operatorname{logit}(FPR)$$

where the symbols are as above. When $b = 0$ the curve is symmetric and the diagnostic odds ratio is the same at every point. When $|b| > 1$ the slope turns negative, so sensitivity falls as the false positive rate rises, which the Cochrane Handbook calls an unintuitive property often caused by an influential outlying study.

The Handbook treats the method as closer to a fixed effect model, because it gives no estimate of between-study heterogeneity, notes that it ignores sampling error in $S$, and says it should not be used for confidence intervals or for testing differences between subgroups. RevMan offered it for exploratory analysis only. Reitsma and colleagues noted that working on the diagnostic odds ratio removes the threshold effect but loses clinical information, and TSD 25 discourages it now that exact binomial methods are available.

The hierarchical SROC curve and the bivariate model

Two hierarchical models replaced the regression. Both model each study's 2×2 counts as binomial at the lower level and add random study effects at the upper level for between-study heterogeneity, as in a random effects model.

The model of Rutter and Gatsonis gives each study its own ROC curve with a cutpoint parameter $\theta_i$, an accuracy parameter $\alpha_i$ (the log diagnostic odds ratio) and a scale parameter $\beta$, which sets the curve's asymmetry, shared by all studies, because one point per study cannot identify each study's shape. Cutpoint and accuracy are normally distributed across studies, so the model has five parameters without covariates, and unlike the regression its curve cannot show sensitivity falling as the false positive fraction rises. The bivariate model of Reitsma and colleagues instead gives the logit sensitivity and logit false positive fraction a bivariate normal distribution across studies, again with five parameters. Harbord and colleagues showed that the two models are identical when no covariates are fitted, so TSD 25 fits the bivariate model and derives the curve from it.

$$\operatorname{logit}(Se) = e^{-\beta}\operatorname{logit}(FPF) + e^{-\beta/2}\Lambda, \qquad \beta = \ln(\sigma_2/\sigma_1), \qquad \Lambda = \sqrt{\sigma_2/\sigma_1}\,m_1 - \sqrt{\sigma_1/\sigma_2}\,m_2$$

where $Se$ is sensitivity, $FPF$ is the false positive fraction, $m_1$ and $m_2$ are the mean logit sensitivity and logit false positive fraction, $\sigma_1$ and $\sigma_2$ are their between-study standard deviations, $\beta$ is the scale parameter and $\Lambda$ is the mean accuracy. Because $e^{-\beta}$ is always positive, sensitivity rises with the false positive fraction along the whole curve, and substituting $m_2$ for $\operatorname{logit}(FPF)$ returns $m_1$, so the curve passes through the summary point. TSD 25 adds that a between-study correlation of one would put every study on the same ROC curve, and that the Rutter and Gatsonis curve is only one of several curves compatible with the bivariate model, though the most used and the one that treats sensitivity and specificity equally. It recommends against extending the curve beyond the observed range of the data.

Choosing between a summary curve and a summary point

The Cochrane Handbook warns that pooling studies that use different thresholds into one sensitivity and specificity gives an estimate for a notional average threshold, which is clinically unhelpful and must be avoided. Its two strategies are a summary point at each common threshold or a summary curve using one threshold per study. TSD 25 recommends a single point where threshold effects are minimal, such as a common explicit threshold or a clearly dichotomous result, and calls the curve an informative summary where thresholds are implicit.

Neither the bivariate nor the HSROC model uses the threshold values themselves, so TSD 25 notes that it is not possible to know which point on the curve a given threshold corresponds to. Where studies report explicit numerical thresholds, often several each, TSD 25 describes a multiple thresholds model that pools all the data and estimates sensitivity and specificity at any cut-off.

What a decision model takes from the meta-analysis

The NICE HealthTech programme manual (PMG48, section 2.2.15) notes that meta-analysis of test accuracy is complicated by the correlation between sensitivity and specificity and by cut-offs that differ between studies, and that the method chosen should be justified. Section 2.3.14 asks for a linked evidence model whose links between test results, care decisions and final outcomes are specified and justified. The technology appraisal manual (PMG36, sections 4.7.18 and 4.7.19) states that separate point estimates with distributions are not usually appropriate for correlated accuracy parameters and asks for the correlation to be reflected in the probabilistic sensitivity analysis, as on the diagnostic accuracy page.

TSD 25 explains that plugging summary values into a deterministic model fails to carry this correlation, and recommends exporting correlated posterior samples into the probabilistic model. It lists the outputs a model could use, including the random effects means, the predictive distribution for a new study, predictions from a model with covariates, a study-specific estimate and the whole random effects distribution. It describes the means as the most common choice but finds it hard to envisage a setting in which test accuracy truly sits at them. A model can also be evaluated at every point on the curve to find the operating point with the highest net benefit, an approach demonstrated by Sutton and colleagues, but with implicit thresholds there is no numerical value at which that point could be implemented in practice. TSD 25 also shows that the optimal threshold depends heavily on prevalence.

Worked example converting a bivariate fit into a curve and an operating point

The numbers below are illustrative. A bivariate meta-analysis of a hypothetical imaging test gives a mean logit sensitivity of 1.5, a mean logit false positive fraction of −2.0, between-study standard deviations of 0.8 and 1.2, and a correlation of 0.5, which shapes the prediction region but does not enter the curve or the summary point.

1. Summary point. Back-transforming gives sensitivity $Se = 1/(1 + e^{-1.5}) = 0.8176$ and false positive fraction $FPF = 1/(1 + e^{2.0}) = 0.1192$, so specificity is 0.8808.

2. Curve parameters. The scale parameter is $\beta = \ln(1.2/0.8) = 0.405$, and the mean accuracy follows.

$$\Lambda = 1.2247 \times 1.5 + 0.8165 \times 2.0 = 3.470$$

where 1.2247 and 0.8165 are $\sqrt{1.5}$ and its reciprocal. The slope is $e^{-0.405} = 0.6667$ and the intercept is $\Lambda/\sqrt{1.5} = 2.833$, so the curve is $\operatorname{logit}(Se) = 0.6667\operatorname{logit}(FPF) + 2.833$.

3. Points along the curve. At a false positive fraction of 0.05 the logit is −2.944, giving a logit sensitivity of 0.870 and a sensitivity of 0.7048. At 0.20 the logit is −1.386, giving 1.909 and a sensitivity of 0.8709. At 0.1192 it returns 0.8176, the summary point.

4. Value at each operating point. Following the simplified decision tree in TSD 25, each person with the disease who tests positive gains a net £4,000 from earlier treatment, each false positive costs £500 in further investigation, and the test costs £40. Incremental net benefit per person tested, against no testing, is as follows.

$$INB = \pi \times G \times Se - (1 - \pi) \times L \times FPF - C$$

where $\pi$ is prevalence, $G$ is the gain from earlier detection, $L$ is the cost of a false positive and $C$ is the cost of the test, all in pounds.

At 10% prevalence the summary point gives $INB = 0.10 \times 4000 \times 0.8176 - 0.90 \times 500 \times 0.1192 - 40 = 233.40$ pounds per person tested. At 3% prevalence the stricter point gives $INB = 0.03 \times 4000 \times 0.7048 - 0.97 \times 500 \times 0.05 - 40 = 20.33$ pounds.

5. Interpretation. The summary point is best at 10% prevalence, but at 3% it barely breaks even and the stricter point is best; the two give equal net benefit at a prevalence of about 7.1%. A model using only the summary point would miss this, although the stricter point helps only if readers can be calibrated to work there.

Misreadings of the summary curve

The curve is an average across studies, not the accuracy any one centre will achieve, for which the prediction region is the better guide. The Cochrane Handbook discourages quoting the point where sensitivity equals specificity, known as Q*, because it misleads when the curve is asymmetric, and does not routinely use univariate heterogeneity statistics, since with threshold variation what matters is how close studies lie to the curve.

The methods in TSD 25 assume an error-free reference standard in every study, so an imperfect reference standard calls for other approaches, such as the latent class models the document mentions but does not cover. A curve fitted where thresholds barely vary is hard to estimate because the points cluster.

Sources

• Derezea E, Ades AE, Rogers G, Sutton AJ, Cooper NJ, Hamilton J, Jones HE. Evidence synthesis of diagnostic test accuracy for decision making. NICE DSU Technical Support Document 25. Sheffield: Decision Support Unit, ScHARR, University of Sheffield; November 2024. Sections 2 to 7. https://doi.org/10.15131/shef.data.33851305
• Harbord RM, Deeks JJ, Egger M, Whiting P, Sterne JA. A unification of models for meta-analysis of diagnostic accuracy studies. Biostatistics. 2007;8(2):239-251. https://doi.org/10.1093/biostatistics/kxl004
• Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y. Chapter 10: Analysing and presenting results. In: Deeks JJ, Bossuyt PM, Gatsonis C, editors. Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Version 1.0. The Cochrane Collaboration; 2010. Sections 10.2.6, 10.3.1, 10.4 and 10.5. https://methods.cochrane.org/sites/methods.cochrane.org.sdt/files/uploads/Chapter%2010%20-%20Version%201.0.pdf
• Moses LE, Shapiro D, Littenberg B. Combining independent studies of a diagnostic test into a summary ROC curve: data-analytic approaches and some additional considerations. Statistics in Medicine. 1993;12(14):1293-1316. https://doi.org/10.1002/sim.4780121403
• National Institute for Health and Care Excellence. NICE HealthTech programme manual (PMG48). London: NICE; 14 July 2025, last updated 17 December 2025. Sections 2.2.15, 2.3.14 and 2.3.15. https://www.nice.org.uk/process/pmg48
• National Institute for Health and Care Excellence. NICE technology appraisal and highly specialised technologies guidance: the manual (PMG36). London: NICE; 31 January 2022, last updated 13 October 2026. Sections 4.7.18 and 4.7.19. https://www.nice.org.uk/process/pmg36/chapter/economic-evaluation
• Reitsma JB, Glas AS, Rutjes AW, Scholten RJ, Bossuyt PM, Zwinderman AH. Bivariate analysis of sensitivity and specificity produces informative summary measures in diagnostic reviews. Journal of Clinical Epidemiology. 2005;58(10):982-990. https://doi.org/10.1016/j.jclinepi.2005.02.022
• Rutter CM, Gatsonis CA. A hierarchical regression approach to meta-analysis of diagnostic test accuracy evaluations. Statistics in Medicine. 2001;20(19):2865-2884. https://doi.org/10.1002/sim.942
• Sutton AJ, Cooper NJ, Goodacre S, Stevenson M. Integration of meta-analysis and economic decision modeling for evaluating diagnostic tests. Medical Decision Making. 2008;28(5):650-667. https://doi.org/10.1177/0272989X08324036