Decision Curve Analysis

Decision Curve Analysis of Risk Models and Tests Across Threshold Probabilities

A prediction model or diagnostic test earns a place in care, and in a health technology assessment, only if acting on its results beats intervening for everyone or for no one. Decision curve analysis checks this in a validation dataset, using the risk at which an intervention would be offered as the exchange rate between true cases found and unnecessary interventions. Its net benefit formula treats the threshold as a harm-benefit trade-off and compares each model with treating everyone and treating no one, and a worked example below reuses the two admission-risk models from the Brier score page. The method is simpler than a full cost-effectiveness analysis, and its net benefit is a different quantity from the economic net benefit.

Why accuracy statistics leave the decision open

Prediction models are usually judged by the area under the ROC curve, by sensitivity and specificity, or by calibration. None of these says how good a model must be before it is worth using, because none carries the consequences of a false negative compared with a false positive. Vickers and Elkin noted that where missing disease is much worse than overtreatment, a model with much greater specificity but slightly lower sensitivity could have the higher AUC and still be the poorer choice. Assel, Sjoberg and Vickers found that the Brier score ranked tests and models wrongly in each scenario they examined, while net benefit always favoured the preferable one.

Full decision analysis handles consequences, but it needs data that a validation set does not hold, such as costs, quality-adjusted life years (QALYs) and health state utilities, and usually a binary result, so continuous risks must be cut at chosen points. Vickers, Van Calster and Steyerberg describe net benefit as a simple type of decision analysis. It keeps the decision-analytic logic while needing only predictions and observed outcomes, so it can judge the clinical utility of binary tests and continuous models alike.

The threshold probability as an exchange rate between harms

The method starts from one treatment decision made without knowing whether the patient has the event. The threshold probability is the risk at which the expected value of treating equals that of not treating. Vickers and Elkin solve the decision tree as follows.

$$p_t a + (1 - p_t) b = p_t c + (1 - p_t) d \quad \Rightarrow \quad \frac{a - c}{d - b} = \frac{1 - p_t}{p_t}$$

where $p_t$ is the threshold probability, and $a$, $b$, $c$ and $d$ are the values of a true positive, a false positive, a false negative and a true negative. The difference $a - c$ is the harm of withholding treatment from a patient who would have benefited, and $d - b$ is the harm of treating a patient unnecessarily. A threshold of 10% has odds of 1 to 9, so a missed case is judged nine times worse than an unnecessary intervention, or a clinician would intervene in at most 10 patients to reach one true case.

Treatment effect enters through the threshold. In their 2019 guide to interpreting decision curves, Vickers, van Calster and Steyerberg give the example of a preventive drug that cuts the relative risk of a cardiac event by 10% and causes a serious side effect in 1% of those treated. If the two events are judged equally harmful, the minimum threshold is 10%, and a 20% relative risk reduction would lower it to 5%.

Calculating net benefit at one threshold

At a chosen threshold, every patient whose predicted risk is at or above $p_t$ counts as positive and would receive the intervention. False positives are weighted by the odds at the threshold, so that both kinds of result are expressed in units of true positives. Net benefit is calculated as follows.

$$NB = \frac{TP}{n} - \frac{FP}{n} \times \frac{p_t}{1 - p_t}$$

where $NB$ is net benefit, $TP$ is the number of patients classed positive who have the event, $FP$ is the number classed positive who do not, $n$ is the number of patients and $p_t$ is the threshold probability. A net benefit of 0.07 is equivalent to intervening in 7 patients per 100, all of whom have the event, and in no one else. The maximum is the prevalence of the event, and there is no lower limit.

Treating no one gives zero net benefit at every threshold. Treating everyone classes all patients as positive, which gives the following.

$$NB_{all} = \pi - (1 - \pi) \times \frac{p_t}{1 - p_t}$$

where $NB_{all}$ is the net benefit of intervening for every patient and $\pi$ is the prevalence in the sample. This line crosses zero where $p_t$ equals the prevalence. A model is of value at a threshold only if it beats both defaults, and the decision curve plots net benefit against $p_t$ for the model, any competitors and the two defaults.

When intervening for everyone is current practice, the gain can be restated as unnecessary interventions avoided, equivalent to a reduction with no more cases missed, as below.

$$\text{Avoided per 100} = (NB_{model} - NB_{all}) \times \frac{1 - p_t}{p_t} \times 100$$

where $NB_{model}$ is the net benefit of the model and $NB_{all}$ that of treating everyone at the same threshold. If obtaining the model's inputs is costly or harmful, a test harm in units of true positives is subtracted at every threshold, for example 1/30, or about 0.033, if clinicians would scan at most about 30 men to find one high-grade cancer with a perfect test.

Reading a decision curve against a plausible range of thresholds

A decision curve is read from the horizontal axis first. The analyst sets in advance the range of thresholds that clinicians and patients could reasonably hold; in a prostate biopsy example of Vickers, Van Calster and Steyerberg it was 5% to 20%. Kerr and colleagues describe the curve as read vertically, giving the net benefit of each strategy at a threshold chosen on clinical and value grounds. A model whose curve lies above every alternative across the range is supported whatever the exact threshold, while crossing curves mean the choice depends on preferences.

Worked example with two admission-risk models across a range of thresholds

The figures below are illustrative and reuse the validation sample from the Brier score page. One risk model for one-year hospital admission places 100 patients in three groups, and a second model, developed in a higher-risk setting, ranks them the same way with higher risks. Patients classed positive would be offered an admission-prevention intervention, and commissioners judge that reasonable thresholds lie between 15% and 40%.

1. Net benefit at 15%. The weight on false positives is 0.15 / 0.85, about 0.1765. Model A classes the medium and high groups as positive, giving 22 true positives (9 + 13) and 28 false positives (21 + 7). Model B gives every group a risk of at least 20%, so it classes all 100 patients positive, exactly as treating everyone does.

$$NB_A = \frac{22}{100} - \frac{28}{100} \times \frac{0.15}{0.85} = 0.2200 - 0.0494 = 0.1706$$

where $NB_A$ is the net benefit of model A at 15%. Treating everyone, and so model B, gives 0.26 − 0.74 × 0.1765, about 0.1294.

2. Net benefit at 25% and 40%. At 25% the weight is 1/3 and both models class the medium and high groups as positive, so both score 0.22 − 0.28 / 3, about 0.1267, while treating everyone scores about 0.0133. At 40% model A classes only the high group as positive, while model B still places the medium group, at a predicted 45%, above the threshold.

$$NB_A = \frac{13}{100} - \frac{7}{100} \times \frac{0.40}{0.60} = 0.1300 - 0.0467 = 0.0833$$

where $NB_A$ is now the net benefit of model A at 40%. The corresponding figure for model B is as follows.

$$NB_B = \frac{22}{100} - \frac{28}{100} \times \frac{0.40}{0.60} = 0.2200 - 0.1867 = 0.0333$$

where $NB_B$ is the net benefit of model B at 40%. The results are shown in the table below.

3. Interventions avoided at 15%. Compared with treating everyone, model A avoids the following number of interventions.

$$(0.1706 - 0.1294) \times \frac{0.85}{0.15} \times 100 \approx 23.3$$

where the result is net unnecessary interventions avoided per 100 patients. Model A spares 46 of the 50 low-risk patients who would not have been admitted, less 4 missed admissions weighted by about 5.67 each.

4. Interpret. Across the plausible range, model A has the highest net benefit or ties for it. At 15% it beats treating everyone by about 0.041 net true positives per patient, while model B adds nothing. At 40% model B's overestimated medium-group risk, 45% against an observed 30%, sends 30 extra patients to the intervention, 21 of whom would not have been admitted, at a cost of 0.05 net true positives per patient. Both models share an AUC of about 0.788 (see the Brier score page); the decision curve shows what their difference in calibration means for the decision.

From net true positives to money in a cost-effectiveness analysis

Decision curve analysis and cost-effectiveness analysis share a decision-analytic root but answer different questions. Because the threshold folds benefits and harms into one number, decision curve analysis needs no explicit costs, utilities or long-term outcomes; the 2019 guide notes that it requires only the range of thresholds, at the price of simplifying assumptions. A cost-effectiveness analysis models the pathway after testing, often with a decision tree or Markov model, and compares costs and QALYs at a cost-effectiveness threshold.

The two can be linked. Kerr and colleagues write the threshold in terms of a benefit to each treated case and a cost to each treated non-case, the cost read broadly to include money, time, stress and adverse health effects. If an economic model supplies both in money, decision curve net benefit converts into an expected monetary gain per patient compared with treating no one, as below.

$$p_t = \frac{L}{G + L}, \qquad G \times NB = G \times \frac{TP}{n} - L \times \frac{FP}{n}$$

where $G$ is the net monetary benefit of intervening for a patient who would have the event, $L$ is the net monetary loss of intervening for a patient who would not, and the other terms are as defined above. The second identity holds because $L / G$ equals $p_t / (1 - p_t)$. As an illustration with assumed values, suppose a decision model, at a stated cost-effectiveness threshold, gives £1,500 for each admission-prone patient reached and a loss of £500 for each patient treated unnecessarily. The implied threshold is 500 / 2,000, or 25%, at which the two models tie. Either model's net benefit of 0.1267 is worth £190 per patient compared with no programme, matching (22 × 1,500 − 28 × 500) / 100, and treating everyone is worth £20 per patient.

The link inherits the assumptions set out below and ignores the cost of running the model unless a test harm is added. A cost-effectiveness analysis relaxes these, adds downstream costs, discounts over a long horizon and characterises uncertainty through probabilistic sensitivity analysis. The 2019 guide suggests that a clear decision curve result, such as a model with no benefit, may make a fuller decision analysis unnecessary, while an equivocal one strengthens the case for it.

Two meanings of net benefit

The shared name can cause confusion. Decision curve net benefit counts net true positives per patient, uses the odds at a risk threshold as its exchange rate and needs only predictions and outcomes, whereas the economic net benefit, $\lambda E - C$ in money or $E - C/\lambda$ in health, values an option's expected health effect $E$ and expected cost $C$ at a cost-effectiveness threshold $\lambda$ in money per QALY.

Assumptions and choices behind a decision curve

Every decision curve rests on assumptions that are rarely stated, and several analytic choices change it. Vickers and Elkin assume that predicted risk and threshold are independent. Kerr and colleagues stress that every case is assumed to gain the same expected benefit and every non-case to bear the same expected cost, whatever their predicted risk. Subgroups with different thresholds therefore need separate curves, unless they share the same distribution of risks and the same prevalence.

Net benefit estimated in the data used to develop a model is optimistic, and simulations by Vickers and colleagues found repeated 10-fold cross-validation the best correction; transfer to a new setting needs external validation of the prediction model. For time-to-event outcomes, true and false positives at a landmark time come from the Kaplan-Meier estimator, or from the cumulative incidence function when there are competing risks. Kerr and colleagues also note that miscalibration reduces net benefit, so a calibrated model has equal or greater net benefit than a miscalibrated version of it.

Common misreadings of decision curves

Most errors come from reading the curve as something other than a population-level comparison at preset thresholds. The methodological literature describes the following.

• Choosing the threshold from the curve. Risks and outcomes carry no information on the benefits and costs of intervention, so the curve cannot identify the best threshold; the range is set first and the curve read within it.
• Comparing only with treating no one. A positive net benefit shows only that a model beats no intervention; it must also beat treating everyone.
• Equating a higher AUC with more value. In the original paper a clinical rule with an AUC of 0.72 did worse than treating everyone at some low thresholds.
• Using the curve for individual patients. Decision curves are a research tool, not for direct use in the clinic, and population net benefit is not well defined for individuals.
• Requiring statistical significance before acting. In general, under decision theory the strategy with the highest expected utility is chosen whatever its significance. Uncertainty bears on whether more research is worthwhile, and value of information methods, including the expected value of perfect information in decision curve net benefit (true positives), have been applied to the validation of risk models for that purpose.

Sources

• Assel M, Sjoberg DD, Vickers AJ. The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models. Diagnostic and Prognostic Research. 2017;1:19. https://doi.org/10.1186/s41512-017-0020-3
• Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the Economic Evaluation of Health Care Programmes. 4th ed. Oxford: Oxford University Press; 2015.
• Kerr KF, Brown MD, Zhu K, Janes H. Assessing the clinical impact of risk prediction models with decision curves: guidance for correct interpretation and appropriate use. Journal of Clinical Oncology. 2016;34(21):2534-2540. https://doi.org/10.1200/JCO.2015.65.5654
• Sadatsafavi M, Lee TY, Wynants L, Vickers AJ, Gustafson P. Value-of-information analysis for external validation of risk prediction models. Medical Decision Making. 2023;43(5):564-575. https://doi.org/10.1177/0272989X231178317
• Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Medical Informatics and Decision Making. 2008;8:53. https://doi.org/10.1186/1472-6947-8-53
• Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. https://doi.org/10.1177/0272989X06295361
• Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. https://doi.org/10.1136/bmj.i6
• Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagnostic and Prognostic Research. 2019;3:18. https://doi.org/10.1186/s41512-019-0064-7