Net benefit of a risk model at one threshold probability

Signature

NB = TP / n - FP / n * p_t / (1 - p_t)

Function

Decision curve net benefit of a risk model or test across threshold probabilities

Computational function

• Computational function: decision curve from predicted risks and outcomes over a threshold grid

  Takes each patient's predicted risk and observed outcome and a grid of thresholds, and returns at each threshold the net benefit of the model (HE-FM-DCA-001), of intervening in everyone (HE-FM-DCA-002) and of intervening in no one (zero), with net interventions avoided per 100 against intervening in everyone (HE-FM-DCA-003). The inputs differ from the formula's: the true and false positive counts are derived from the risks for each threshold instead of being entered.

  Inputs and outputs: risk: Predicted risk for each patient; required, vector, values from 0 to 1. Unit: probability.; y: Observed outcome for each patient, 1 for the event and 0 otherwise; required, vector of the same length. Unit: none.; p_t: Thresholds to evaluate; required, vector, each above 0 and below 1. Unit: probability.; NB_model, NB_all, NB_none: Net benefit of the model, of intervening in everyone and of intervening in no one at each threshold. Unit: true positives per patient.; avoided: Net unnecessary interventions avoided per 100 patients compared with intervening in everyone. Unit: interventions per 100 patients.

  Assumption: One validation sample with known outcomes for every patient (no censoring), patients classed positive when their risk is at or above the threshold, and the threshold range set before the curve is read.

  Worked example (Model A from the article at 15, 25 and 40 per cent): One hundred patients in groups of 50, 30 and 20 with predicted risks of 0.10, 0.30 and 0.60 and 4, 9 and 13 admissions give net benefits of 0.1706, 0.1267 and 0.0833 for the model and 0.1294, 0.0133 and minus 0.2333 for intervening in everyone, and 23.33, 34.0 and 47.5 interventions avoided per 100. p_t = [0.15, 0.25, 0.40]; NB_model = [0.1706, 0.1267, 0.0833]; NB_all = [0.1294, 0.0133, -0.2333]; NB_none = [0, 0, 0]; avoided = [23.33, 34.0, 47.5]

  Worked example (Model B with overestimated risks): With risks of 0.20, 0.45 and 0.80 for the same groups, model B's net benefits are 0.1294, 0.1267 and 0.0333: equal to intervening in everyone at 15 per cent (0 avoided), tied with model A at 25 per cent and below it at 40 per cent. NB_model = [0.1294, 0.1267, 0.0333]; avoided = [0, 34.0, 40.0]

  Excel: With risks in A2:A101, outcomes in B2:B101 and a threshold in D2, =COUNTIFS(A$2:A$101,">="&D2,B$2:B$101,1)/COUNT(B$2:B$101)-COUNTIFS(A$2:A$101,">="&D2,B$2:B$101,0)/COUNT(B$2:B$101)*D2/(1-D2) in E2 gives the model's net benefit and =AVERAGE(B$2:B$101)-(1-AVERAGE(B$2:B$101))*D2/(1-D2) in F2 the net benefit of intervening in everyone; fill down for a column of thresholds.

  R: dca <- function(risk, y, p_t) { n <- length(y); sapply(p_t, function(p) { pos <- risk >= p; w <- p/(1-p); m <- sum(pos & y == 1)/n-sum(pos & y == 0)/n*w; a <- mean(y)-(1-mean(y))*w; c(p_t = p, NB_model = m, NB_all = a, NB_none = 0, avoided = (m-a)/w*100) }) } With risk <- rep(c(0.10, 0.30, 0.60), c(50, 30, 20)) and y <- c(rep(1, 4), rep(0, 46), rep(1, 9), rep(0, 21), rep(1, 13), rep(0, 7)), dca(risk, y, c(0.15, 0.25, 0.40)) returns the values of the first example.

  Python: def dca(risk, y, p_t): n = len(y); pi = sum(y)/n; return [{"p_t": p, "NB_model": m, "NB_all": a, "NB_none": 0, "avoided": (m-a)*(1-p)/p*100} for p in p_t for m, a in [(sum(v for r, v in zip(risk, y) if r >= p)/n-sum(1-v for r, v in zip(risk, y) if r >= p)/n*p/(1-p), pi-(1-pi)*p/(1-p))]] Returns the same values as the R function for the same inputs.

  Test (Intervening in everyone crosses zero at the prevalence): At a threshold equal to the sample prevalence, 0.26 here, NB_all is zero. Expected result: TRUE. Excel check: =IF(ABS(D2-AVERAGE(B$2:B$101))<1E-12,ABS(F2)<1E-12,TRUE)

  Test (A model classing everyone positive equals intervening in everyone): When every risk is at or above the threshold, as for model B at 15 per cent, NB_model equals NB_all and avoided is zero. Expected result: TRUE. Excel check: =IF(MIN(A2:A101)>=D2,ABS(E2-F2)<1E-12,TRUE)

  Common error (Choosing the threshold from the curve): Risks and outcomes carry no information on the benefits and harms of intervention, so the threshold that maximises the model's net benefit is not the right threshold; the range is set first and the curve read within it.

  Source: Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361. Application section and Table 2 note.

  NB_model(p_t) = sum_(i=1)^n [r_i >= p_t and y_i = 1] / n - sum_(i=1)^n [r_i >= p_t and y_i = 0] / n * p_t / (1 - p_t); NB_all(p_t) = pi - (1 - pi) * p_t / (1 - p_t); avoided = (NB_model - NB_all) * (1 - p_t) / p_t * 100

Try this function

Implementations

• Excel

  Decision curve net benefit from named counts

  With the counts in cells named TruePos, FalsePos and SampleSize and the threshold in Threshold, the formula returns the net benefit, held in a cell named NetBenefit.

  =TruePos/SampleSize-FalsePos/SampleSize*Threshold/(1-Threshold)

Assumptions

• Threshold reflects the harm of a missed case against an unnecessary intervention

  Vickers and Elkin derive p_t from a decision tree in which the expected value of treating equals that of not treating, so the odds p_t / (1 minus p_t) equal the harm of an unnecessary intervention divided by the harm of a missed case. The threshold range is set before the curve is read.

• Same expected benefit for every case and harm for every non-case in net benefit

  Kerr and colleagues note that the formula assumes every case gains the same expected benefit and every non-case bears the same expected cost, whatever the predicted risk. Subgroups with different thresholds need separate curves.

• Decision curve net benefit from validation data separate from model development

  Net benefit estimated in the data used to build the model is optimistic; simulations by Vickers and colleagues found repeated 10-fold cross-validation the best correction, and transfer to a new setting needs external validation.

Worked examples

• Seminal vesicle invasion model at a 10 per cent threshold

  In Vickers and Elkin's validation sample of 902 men, the prediction model classed 65 men with seminal vesicle invasion and 225 without it as positive at a 10 per cent threshold, giving a net benefit of 0.0443.

  TP = 65; FP = 225; n = 902; p_t = 0.1; NB = 0.0443

• Admission-risk model A at a 15 per cent threshold

  Model A classes the medium and high risk groups as positive: 22 admissions and 28 patients not admitted among 100. The false positive weight is 0.15 / 0.85, about 0.1765, and the net benefit 0.1706, as in the article.

  TP = 22; FP = 28; n = 100; p_t = 0.15; NB = 0.1706

• Admission-risk model B at a 40 per cent threshold

  Model B overestimates the medium group's risk at 45 per cent, so at 40 per cent it still classes 22 admissions and 28 non-admissions as positive. With a weight of about 0.6667 its net benefit is 0.0333, against 0.0833 for model A, which classes only the high group (13 and 7) as positive.

  TP = 22; FP = 28; n = 100; p_t = 0.4; NB = 0.0333

Common errors

• Weighting false positives by the threshold instead of its odds

  Charging each false positive p_t instead of p_t / (1 minus p_t) understates the penalty; for model A at 40 per cent it gives 0.13 minus 0.07 x 0.40, or 0.102, instead of 0.0833.

• Reading net benefit as money or QALYs

  Decision curve net benefit is in units of true positives per patient. Converting it to money needs an explicit value per case found (HE-FM-DCA-005), and it is not the economic net monetary benefit of an option.

• Ignoring the test harm of obtaining a risk model's inputs in net benefit

  If the inputs need an invasive or costly test, a test harm in units of true positives is subtracted at every threshold, for example 1/30, about 0.033, when clinicians would scan at most about 30 men to find one high-grade cancer.

Sources

• Net benefit formula and its range in the original decision curve paper

  Vickers AJ, Elkin EB. Decision curve analysis: a novel method for evaluating prediction models. Medical Decision Making. 2006;26(6):565-574. doi:10.1177/0272989X06295361. Introductory theory, Equation 1 and the application section: positive if predicted probability is equal to or higher than p_t; Net Benefit = TruePositiveCount/n minus FalsePositiveCount/n times p_t/(1 minus p_t); worked example 65 true and 225 false positives among 902 at 10 per cent giving 0.0443; range from negative infinity to the incidence of disease; a version subtracting TestHarm.

  View source →

• Unit of decision curve net benefit and the reading of 0.07

  Vickers AJ, van Calster B, Steyerberg EW. A simple, step-by-step guide to interpreting decision curve analysis. Diagnostic and Prognostic Research. 2019;3:18. doi:10.1186/s41512-019-0064-7. Step 4: the unit of net benefit is true positives; a net benefit of 0.07 means 7 true positives for every 100 patients in the target population.

  View source →

• Test harm subtracted from net benefit for a costly or invasive input

  Vickers AJ, Van Calster B, Steyerberg EW. Net benefit approaches to the evaluation of prediction models, molecular markers, and diagnostic tests. BMJ. 2016;352:i6. doi:10.1136/bmj.i6. Extensions to net benefit methods: investigators who would do no more than about 30 scans to find one high grade cancer would set a test harm of 0.033 (1/30), subtracted from the net benefit of the prediction model across all thresholds.

  View source →

• Repeated 10-fold cross-validation to correct decision curves for overfit

  Vickers AJ, Cronin AM, Elkin EB, Gonen M. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Medical Informatics and Decision Making. 2008;8:53. doi:10.1186/1472-6947-8-53. Abstract (Results): simulation studies showed that repeated 10-fold cross-validation provided the best method for correcting a decision curve for overfit.

  View source →

Canonical Identity