Net Benefit

A summary measure, in evaluating prediction models, combining the benefit of correctly identifying an outcome against the harm of unnecessary treatment, at a given threshold.

Source: Vickers & Elkin 2006

Net benefit puts the good and harm of acting on a prediction model onto a single scale so they can be compared. It counts the true positives, patients correctly identified and treated, as the benefit, and subtracts the false positives, patients treated unnecessarily, weighted by how much worse an unnecessary treatment is than a missed case. That weighting comes from the threshold probability at which a clinician would choose to act. The measure thus expresses the model's value net of its harms. It nets benefit against harm. Vickers and Elkin (2006) define it.

Source: Vickers & Elkin 2006

Net benefit is calculated by counting the true positives, those correctly identified and treated, and subtracting the false positives, those unnecessarily treated, with the false positives weighted by a factor derived from the threshold probability that reflects how the decision maker values avoiding a missed case relative to avoiding an unnecessary treatment. The result, often expressed per person, represents the net gain from using the model at that threshold. Computing net benefit across a range of thresholds produces the decision curve, showing over what thresholds using the model is beneficial.

Source: Vickers & Elkin 2006

The threshold, the probability at which one would choose to intervene, plays a central role in net benefit because it sets the relative weight given to the harms of unnecessary treatment versus the benefits of treating those who need it. A low threshold implies that missing a case is much worse than treating unnecessarily, weighting false positives lightly, while a high threshold implies the opposite. So the threshold encodes the decision maker's trade-off, and net benefit is computed for the relevant threshold or across a range, since the value of a model depends on the threshold used.

Source: Weinstein & Fineberg 1980

Net benefit is useful because it incorporates the consequences of decisions, weighing benefits against harms on a common scale, so it evaluates a model by its clinical value rather than by statistical measures such as discrimination or calibration alone, which do not reflect the trade-offs of acting on the model. By showing whether using a model yields more good than harm at relevant thresholds, net benefit directly addresses whether the model helps in practice. This makes it valuable for judging the clinical usefulness of prediction and diagnostic models beyond their statistical performance.

Source: Vickers & Elkin 2006

The limitations of net benefit include its reliance on the threshold probability to represent the trade-off between benefits and harms, which may vary between patients and decision makers and may not be precisely known; its focus on the benefit-harm trade-off without capturing all relevant considerations, such as costs beyond that trade-off; and its dependence on the data and model used to estimate it. It also summarises value at chosen thresholds rather than giving a single absolute figure for all contexts. So net benefit is interpreted across relevant thresholds and alongside other evidence about a model's value.

Source: Vickers & Elkin 2006