Functions & Formulae

Each applied formula has its own function page, with a signature, implementations, and tests.

Brier score as a proper scoring rule for predicted risks

BS(p, y) = mean((p - y)^2)

Maps a set of predicted event probabilities and the observed binary outcomes to the mean squared difference between them, a proper scoring rule in which lower values are better. The score rewards predictions that are both well calibrated and able to separate patients who have the event from those who do not, and it can be partitioned into reliability, resolution and uncertainty terms. In health economic models it is one check on a risk equation before its predictions become event probabilities. Discrimination alone is covered by HE-FM-AUC-001.

  • Brier score from patient-level predicted risks and outcomes

    BS = sum_(i=1)^N [(p_i - y_i)^2] / N

    Averages the squared differences between each patient's predicted probability of the event and the observed outcome, coded 1 for an event and 0 otherwise. The score runs from 0 for perfect predictions to 1 for predictions that are confidently and completely wrong, and its square root is a typical distance between prediction and outcome on the probability scale.

  • Murphy partition of the Brier score into reliability, resolution and uncertainty

    REL = sum_(k=1)^K [n_k * (f_k - o_k)^2] / N; RES = sum_(k=1)^K [n_k * (o_k - o_bar)^2] / N; UNC = o_bar * (1 - o_bar); REF = UNC - RES; BS = REL - RES + UNC

    When predictions take a limited set of K distinct values, the Brier score splits exactly into a reliability term, which measures miscalibration, minus a resolution term, which measures how far the group event rates spread from the overall rate, plus an uncertainty term, the score of the non-informative model. Refinement, uncertainty minus resolution, is the average uncertainty left within the groups, and the score also equals reliability plus refinement.

  • Scaled Brier score against the non-informative model

    BS_max = o_bar * (1 - o_bar); BS_scaled = 1 - BS / BS_max

    Scales the Brier score against a non-informative model that predicts the overall event rate for every patient, so that 0 means no improvement on that model and 1 means perfect prediction. Steyerberg and colleagues write the reference in terms of the mean predicted risk; basing it on the observed rate keeps it fixed when models that are miscalibrated in the large are compared. A negative value means the model does worse than predicting the overall rate.

Brier Score — Functions & Formulae | HealthEconomics.wiki