Brier Score

Brier Score for Judging Risk Predictions Before They Enter a Decision Model

Health economic models often take their event probabilities from a risk equation or prognostic score, so the quality of those predicted risks feeds straight into expected costs, quality-adjusted life years and the resulting cost-effectiveness estimates. The Brier score is the traditional overall measure of such probabilities, rewarding predictions that are both well calibrated and able to separate patients who have the event from those who do not, the ability known as discrimination and usually measured by the area under the ROC curve.

From weather forecasts to clinical risk models

The score was introduced in 1950 by Glenn Brier to verify weather forecasts expressed as probabilities, and it was later adopted outside meteorology, including in medicine, as a simple scoring rule for predictions of binary outcomes. Its appeal rests on being a proper scoring rule. A scoring rule is proper when the expected score is best for the correct probabilities, so there is no gain from shading predictions towards 0, 1 or the average, and miscalibrated probabilities are penalised. Because overfitting produces predictions that are too extreme in new data, a model that has been fitted too closely to its development sample tends to show a worse Brier score when it is validated elsewhere, which links the score to concerns about overfitting.

The formula and its range

The calculation needs only two columns, the predicted probability for each patient and whether the event occurred. For a sample of patients with binary outcomes, the score averages the squared gaps between the two. It is defined as follows.

$$BS = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2$$

where $BS$ is the Brier score, $N$ is the number of patients, $p_i$ is the predicted probability of the event for patient $i$, and $y_i$ equals 1 if patient $i$ had the event and 0 otherwise. The score is the mean squared error of probability forecasts, and its square root can be read as a typical distance between prediction and outcome on the probability scale.

The score runs from 0 for perfect predictions to 1 for predictions that are confidently and completely wrong, and lower values are better. A useful reference point is a non-informative model that gives every patient the same risk equal to the overall event rate. With an event rate of 50% that model scores 0.25, and with an event rate of 10% it scores 0.090, so the score that counts as poor depends on how common the outcome is.

Murphy's partition into reliability, resolution and uncertainty

A single number hides two different strengths of a set of predictions. Allan Murphy partitioned the score in 1973 into a term for the uncertainty inherent in the outcome, a reliability term and a resolution term. When the predictions take a limited set of distinct values, the partition is exact and is written as follows.

$$BS = \frac{1}{N} \sum_{k=1}^{K} n_k (f_k - \bar{o}_k)^2 - \frac{1}{N} \sum_{k=1}^{K} n_k (\bar{o}_k - \bar{o})^2 + \bar{o}(1 - \bar{o})$$

where $K$ is the number of distinct predicted values, $n_k$ is the number of patients given predicted value $f_k$, $\bar{o}_k$ is the observed event proportion among those patients, and $\bar{o}$ is the overall event proportion in the sample. The first term is the reliability term, which measures miscalibration and is zero when every predicted risk matches the observed rate in its group. The second term is resolution, which measures how far the group event rates spread away from the overall rate and is subtracted because more spread is better. The third term is uncertainty, the score of the non-informative model, which depends only on the outcome and not on the model.

Combining the last two terms gives the calibration and refinement form of the same identity, in which refinement equals uncertainty minus resolution and the score equals reliability plus refinement. Refinement is the average uncertainty that remains within the groups the model creates. Recalibrating a model can drive the reliability term towards zero, but only better separation of patients, through stronger predictors or a better model, reduces refinement. With continuous predicted risks the partition is usually computed after grouping predictions into bins, and the result then depends on the number and width of bins chosen.

The scaled Brier score

Because the raw score depends on the event rate, it is hard to compare across populations with different incidence. Steyerberg and colleagues proposed scaling it against the non-informative model. The scaled score is defined as follows.

$$BS_{scaled} = 1 - \frac{BS}{BS_{max}}, \quad BS_{max} = \bar{o}(1 - \bar{o})$$

where $BS_{scaled}$ is the scaled Brier score, $BS_{max}$ is the score of a non-informative model that predicts the overall event rate $\bar{o}$ for everyone, and $BS$ is as defined above. Steyerberg and colleagues write the reference in terms of the mean predicted risk, which gives the same value when a model is correct on average. When models that are miscalibrated in the large are compared, basing the reference on the observed rate keeps it fixed across models. Basing it on each model's own mean prediction gives an over-predicting model a larger denominator when the event rate is below 50%, as with most clinical outcomes, and can make it look better, as the worked example below shows. The scaled score runs from 0% for a model no better than the overall event rate to 100% for perfect prediction, and the authors note that it is very similar to Pearson's $R^2$. A negative value means the model does worse than predicting the overall rate, a clear signal that it should not feed a decision model in that population without recalibration.

How the Brier score relates to discrimination

Discrimination asks whether patients who have the event receive higher predicted risks than those who do not. It is usually summarised by the c statistic, which equals the area under the ROC curve, and it depends only on the rank order of predictions. The Brier score, by contrast, captures both calibration and discrimination, because the resolution term rewards separation and the reliability term penalises predictions that are wrong in absolute level.

The distinction matters because two models can rank patients almost identically yet imply very different numbers of people at high risk. Van Calster and colleagues describe an external validation in about 2 million patients in the United Kingdom in which QRISK2-2011 was well calibrated with an AUC of 0.771, whereas a Framingham-based equation overestimated risk with an AUC of 0.776. At a 20% ten-year risk threshold, QRISK2-2011 would select 110 per 1000 men aged 35 to 74, while the Framingham-based equation would select 206 per 1000. Discrimination gave no warning of this near doubling of the treated population; the reliability part of the Brier score is designed to pick it up.

Why absolute risk matters in health economic models

A cohort Markov model or a decision tree uses predicted risks as inputs, for example as the transition probability from a healthy state to an event state for a subgroup defined by a risk score, after conversion to the model's cycle length. Expected costs and QALYs are weighted sums over these probabilities, so a risk equation that overestimates events inflates the events a treatment can prevent, and with them the health gain and cost offsets. A model that discriminates well but is miscalibrated in the target population can therefore produce misleading cost-effectiveness analysis results, especially when treatment is targeted at patients above a risk threshold.

For this reason the Brier score, read together with its reliability component and a calibration curve, is a natural part of model validation when a prediction model is imported into an economic evaluation. Van Calster and colleagues recommend assessing calibration with calibration curves and summary measures such as the calibration intercept and slope. For time-to-event outcomes, which drive many survival analysis inputs, the Brier score can be computed at fixed time points with weighting for censoring, and compared with a benchmark curve based on the overall Kaplan-Meier estimator, which uses no predictive information.

Worked example with two risk models that rank patients identically

The figures below are illustrative. A risk model for one-year hospital admission is validated in 100 patients, and it assigns each patient to one of three predicted risks. A second model, developed in a higher-risk setting, places the same patients in the same order but with higher predicted risks.

1. Calculate the Brier score for model A. Within each group, every admitted patient contributes $(1 - f_k)^2$ and every other patient contributes $f_k^2$. The low group contributes 4 × 0.81 + 46 × 0.01, giving 3.70, the medium group 9 × 0.49 + 21 × 0.09, giving 6.30, and the high group 13 × 0.16 + 7 × 0.36, giving 4.60.

$$BS_A = \frac{3.70 + 6.30 + 4.60}{100} = \frac{14.60}{100} = 0.146$$

where $BS_A$ is the Brier score of model A.

2. Partition the score. The reliability term is small because each predicted risk is close to its observed rate.

$$REL_A = \frac{50 \times (0.10 - 0.08)^2 + 30 \times (0.30 - 0.30)^2 + 20 \times (0.60 - 0.65)^2}{100} = \frac{0.02 + 0 + 0.05}{100} = 0.0007$$

where $REL_A$ is the reliability term of model A. Resolution and uncertainty depend only on the groups and the outcomes.

$$RES = \frac{50 \times (0.08 - 0.26)^2 + 30 \times (0.30 - 0.26)^2 + 20 \times (0.65 - 0.26)^2}{100} = \frac{1.62 + 0.048 + 3.042}{100} = 0.0471$$

where $RES$ is the resolution term. The uncertainty term is 0.26 × 0.74, or 0.1924, and 0.0007 minus 0.0471 plus 0.1924 reproduces the score of 0.146. Refinement is 0.1924 minus 0.0471, or 0.1453.

3. Repeat for model B. Model B has the same groups, so its resolution, uncertainty and refinement are unchanged. Only its reliability differs.

$$REL_B = \frac{50 \times (0.20 - 0.08)^2 + 30 \times (0.45 - 0.30)^2 + 20 \times (0.80 - 0.65)^2}{100} = \frac{0.72 + 0.675 + 0.45}{100} = 0.01845$$

where $REL_B$ is the reliability term of model B. Its Brier score is therefore $BS_B = 0.01845 + 0.1453 = 0.16375$.

4. Scale both scores. Model A predicts 26 admissions in total (50 × 0.10 + 30 × 0.30 + 20 × 0.60), matching the 26 observed, so the reference score is 0.1924.

$$BS_{scaled,A} = 1 - \frac{0.146}{0.1924} = 0.241$$

where $BS_{scaled,A}$ is the scaled score of model A. Against the same reference, model B scores 1 minus 0.16375 divided by 0.1924, about 0.149. Model A captures about 24% of the possible improvement over the non-informative model and model B about 15%. Had model B been scaled against its own mean prediction of 0.395, the reference would have been 0.395 × 0.605, about 0.239, and the scaled score about 0.315, wrongly suggesting that model B outperforms model A.

5. Compare discrimination. Because both models order the groups in the same way, they share the same c statistic. Counting tied pairs as half, 1,517 of the 1,924 pairs of an admitted and a non-admitted patient are ranked correctly, an AUC of about 0.788 for both models.

6. Interpret for a decision model. Model B predicts 39.5 admissions per 100 patients (50 × 0.20 + 30 × 0.45 + 20 × 0.80) against 26 observed. If an intervention cuts admissions by an assumed 25%, a decision model fed by model B would count about 9.9 admissions avoided per 100 patients treated instead of 6.5, overstating the health gain and any cost offsets by about half. With a treatment threshold of 0.40, model A would treat the 20 high-risk patients and model B would treat 50, the medium and high groups together. The AUC cannot separate the two models; the Brier score and its reliability term can.

Choices that change the score

The Brier score is a property of a model in a particular population, so several analytic choices alter it and should be reported with it. Apparent performance in the development data is optimistic, so the score should be estimated in an external sample or corrected with cross-validation or bootstrapping. Raw scores from a low-risk primary care population and a high-risk hospital population are not directly comparable, because the prevalence of the outcome sets the uncertainty term. For survival outcomes, the time point and the method used to handle censoring both change the result.

What the Brier score cannot tell a decision maker

A lower Brier score indicates more accurate probabilities, but it does not show whether using the model leads to better decisions. Assel, Sjoberg and Vickers examined common clinical scenarios and found that the score could rank tests and models in the wrong order. When prevalence was low, it favoured a highly specific test even where the clinical context called for high sensitivity, and continuous models scored better than binary tests even when the test had clearly superior properties and the model was badly miscalibrated. They concluded that the score does not evaluate clinical value and recommended decision-analytic measures such as net benefit, often reported through decision curve analysis, which weight false positives against true positives at a chosen risk threshold.

A small overall score can also hide poor calibration in the high-risk group that a targeted programme would treat, so the reliability term and a calibration curve should be examined alongside it. For a health economic model the score is best read as one part of validating the risk inputs, with the economic consequences of any remaining miscalibration explored in scenario analysis or probabilistic sensitivity analysis.

Sources

• Assel M, Sjoberg DD, Vickers AJ. The Brier score does not evaluate the clinical utility of diagnostic tests or prediction models. Diagnostic and Prognostic Research. 2017;1:19. https://doi.org/10.1186/s41512-017-0020-3
• Blattenberger G, Lad F. Separating the Brier score into calibration and refinement components: a graphical exposition. The American Statistician. 1985;39(1):26-32. https://doi.org/10.1080/00031305.1985.10479382
• Brier GW. Verification of forecasts expressed in terms of probability. Monthly Weather Review. 1950;78(1):1-3. https://doi.org/10.1175/1520-0493(1950)078%3C0001:VOFEIT%3E2.0.CO;2
• Briggs A, Claxton K, Sculpher M. Decision Modelling for Health Economic Evaluation. Oxford: Oxford University Press; 2006.
• Bröcker J. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the Royal Meteorological Society. 2009;135(643):1512-1519. https://doi.org/10.1002/qj.456 (preprint: https://arxiv.org/abs/0806.0813)
• Murphy AH. A new vector partition of the probability score. Journal of Applied Meteorology. 1973;12(4):595-600. https://doi.org/10.1175/1520-0450(1973)012%3C0595:ANVPOT%3E2.0.CO;2
• Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, Pencina MJ, Kattan MW. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128-138. https://pmc.ncbi.nlm.nih.gov/articles/PMC3575184/
• Van Calster B, McLernon DJ, van Smeden M, Wynants L, Steyerberg EW. Calibration: the Achilles heel of predictive analytics. BMC Medicine. 2019;17:230. https://doi.org/10.1186/s12916-019-1466-7