Brier Score

A measure of probabilistic prediction accuracy, the mean squared difference between predicted probabilities and actual observed binary outcomes, lower being better.

Source: Brier 1950

The Brier score measures how accurate probabilistic predictions are by taking the mean squared difference between the predicted probabilities and the outcomes that actually occurred, coded as zero or one. A lower score is better, since it means the forecasts sat close to reality; a prediction of ninety per cent for events that happen scores well, while confident predictions that prove wrong are penalised heavily. It rewards forecasts that are both correct in direction and honest about their certainty. Scoring the closeness of probabilities to outcomes is its purpose. Steyerberg (2009) describes this measure.

Source: Steyerberg 2009

The Brier score is calculated by taking, for each observation, the difference between the predicted probability of the event and the actual outcome coded as one if the event occurred and zero if not, squaring this difference, and averaging the squared differences across all observations. So the Brier score is calculated as the mean squared error between predicted probabilities and observed binary outcomes, which ranges from zero for perfect predictions to one for the worst possible, and because it penalises predictions that are far from the outcome, it rewards probabilities that are both well calibrated and appropriately confident in the correct direction.

Source: Brier 1950

The Brier score is interpreted such that lower values indicate better predictions, with zero being perfect and higher values worse; a score can be compared against that of a naive prediction, such as always predicting the overall event rate, to judge whether a model adds value. It reflects both calibration and discrimination together. So the Brier score is interpreted as an overall measure of probabilistic prediction accuracy, with smaller values meaning predicted probabilities are closer to outcomes, and it is often compared with a reference to gauge improvement, though because it combines aspects of performance, it is used alongside separate assessments of calibration and discrimination.

Source: Brier 1950

The Brier score captures the overall accuracy of probabilistic predictions, reflecting both calibration, how well predicted probabilities match observed frequencies, and discrimination, how well the predictions separate events from non-events. It can be decomposed into components relating to these aspects. So the Brier score captures a combination of calibration and discrimination in a single measure of how close predictions are to outcomes, which makes it a useful summary of a probabilistic model's performance, though because it aggregates these aspects, examining its components or complementary measures gives a fuller understanding of where a model performs well or poorly.

Source: Brier 1950

The limitations of the Brier score include that, as a single summary, it does not by itself show whether poor performance stems from calibration or discrimination, so its components or other measures are needed for detail; and that it can be affected by the outcome's prevalence, complicating comparisons across settings with different event rates. So the Brier score is used alongside separate assessments of calibration and discrimination and with attention to the event rate, since a single value does not reveal the source of prediction error and comparisons between populations require accounting for differing prevalences, which is why it is one of several tools for evaluating probabilistic predictions rather than a complete assessment.

Source: Brier 1950