Root Mean Square Error

Root Mean Square Error

Root mean square error (RMSE) summarises the typical scale of differences between numerical predictions and observed values by taking the square root of their average squared error. It is expressed in the outcome's original units and gives comparatively greater weight to large errors than mean absolute error (MAE). Its usefulness depends on what was predicted, when predictions were made, which observations were evaluated and whether squared-error loss reflects the decision problem.

Definition and calculation

For $n>0$ matched observation–prediction pairs $(y_i,\widehat y_i)$, define residual or prediction error $e_i=\widehat y_i-y_i$. The sample evaluation metric is

$$\mathrm{RMSE}=\sqrt{\frac{1}{n}\sum_{i=1}^{n}(\widehat y_i-y_i)^2}.$$

Squaring removes error signs and penalises larger magnitudes; averaging yields mean squared error (MSE), and taking the square root returns to the units of $y$. RMSE is zero if and only if every evaluated prediction equals its corresponding observed value. It cannot be negative. Specify the sign convention for residuals if also reporting mean error; RMSE itself is unchanged if every residual sign is reversed.

This is an evaluation formula dividing by the number of evaluated pairs, not an unbiased estimate of residual variance with an $n-p$ or $n-1$ degrees-of-freedom correction. The predictive accuracy of a fitted model should ordinarily be estimated on data not used to fit or tune it, using a split, cross-validation or external test set suited to the deployment setting. Repeated observations from one patient and time-dependent forecasts require evaluation procedures that respect those dependencies.

Worked health service example

Suppose a service forecasts four days' urgent-care visits. The table pairs each day's prediction with the count subsequently observed; its small size is for teaching, not validation of a forecasting system.

The squared errors total $4+1+9+4=18$. Thus $\mathrm{MSE}=18/4=4.5$ visits squared and $\mathrm{RMSE}=\sqrt{4.5}\approx2.1213$ visits. For comparison, $\mathrm{MAE}=(2+1+3+2)/4=2$ visits; RMSE is larger because squaring puts more emphasis on the three-visit error. The mean signed error under the stated prediction-minus-observation convention is $(2-1+3-2)/4=0.5$ visits, which indicates average overprediction in these four days but does not reveal the spread of errors.

The same arithmetic can be checked through the sample's population-denominator decomposition: $\mathrm{RMSE}^2=\overline{e}^{,2}+n^{-1}\sum_i(e_i-\overline{e})^2$. Here $\overline{e}=0.5$, and the mean squared deviation around it is $4.25$; $0.25+4.25=4.5$. This identity describes the evaluated errors, not an independent claim about performance in future patients or days.

Interpretation and comparison

RMSE has the units of the outcome: visits for counts, currency units for costs and years for life-years. It is not a percentage, and it should not be called an average absolute error: the latter is MAE. In the same unweighted evaluation sample, $\mathrm{RMSE}\geq\mathrm{MAE}$ by the relationship between quadratic and arithmetic means of nonnegative absolute errors. The inequality can be strict even when the mean signed error is zero.

RMSE is sensitive to unusually large residuals. That can be desirable if a missed surge or a severe cost forecast miss carries disproportionate consequences, but a few noisy or erroneous observations can dominate the score. Examine residual distributions and errors by subgroup, time and prediction level. Compare with a simple relevant baseline, and report MAE, bias or quantiles of absolute error where they help explain practical performance. A single RMSE does not tell whether errors are systematically high or low, whether intervals are calibrated, or whether the model is fair across populations.

Scores should be compared on the same target definition, units, horizon, sample, missing-data rule and weighting scheme. An RMSE of 2 visits on a small clinic is not directly comparable with 20 visits on a large hospital without a meaningful scale or baseline. In hierarchical or multivariate predictions, state whether RMSE was calculated separately for each output before aggregation; averaging component RMSEs is generally different from taking the square root after pooling all squared errors.

Weighted and population-specific versions

For specified nonnegative weights $w_i$ with $\sum_iw_i>0$, a weighted version is

$$\mathrm{RMSE}_{w}=\sqrt{\frac{\sum_iw_i(\widehat y_i-y_i)^2}{\sum_iw_i}}.$$

Weights can reflect a target population, sampling design or agreed decision relevance, but they change the estimand and must be justified. If a day's cost of error is asymmetric or rises differently from the square of the residual, use a loss function matching that decision rather than assuming RMSE captures it. Record whether zero-weight observations were excluded and whether weights came from the evaluation design, not from the observed errors chosen after seeing results.

Missing observations can bias the evaluation if missingness relates to actual outcomes or forecast difficulty. For hospital utilisation or cost models, assess rare extreme values and data quality rather than trimming them solely to improve RMSE. For censored survival outcomes, ordinary squared error between a predicted event time and observed follow-up time is generally invalid because a censored time is not the true event time; use an appropriate time-specific, censoring-aware prediction metric instead. For probability forecasts of a binary event, the mean squared probability error is the Brier score; its square root has a related mathematical form but should be clearly labelled and interpreted in that probability-prediction context.

Health economic use and limits

RMSE can evaluate forecasts of health service demand, costs, resource use or modelled continuous outcomes. When choosing a model, an apparently lower test-set RMSE should be considered with calibration, uncertainty, interpretability, transportability and decision consequences. Better point prediction of costs does not automatically imply better treatment allocation or greater cost-effectiveness. In particular, decisions may depend on the joint distribution of costs and benefits, not on marginal prediction error for one outcome.

Report the target and units, prediction and observation timing, sample size, train–test separation, missing-data handling, any transformation and back-transformation, weights, RMSE with uncertainty where feasible, benchmark performance and key residual diagnostics. Do not calculate RMSE on a transformed scale and describe it as if it were in original currency or clinical units. If forecasts were updated over time, specify the information actually available at each forecast origin to avoid leakage.

Methodological sources

The scikit-learn metric reference documents the root mean squared error calculation and optional weighting for regression outputs. Chai and Draxler's RMSE versus MAE discussion examines how error distributions affect interpretation. Hyndman and Koehler's study of forecast accuracy measures explains limitations of scale-dependent comparisons. These methodological sources support the calculation and reporting principles; the four-day forecast is an original illustrative example.