Mean Squared Error

Mean Squared Error: Concept Architecture

Orientation and learning roadmap

This page moves from the choice of a prediction target to the calculation and interpretation of squared errors. It then separates held-out prediction performance from estimator risk and residual variance, before applying the measure to utility mapping and other health economic predictions.

Define the target and evaluation set

Predictions can concern patient utilities, healthcare costs, event rates, or model parameters, but an error has meaning only relative to a named observed or reference target. For a fixed set of $n$ paired observations, denote the target by $y_i$ and the prediction by $\widehat{y}_i$, with both measured on the same scale and at the same relevant time.

The unweighted empirical mean squared error (MSE) is

$$\mathrm{MSE}=\frac{1}{n}\sum_{i=1}^{n}(y_i-\widehat{y}_i)^2.$$

Each difference is squared before averaging, so positive and negative errors cannot cancel and larger absolute errors receive disproportionate weight. The result has squared units: a utility prediction has utility-squared MSE, while a cost prediction in pounds has pounds-squared MSE. A smaller value indicates closer predictions on this particular dataset and scale, not necessarily a better decision model overall.

Worked example: utility predictions

Suppose three patients have observed utility values of $0.80$, $0.60$, and $0.40$, while a mapping algorithm predicts $0.70$, $0.50$, and $0.60$. The example treats each patient equally and uses paired observations measured with the same utility value set.

The sum of squared errors is $0.01+0.01+0.04=0.06$, so $\mathrm{MSE}=0.06/3=0.02$ utility squared. The corresponding root mean squared error is $\mathrm{RMSE}=\sqrt{0.02}\approx 0.1414$ utility units, while the signed mean error is zero because $0.10+0.10-0.20=0$. Zero average signed error does not imply good prediction for each patient or for a clinically important subgroup.

Prediction error versus estimator mean squared error

The same mathematical idea also evaluates an estimator $\widehat{\theta}$ of a fixed parameter $\theta$ across repeated hypothetical samples. In that setting, the expectation is taken over the sampling distribution and the classical decomposition separates sampling variance from squared bias.

$$\operatorname{MSE}(\widehat{\theta}) =E\!\left[(\widehat{\theta}-\theta)^2\right] =\operatorname{Var}(\widehat{\theta}) +\left[E(\widehat{\theta})-\theta\right]^2.$$

This estimator-risk quantity is not the same as the empirical average of patient-level prediction errors, even though both use squared differences. In prediction, noise in future outcomes and the setting of evaluation matter; in parameter estimation, the true parameter and sampling process define the expectation. Authors should state which use of MSE they mean rather than importing a bias–variance conclusion from one target to the other without qualification.

Validation and comparison

An error calculated on the same records used to fit and tune a flexible model can be optimistically small. Hold-out validation, cross-validation with all preprocessing and tuning performed inside each training fold, or independent external validation better evaluates performance in new data.

Compare models on the same outcome definition, valuation set, case mix, and evaluation records. Check error across severe and mild health states as well as treatment groups, because an acceptable overall MSE can hide systematic bias at the ends of a utility scale. If observations have prespecified weights, a weighted average can be used, but the weights, denominator, and population they represent must be disclosed.

Related measures and limitations

Root mean squared error (RMSE) is the square root of MSE and returns the result to the original outcome unit; mean absolute error (MAE) averages absolute errors instead. Both reveal magnitude without the signed direction of error, so calibration and mean error should be reported when systematic overprediction or underprediction matters.

MSE strongly penalises an extreme error and may be sensitive to outliers, miscoding, or heavy-tailed costs. Its numerical value also changes when the outcome is rescaled, making raw MSE values for different outcome units directly incomparable. A low utility-level MSE does not establish that between-treatment utility differences or incremental quality-adjusted life-years (QALYs) are unbiased.

Use in health economic decisions

Utility mapping, survival prediction, demand forecasts, and cost models can use MSE as one diagnostic of prediction accuracy. Decision-makers need to know whether errors distort the particular quantities feeding a cost-effectiveness or budget-impact model, especially in patient groups whose health states, resource use, or survival differ by strategy.

For cost data with a few very expensive cases, minimising squared error may prioritise those cases while worsening typical-patient predictions; whether that is desirable depends on the use of the forecasts. The chosen evaluation set should reflect the application population, and uncertainty in a fitted model should propagate into downstream economic analysis. MSE is a statistical loss measure, not a measure of health-system efficiency or social welfare.

Source basis and further learning

The sources below anchor the metric in statistical software and in the evaluation of mapped health-state utilities. Their application depends on a clear outcome scale and an honest separation of training and validation data.

• scikit-learn developers, official mean_squared_error metric documentation. This documents the regression loss and evaluation inputs.
• NICE Decision Support Unit, Technical Support Document 10, The Use of Mapping Methods to Estimate Health State Utility Values. This discusses predictive error, external validation, and health-utility mapping.
• Longworth and Rowen (2013), “Mapping to Obtain EQ-5D Utility Values for Use in NICE Health Technology Assessments”, Value in Health. This addresses MSE and RMSE in mapping assessment.