Signature
e_1 = y_1 - f_1; e_2 = y_2 - f_2; e_3 = y_3 - f_3; MAE = (abs(e_1) + abs(e_2) + abs(e_3)) / 3; MAPE = 100 / 3 * (abs(e_1 / y_1) + abs(e_2 / y_2) + abs(e_3 / y_3))
| Inputs | Definition | Unit |
|---|---|---|
y_1 | Observed use in test period 1 | services per period |
f_1 | Forecast made from the training data for test period 1 | services per period |
y_2 | Observed use in test period 2 | services per period |
f_2 | Forecast made from the training data for test period 2 | services per period |
y_3 | Observed use in test period 3 | services per period |
f_3 | Forecast made from the training data for test period 3 | services per period |
e_1 | Observed minus forecast in test period 1 | services per period |
|---|---|---|
e_2 | Observed minus forecast in test period 2 | services per period |
e_3 | Observed minus forecast in test period 3 | services per period |
MAE | Average absolute forecast error over the test periods | services per period |
MAPE | Average absolute error as a percentage of the observed value | per cent |
Function
Forecasting health service use and measuring forecast accuracy
Maps past use and projected populations to forecasts of future service use, by applying age-specific use rates to a projected population or by smoothing a time series of activity, and compares forecasts with data held back from fitting to measure their accuracy. The notation follows the Demand Forecasting article and its ageing district.
Try this function
Implementations
Excel
MAE and MAPE from named observed and forecast ranges
With the test-period observations in TestObs and the forecasts in Forecasts, the formulas return the mean absolute error and the mean absolute percentage error, held in MeanAbsErr and MeanAbsPctErr (dynamic-array Excel; older versions need array entry).
=AVERAGE(ABS(TestObs-Forecasts)); =100*AVERAGE(ABS((TestObs-Forecasts)/TestObs))
Assumptions
Test periods not used to fit the forecasting method
Accuracy is measured on data held back from fitting; errors on the training data overstate how well the method forecasts.
Observed values above zero for the percentage error
Every observed value in the test periods is positive and the data have a meaningful zero; otherwise the MAPE is undefined or misleading and the MAE or a scaled error is used.
Worked examples
Three months of admissions against a flat forecast of 2,030
Observed admissions of 2,050, 1,980 and 2,200 give errors of 20, minus 50 and 170, an MAE of 80 admissions and a MAPE of about 3.7427 per cent (figures illustrative).
y_1 = 2050; y_2 = 1980; y_3 = 2200; f_1 = 2030; f_2 = 2030; f_3 = 2030; e_1 = 20; e_2 = -50; e_3 = 170; MAE = 80; MAPE = 3.7427
Three months of a rare procedure against a flat forecast of 2.5
Counts of 2, 1 and 4 give errors of minus 0.5, minus 1.5 and 1.5, an MAE of about 1.1667 procedures but a MAPE of about 70.8333 per cent, driven by the month with a count of 1 (figures illustrative).
y_1 = 2; y_2 = 1; y_3 = 4; f_1 = 2.5; f_2 = 2.5; f_3 = 2.5; e_1 = -0.5; e_2 = -1.5; e_3 = 1.5; MAE = 1.1667; MAPE = 70.8333
Common errors
Using the MAPE for small or zero counts
Percentage errors are infinite or undefined when an observed value is zero and extreme when it is close to zero, a real problem for rare procedures; the example's MAPE of about 71 per cent rests on one month with a single procedure.
Comparing MAE across series in different units
Errors on the scale of the data cannot be used to compare series that involve different units, such as admissions and bed-days; percentage or scaled errors are needed for that comparison.
Measuring accuracy on the data used to fit the method
A method that fits the training data well will not necessarily forecast well, so accuracy is judged only on data held back from fitting.
Sources
Forecast errors, MAE and percentage errors in Forecasting: Principles and Practice
Hyndman RJ, Athanasopoulos G. Forecasting: Principles and Practice. 3rd ed. Melbourne: OTexts; 2021. Section 5.8, evaluating point forecast accuracy: accuracy can only be determined on new data not used when fitting; the forecast error is the observed value minus the forecast; MAE is the mean of the absolute errors; measures based on the errors are scale-dependent and cannot compare series with different units; percentage errors are infinite or undefined if y_t is 0 and extreme if it is close to zero.
Mean absolute error of 1.9 percentage points in Stockholm's expenditure forecasts
Linnér L, Eriksson I, Persson M, Wettermark B. Forecasting drug utilization and expenditure: ten years of experience in Stockholm. BMC Health Services Research. 2020;20:410. doi:10.1186/s12913-020-05170-0 (abstract read). Abstract: forecasts of total pharmaceutical expenditure were estimated to increase between 2 and 8 per cent annually, and their accuracy varied over the years with a mean absolute error of 1.9 percentage points; forecasts for the same year were more accurate than forecasts for the next year.
Canonical Identity
Stable URI · Machine-readable · Resolvable · CC BY 4.0