Evaluation metrics

To evaluate the peformance of both point and probabilistic forecasts, PostForecasts.jl offers methods for calculating popular evaluation metrics.

PostForecasts.mae — Function

mae(pf::PointForecasts)

Calculate Mean Absolute Error of predictions from pf. Return the vector of MAE corresponding to each forecaster.

source

PostForecasts.mape — Function

mape(pf::PointForecasts; eps=1e-9)

Calculate Mean Absolute Percentage Error of predictions from pf. Return the vector of MAPE corresponding to each forecaster.

source

PostForecasts.smape — Function

smape(pf::PointForecasts; eps=1e-9)

Calculate Symmetric Mean Absolute Percentage Error of predictions from pf. Return the vector of SMAPE corresponding to each forecaster.

source

PostForecasts.mse — Function

mse(pf::PointForecasts)

Calculate Mean Squared Error of predictions from pf. Return the vector of MSE corresponding to each forecaster.

source

PostForecasts.pinball — Function

pinball(qf::QuantForecasts)

Calculate Pinball Loss over all quantiles in qf. Return the vector of Pinball Loss values corresponding to each quantile. See Gneiting 2011 for more details about Pinball Loss.

Note

Average Pinball Loss over equidistant quantiles approximates Continuous Ranked Probability Score.

source

PostForecasts.crps — Function

crps(qf::QuantForecasts)

Approximate Continuous Ranked Probability Score using the Pinball Loss of quantile forecasts in qf, with 2mean(pinball(qf)).

Note

Approximating CRPS with the average Pinball Loss requires a dense grid of equidistant quantiles.

source

PostForecasts.coverage — Function

coverage(qf::QuantForecasts)

Calculate empirical coverage of quantile predictions in qf. Return the vector of coverage corresponding to each quantile.

source