Abstract
We focus on forecast evaluation techniques that comply with the design principles of Industry 4.0 (or I4.0). The I4.0 concept refers to trends and principles attributed to the 4th industrial revolution and, in particular, assumes the following capabilities of automated systems: interoperability, decentralization, real-time processing, and service-orientation. Generally, effective forecast evaluation requires us to store both actuals and forecasts. We look at how to handle rolling-origin forecasts produced for many series over multiple horizons. This setup is met both in research (e.g., in forecasting competitions or when proposing a new method) and in practice (when tracking/reporting forecasting performance). We show how to ensure access to all the variables required for exploratory analysis and performance measurement. We propose flexible yet simple and effective data schemas allowing the storage and exchange of actuals, forecasts, and any additional relevant info. We show how to construct various tools for forecast exploration and evaluation using the schemas proposed. In particular, we present our implementation of a prediction-realization diagram showing forecasts from different methods on one plot. We propose special tools for measuring the quality of point and interval rolling-origin predictions across many time series and over multiple horizons. The workflow for using techniques proposed is illustrated using R codes.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Makridakis, S., Hibon, M.: The M3-Competition: results, conclusions and implications. Int. J. Forecast. 16(4), 451–476 (2000)
Makridakis, S., Spiliotis, E., Assimakopoulos, V.: The M4 competition: results, findings, conclusion and way forward. Int. J. Forecast. 34(4), 802–808 (2018)
Davydenko, A., Fildes, R.: Measuring forecasting accuracy: the case of judgmental adjustments to SKU-level demand forecasts. Int. J. Forecast. 29(3), 510–522 (2013)
Davydenko, A., Sai, C., & Shcherbakov, M. (2020, September 14–16). Data formats and visual tools for forecast evaluation in cyber-physical system design [conference presentation]. International scientific multiconference “Cyber-physical systems design and modelling” CyberPhy-2020, Kazan, Russia. https://doi.org/10.6084/m9.figshare.12981329.
Davydenko, A., Charith, K. (2020, July 29–30).: A Visual Framework for Longitudinal and Panel Studies (with Examples in R) [ePoster]. IRCUWU2020. https://doi.org/10.6084/m9.figshare.12749432
Davydenko, A., Fildes, R.: A joint Bayesian forecasting model of judgment and observed data (LUMS Working Paper 2012:4). Lancaster University, The Department of Management Science (2012). https://www.researchgate.net/publication/282136270_A_joint_Bayesian_forecasting_model_of_judgment_and_observed_data
Industry 4.0: Definition, Design Principles, Challenges, and the Future of Employment. https://www.cleverism.com/industry-4-0/. Accessed 14 Jun 2020.
Davydenko, A., Fildes, R.: Forecast error measures: critical review and practical recommendations. In: Business forecasting: practical problems and solutions. Wiley, Hoboken, NJ (2016)
Armstrong, J.S.: Principles of Forecasting: A Handbook for Researchers and Practitioners. Kluwer Academic, Boston, MA (2001)
Hyndman, R.: Cross-validation for time series [Blog post] (2016, December 5). Retrieved from https://robjhyndman.com/hyndsight/tscv/
Glossary: Density forecast. https://ec.europa.eu/eurostat/statistics-explained/index.php/Glossary:Density_forecast. Accessed 14 Jul 2020.
Hyndman, R.: Mcomp: Data from the M-Competitions. R package version 2.8 (2018). CRAN.R-project.org/package=Mcomp
Ellis, P: Tcomp: Data from the 2010 Tourism Forecasting Competition. R package version 1.0.1 (2018). CRAN.R-project.org/package=Tcomp
Hyndman, R.: tscompdata: Time series data from various forecasting competitions. R package version 0.0.1 (2018). github.com/robjhyndman/tscompdata
Montero-Manso, P., Netto, C., Talagala, T.: M4comp2018: Data from the M4-Competition. R package version 0.1.0 (2018). github.com/carlanetto/M4comp2018
Sai, C., Davydenko, A., Shcherbakov, M.: Forvision: Tools for forecast visualisation and evaluation. R package version 0.0.1 (2019). github.com/forvis/forvision
Theil, H.: Applied economic forecasting. North-Holland, Amsterdam (1996)
Hyndman, R., Koehler, A.: Another look at measures of forecast accuracy. Int. J. Forecast. 22(4), 679–688 (2006)
Davydenko, A., Fildes, R.: Measuring forecasting accuracy: problems and recommendations (by the example of SKU-level judgmental adjustments). In: Intelligent Fashion Forecasting Systems: Models and Applications, pp. 43–70. Springer, Berlin Heidelberg (2014).
Zellner, A.: An Introduction to Bayesian Inference in Econometrics. Wiley Classics Library. Wiley, New York (1996)
Davydenko, A.: Integration of judgmental and statistical approaches for demand forecasting: Models and methods. PhD Thesis. Lancaster University, UK (2012). https://www.researchgate.net/publication/338885739_Integration_of_judgmental_and_statistical_approaches_for_demand_forecasting_Models_and_methods
Goodwin, P.: Profit from your Forecasting Software: A Best Practice Guide for Sales Forecasters. Wiley and SAS Business Series, Wiley, Hoboken, New Jersey (2018)
Gilliland, M.: Forecast value added analysis: Step-by-step. SAS Institute whitepaper (2008)
Ma, S., Fildes, R., Huang, T.: Demand forecasting with high dimensional data: The case of SKU retailsales forecasting with intra- and inter-category promotional information. Eur. J. Oper. Res. 249(1), 245–257 (2016)
Huang, T., Fildes, R., Soopramanien, D.: Forecasting retailer product sales in the presence of structural change. Eur. J. Oper. Res. 279(2), 459–470 (2019)
Fildes, R., & Goodwin, P. (2020). Stability in the inefficient use of forecasting systems: A case study in a supply chain company. International Journal of Forecasting. https://doi.org/10.1016/j.ijforecast.2020.11.004.
Spiliotis, E., Petropoulos, F., Kourentzes, N., & Assimakopoulos, V. (2020). Cross-temporal aggregation: Improving the forecast accuracy of hierarchical electricity consumption. Applied Energy, 261, 114339. https://doi.org/10.1016/j.apenergy.2019.114339.
Chen C., Twycross J., Garibaldi J.M.: A new accuracy measure based on bounded relative error for time series forecasting. PloS ONE 12(3), e0174202 (2017). https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0174202
Athanasopoulos, G., Hyndman, R.J., Song, H., Wu, D.C.: The tourism forecasting competition. Int. J. Forecast. 27(3), 822–844 (2011)
Acknowledgements
The reported study was supported by RFBR research projects 19-47-340010 r_a.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2021 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Davydenko, A., Sai, C., Shcherbakov, M. (2021). Forecast Evaluation Techniques for I4.0 Systems. In: Kravets, A.G., Bolshakov, A.A., Shcherbakov, M. (eds) Cyber-Physical Systems: Modelling and Intelligent Control. Studies in Systems, Decision and Control, vol 338. Springer, Cham. https://doi.org/10.1007/978-3-030-66077-2_7
Download citation
DOI: https://doi.org/10.1007/978-3-030-66077-2_7
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-66076-5
Online ISBN: 978-3-030-66077-2
eBook Packages: EngineeringEngineering (R0)