Abstract
Recurrent neural networks (RNNs) have evolved to become one of the most powerful tools for making predictions on sequenced data, such as time series, textual data, signals, music etc. In many real-life cases, however, sequenced data are additionally characterized by static features which, due to their non-sequential nature, cannot be transferred directly into RNNs. In this paper, we discuss a method which incorporates static features into RNNs in order to influence and generalize the learning process. Furthermore, we will demonstrate that our approach significantly enhances the performance of RNNs, enabling the networks to learn the sequenced data exhibiting varying characteristics and then distinguish between them through the use of static supplementary information. Finally, we will evaluate our model against real energy consumption measurements of energy time series and verify that high-accuracy demand forecasts for different types of customers can be achieved only by way of incorporation of static features.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Bengio Y, Frasconi P, Simard P (1993) The problem of learning long-term dependencies in recurrent networks. In: IEEE international conference on neural networks, IEEE. https://doi.org/10.1109/ICNN.1993.298725, https://ieeexplore.ieee.org/document/298725
Breiman L (2001) Random forests. Mach Learn 45(1):5–32. https://doi.org/10.1023/A:1010933404324
Chen D, Li S, Lin FJ (2019) New super-twisting zeroing neural-dynamics model for tracking control of parallel robots: a finite-time and robust solution. IEEE Trans Cybern. https://doi.org/10.1109/TCYB.2019.2930662
Chen D, Li S, Lin FJ, Wu Q (2019b) Rejecting chaotic disturbances using a super-exponential-zeroing neurodynamic approach for synchronization of chaotic sensor systems. IEEE Trans Cybern 19(1):74. https://doi.org/10.3390/s19010074
Chen D, Li S, Wu Q, Luo X (2019) New disturbance rejection constraint for redundant robot manipulators: an optimization perspective. IEEE Trans Ind Inf. https://doi.org/10.1109/TII.2019.2930685
Dudek G (2015) Short-term load forecasting using random forests, vol 323. Springer, Berlin, pp 821–828
Eck D, Schmidhuber J (2002) Learning the long-term structure of the blues, vol 2415. Springer, Berlin, pp 284–289
Esteban C, Staeck O, Yang Y, Tresp V (2016) Predicting clinical events by combining static and dynamic information using recurrent neural networks. arXiv: 1602.02685 [cs], arXiv:1602.02685
Feilat EA, Bouzguenda M (2011) Medium-term load forecasting using neural network approach. In: 2011 IEEE PES conference on innovative smart grid technologies—Middle East, IEEE, pp 1–5. https://doi.org/10.1109/ISGT-MidEast.2011.6220810, http://ieeexplore.ieee.org/document/6220810/
Gers FA, Schmidhuber J, Cummins F (2000) Learning to forget: continual prediction with LSTM. Neural Comput 12(10):2451–2471. https://doi.org/10.1162/089976600300015015
González-Romera E, Jaramillo-Morán M, Carmona-Fernández D (2008) Monthly electric energy demand forecasting with neural networks and fourier series. Energy Convers Manag 49(11):3135–3142. https://doi.org/10.1016/j.enconman.2008.06.004
Graves A, Fernández S, Liwicki M, Bunke H, Schmidhuber J (2007) Unconstrained online handwriting recognition with recurrent neural networks. In: NIPS’07 Proceedings of the 20th international conference on neural information processing systems, Curran Associates Inc., https://dl.acm.org/citation.cfm?id=2981562.2981635
Graves A, Mohamed A, Hinton GE (2013) Speech recognition with deep recurrent neural networks. In: 2013 IEEE international conference on acoustics, speech and signal processing, pp 6645–6649
Hochreiter S, Schmidhuber J (1997) Long short-term memory. Neural Comput 9(8):1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735
Kaytez F, Taplamacioglu MC, Cam E, Hardalac F (2015) Forecasting electricity consumption: a comparison of regression analysis, neural networks and least squares support vector machines. Int J Electr Power Energy Syst 67:431–438. https://doi.org/10.1016/j.ijepes.2014.12.036
Khosravani H, Castilla M, Berenguel M, Ruano A, Ferreira P (2016) A comparison of energy consumption prediction models based on neural networks of a bioclimatic building. Energies 9(1):57. https://doi.org/10.3390/en9010057
Kuo PH, Huang CJ (2018) A high precision artificial neural networks model for short-term energy load forecasting. Energies 11(1):213. https://doi.org/10.3390/en11010213
Leontjeva A, Kuzovkin I (2016) Combining static and dynamic features for multivariate sequence classification. In: 2016 IEEE international conference on data science and advanced analytics (DSAA) pp. 21–30. https://doi.org/10.1109/DSAA.2016.10, arXiv: 1712.08160
Liang Y, Niu D, Ye M, Hong WC (2016) Short-term load forecasting based on wavelet transform and least squares support vector machine optimized by improved cuckoo search. Energies 9(10):827. https://doi.org/10.3390/en9100827
Liu N, Tang Q, Zhang J, Fan W, Liu J (2014) A hybrid forecasting model with parameter optimization for short-term load forecasting of micro-grids. Appl Energy 129:336–345. https://doi.org/10.1016/j.apenergy.2014.05.023
Mayer H, Gome F, Wierstra D, Nagy I, Knoll A, Schmidhuber J (2006) A system for robotic heart surgery that learns to tie knots using recurrent neural networks. In: 2006 IEEE/RSJ international conference on intelligent robots and systems. https://doi.org/10.1109/IROS.2006.282190
Niu D, Dai S (2017) A short-term load forecasting model with a modified particle swarm optimization algorithm and least squares support vector machine based on the denoising method of empirical mode decomposition and grey relational analysis. Energies 10(3):408. https://doi.org/10.3390/en10030408
Niu D, Shi H, Wu DD (2012) Short-term load forecasting using bayesian neural networks learned by hybrid Monte Carlo algorithm. Appl Soft Comput 12(6):1822–1827. https://doi.org/10.1016/j.asoc.2011.07.001
Ringwood JV, Bofelli D, Murray FT (2001) Forecasting electricity demand on short, medium and long time scales using neural networks. J Intell Rob Syst 31(1):129–147. https://doi.org/10.1023/A:1012046824237
Rumelhart DE, Hinton GE, Williams RJ (1986) Learning representations by back-propagating errors. Nature 323:533–536. https://doi.org/10.1038/323533a0
Salehinejad H, Sankar S, Barfett J, Colak E, Valaee S (2018) Recent advances in recurrent neural networks. arXiv:1801.01078
Selakov A, Cvijetinović D, Milović L, Mellon S, Bekut D (2014) Hybrid pso-svm method for short-term load forecasting during periods with significant temperature variations in city of burbank. Appl Soft Comput 16:80–88. https://doi.org/10.1016/j.asoc.2013.12.001
Suganthi L, Iniyan S, Samuel AA (2015) Applications of fuzzy logic in renewable energy systems—a review. Renew Sustain Energy Rev 48:585–607. https://doi.org/10.1016/j.rser.2015.04.037
Tanti M, Gatt A, Camilleri KP (2018) Where to put the image in an image caption generator. Nat Lang Eng 24(3):467–489. https://doi.org/10.1017/S1351324918000098
Vinyals O, Toshev A, Bengio S, Erhan D (2014) Show and tell: a neural image caption generator. arXiv:1411.4555
Wu Y, Schuster M, Chen Z, Le QW, Norouzi M, Macherey W, Krikun M, Cao Y, Gao Q, Macherey K, Klingner J, Shah A, Johnson M, Liu X, Kaiser L, Gouws S, Kato Y, Kudo T, Kazawa H, Stevens K, Kurian G, Patil N, Wang W, Young C, Smith J, Riesa J, Rudnick A, Vinyals O, Corrado GS, Hughes M, Dean J (2016) Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv:1609.08144
Yang Y, Fasching PA, Tresp V (2017) Predictive modeling of therapy decisions in metastatic breast cancer with recurrent neural network encoder and multinomial hierarchical regression decoder. In: 2017 IEEE international conference on healthcare informatics (ICHI), IEEE, pp 46–55. https://doi.org/10.1109/ICHI.2017.51, http://ieeexplore.ieee.org/document/8031131/
Zheng H, Yuan J, Chen L (2017) Short-term load forecasting using EMD-LSTM neural networks with a Xgboost algorithm for feature importance evaluation. Energies 10(8):1168. https://doi.org/10.3390/en10081168
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
About this article
Cite this article
Miebs, G., Mochol-Grzelak, M., Karaszewski, A. et al. Efficient Strategies of Static Features Incorporation into the Recurrent Neural Network. Neural Process Lett 51, 2301–2316 (2020). https://doi.org/10.1007/s11063-020-10195-x
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11063-020-10195-x