[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

DTW-NN: : A novel neural network for time series recognition using dynamic alignment between inputs and weights

Published: 05 January 2020 Publication History

Abstract

This paper describes a novel model for time series recognition called a Dynamic Time Warping Neural Network (DTW-NN). DTW-NN is a feedforward neural network that exploits the elastic matching ability of DTW to dynamically align the inputs of a layer to the weights. This weight alignment replaces the standard dot product within a neuron with DTW. In this way, the DTW-NN is able to tackle difficulties with time series recognition such as temporal distortions and variable pattern length within a feedforward architecture. We demonstrate the effectiveness of DTW-NNs on four distinct datasets: online handwritten characters, accelerometer-based active daily life activities, spoken Arabic numeral Mel-Frequency Cepstrum Coefficients (MFCC), and one-dimensional centroid-radii sequences from leaf shapes. We show that the proposed method is an effective general approach to temporal pattern learning by achieving state-of-the-art results on these datasets.

Highlights

Proposes Dynamic Time Warping Neural Network, a feed-forward network for time series.
DTW-NN uses a temporal kernel-like function in replace of a typical inner product.
DTW-NN uses dynamic programming to align weights and inputs.
We evaluate on Unipen 1a, 1b, 1c, UCI Spoken Arabic, UCI ADL, and Flavia leaf shapes.

References

[1]
Xing Z., Pei J., Keogh E., A brief survey on sequence classification, ACM SIGKDD Explor. Newsl. 12 (1) (2010) 40,.
[2]
Felzenszwalb P.F., Zabih R., Dynamic programming and graph algorithms in computer vision, IEEE Trans. Pattern Anal. Mach. Intell. 33 (4) (2011) 721–740,.
[3]
X. Xi, E. Keogh, C. Shelton, L. Wei, C.A. Ratanamahatana, Fast time series classification using numerosity reduction, in: Int. Conf. Mach. Learning, 2006, pp. 1033–1040, https://doi.org/10.1145/1143844.1143974.
[4]
Ding H., Trajcevski G., Scheuermann P., Wang X., Keogh E., Querying and mining of time series data, Proc. Very Large Data Base Endow. 1 (2) (2008) 1542–1552,.
[5]
T. Rakthanmanon, B. Campana, A. Mueen, G. Batista, B. Westover, Q. Zhu, J. Zakaria, E. Keogh, Searching and mining trillions of time series subsequences under dynamic time warping, in: Int. Conf. Knowl. Discovery and Data Mining, 2012, pp. 262–270, https://doi.org/10.1145/2339530.2339576.
[6]
Schmidhuber J., Deep learning in neural networks: An overview, Neural Netw. 61 (2015) 85–117,.
[7]
Touvron H., Vedaldi A., Douze M., Jégou H., Fixing the train-test resolution discrepancy, CoRR abs/1906.06423 (2019) URL http://arxiv.org/abs/1906.06423.
[8]
S. Uchida, S. Ide, B.K. Iwana, A. Zhu, A further step to perfect accuracy by training CNN with larger data, in: Int. Conf. Frontiers in Handwriting Recognition, 2016, pp. 405–410, https://doi.org/10.1109/ICFHR.2016.0082.
[9]
K. Kowsari, M. Heidarysafa, D.E. Brown, K.J. Meimandi, L.E. Barnes, RMDL: Random multimodel deep learning for classification, in: Int. Conf. Inform. Sys. and Data Mining, 2018, https://doi.org/10.1145/3206098.3206111.
[10]
Huang Y., Cheng Y., Chen D., Lee H., Ngiam J., Le Q.V., Chen Z., Efficient training of giant neural networks using pipeline parallelism, CoRR abs/1811.06965 (2019) URL http://arxiv.org/abs/1811.06965.
[11]
M. Banko, E. Brill, Scaling to very very large corpora for natural language disambiguation, in: Annu. Meeting Assoc. for Computational Linguistics, 2001, pp. 26–33, https://doi.org/10.3115/1073012.1073017.
[12]
Krizhevsky A., Sutskever I., Hinton G.E., Magenet classification with deep convolutional neural networks, in: Advances in Neural Inform. Process. Syst., 2012.
[13]
LeCun Y., Bottou L., Bengio Y., Haffner P., Gradient-based learning applied to document recognition, Proc. IEEE 86 (11) (1998) 2278–2324,.
[14]
A. Graves, A.-R. Mohamed, G. Hinton, Speech recognition with deep recurrent neural networks,in: Int. Conf. Acoustics, Speech and Signal Process., 2013, pp. 6645–6649, https://doi.org/10.1109/icassp.2013.6638947.
[15]
A. Graves, N. Jaitly, Towards end-to-end speech recognition with recurrent neural networks, in: Int. Conf. Mach. Learning, 2014, pp. 1764–1772.
[16]
Sundermeyer M., Schlüter R., Ney H., LSTM neural networks for language modeling, in: Interspeech, 2010, pp. 194–197.
[17]
Bengio Y., Simard P., Frasconi P., Learning long-term dependencies with gradient descent is difficult, IEEE Trans. Neural Netw. 5 (2) (1994) 157–166,.
[18]
B.K. Iwana, V. Frinken, S. Uchida, A robust dissimilarity-based neural network for temporal pattern recognition, in: Int. Conf. Frontiers in Handwriting Recognition, 2016, pp. 265–270 https://doi.org/10.1109/ICFHR.2016.0058.
[19]
Zhang G., Patuwo B.E., Hu M.Y., Forecasting with artificial neural networks:: The state of the art, Int. J. Forecast. 14 (1) (1998) 35–62,.
[20]
Hwarng H.B., Ang H.T., A simple neural network for ARMA, (p, q) time series, Omega 29 (4) (2001) 319–333,.
[21]
Zhang G.P., Time series forecasting using a hybrid ARIMA and neural network model, Neurocomputing 50 (2003) 159–175,.
[22]
Jaeger H., Tutorial on Training Recurrent Neural Networks, Covering BPPT, RTRL, EKF and the Echo State Network Approach, German National Research Center for Inform, 2002.
[23]
Hochreiter S., Schmidhuber J., Long short-term memory, Neural Comput. 9 (8) (1997) 1735–1780,.
[24]
Hochreiter S., Bengio Y., Frasconi P., Schmidhuber J., Gradient Flow in Recurrent Nets: the Difficulty of Learning Long-Term Dependencies, Wiley-IEEE Press, 2001,.
[25]
Graves A., Liwicki M., Fernandez S., Bertolami R., Bunke H., Schmidhuber J., A novel connectionist system for unconstrained handwriting recognition, IEEE Trans. Pattern Anal. Mach. Intell. 31 (5) (2009) 855–868,.
[26]
Graves A., Schmidhuber J., Offline handwriting recognition with multidimensional recurrent neural networks, in: Advances in Neural Inform. Process. Syst, 2009, pp. 545–552.
[27]
K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using RNN encoder–decoder for statistical machine translation, in: Conf. Empirical Methods in Nat. Lang. Process., 2014, pp. 1724–1734 https://doi.org/10.3115/v1/d14-1179.
[28]
Jaeger H., Haas H., Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication, Science 304 (5667) (2004) 78–80,.
[29]
Chung J., Gulcehre C., Cho K., Bengio Y., Empirical evaluation of gated recurrent neural networks on sequence modeling, CoRR abs/1412.3555 (2014) URL http://arxiv.org/abs/1412.3555.
[30]
M. Čerňanský, P. Tiňo, Comparison of echo state networks with simple recurrent networks and variable-length markov models on symbolic sequences, in: Lecture Notes in Computer Science, Springer, 2007, pp. 618–627 https://doi.org/10.1007/978-3-540-74690-4_63.
[31]
Tanisaro P., Heidemann G., Time series classification using time warping invariant echo state networks, in: IEEE Int. Conf. Mach. Learning and Appl, IEEE, 2016, pp. 831–836,.
[32]
Waibel A., Hanazawa T., Hinton G., Shikano K., Lang K.J., Phoneme recognition using time-delay neural networks, in: Readings in Speech Recog., Elsevier, 1990, pp. 393–404,.
[33]
Lang K.J., Waibel A.H., Hinton G.E., A time-delay neural network architecture for isolated word recognition, Neural Netw. 3 (1) (1990) 23–43,.
[34]
Wöhler C., Anlauf J.K., An adaptable time-delay neural-network algorithm for image sequence analysis, IEEE Trans. Neural Netw. 10 (6) (1999) 1531–1536,.
[35]
Y. Zheng, Q. Liu, E. Chen, Y. Ge, J.L. Zhao, Time series classification using multi-channels deep convolutional neural networks, in: Int. Conf. Web-Age Inform. Management, 2014, pp. 298–310 https://doi.org/10.1007/978-3-319-08010-9_33.
[36]
Razavian N., Sontag D., Temporal convolutional neural networks for diagnosis from lab tests, CoRR abs/1511.07938 (2015) URL http://arxiv.org/abs/1511.07938.
[37]
J. Yang, M.N. Nguyen, P.P. San, X. Li, S. Krishnaswamy, Deep convolutional neural networks on multichannel time series for human activity recognition, in: Int. Joint Conf. Artificial Intell., 2015, pp. 3995–4001.
[38]
Zhang A., Zhu W., Li J., Spiking echo state convolutional neural network for robust time series classification, IEEE Access (2018) 1,.
[39]
v. d. Oord A., Dieleman S., Zen H., Simonyan K., Vinyals O., Graves A., Kalchbrenner N., Senior A., Kavukcuoglu K., Wavenet: A generative model for raw audio, CoRR abs/1609.03499 (2016) URL http://arxiv.org/abs/1609.03499.
[40]
Bai S., Kolter J.Z., Koltun V., An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, CoRR abs/1803.01271 (2019) URL http://arxiv.org/abs/1803.01271.
[41]
A. Ahadi, X. Liang, Wind speed time series predicted by neural network, in: IEEE Canadian Conf. Electrical & Computer Eng., 2018, pp. 264–269, https://doi.org/10.1109/ccece.2018.8447635.
[42]
Qin M., Li Z., Du Z., Red tide time series forecasting by combining ARIMA and deep belief network, Knowl.-Based Syst. 125 (2017) 39–52,.
[43]
Kuremoto T., Kimura S., Kobayashi K., Obayashi M., Time series forecasting using a deep belief network with restricted boltzmann machines, Neurocomputing 137 (2014) 47–56,.
[44]
Alexandridis A.K., Zapranis A.D., Wavelet neural networks: A practical guide, Neural Netw. 42 (2013) 1–27,.
[45]
Song G., Dai Q., A novel double deep ELMs ensemble system for time series forecasting, Knowl.-Based Syst. 134 (2017) 31–49,.
[46]
Maass W., Networks of spiking neurons: the third generation of neural network models, Neural Netw. 10 (9) (1997) 1659–1671,.
[47]
Brette R., Rudolph M., Carnevale T., Hines M., Beeman D., Bower J.M., Diesmann M., Morrison A., Goodman P.H., Harris F.C., et al., Simulation of networks of spiking neurons: a review of tools and strategies, J. Comput. Neurosci. 23 (3) (2007) 349–398,.
[48]
N. Kalchbrenner, E. Grefenstette, P. Blunsom, A convolutional neural network for modelling sentences, in: Annu. Meeting Assoc. Computational Linguistics, 2014, pp. 655–665, https://doi.org/10.3115/v1/p14-1062.
[49]
J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, Y. Wei, Deformable convolutional networks, in: IEEE Int. Conf. Computer Vision, 2017, pp. 764–773, https://doi.org/10.1109/iccv.2017.89.
[50]
Yu Z., Niu Z., Tang W., Wu Q., Deep learning for daily peak load forecasting–a novel gated recurrent neural network combining dynamic time warping, IEEE Access 7 (2019) 17184–17194,.
[51]
K. Xu, J. Ba, R. Kiros, K. Cho, A. Courville, R. Salakhudinov, R. Zemel, Y. Bengio, Show, attend and tell: Neural image caption generation with visual attention, in: Int. Conf. Mach. learning, 2015, pp. 2048–2057.
[52]
Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I., Attention is all you need, in: Advances in Neural Inform. Process. Sys., 2017, pp. 5998–6008.
[53]
Shi K., Tang Y., Zhong S., Yin C., Huang X., Wang W., Nonfragile asynchronous control for uncertain chaotic lurie network systems with bernoulli stochastic process, Int. J. Robust Nonlinear Control 28 (5) (2017) 1693–1714,.
[54]
Wang J., Shi K., Huang Q., Zhong S., Zhang D., Stochastic switched sampled-data control for synchronization of delayed chaotic neural networks with packet dropout, Appl. Math. Comput. 335 (2018) 211–230,.
[55]
Shi K., Wang J., Zhong S., Zhang X., Liu Y., Cheng J., New reliable nonuniform sampling control for uncertain chaotic neural networks under markov switching topologies, Appl. Math. Comput. 347 (2019) 169–193,.
[56]
Lin J., Rao Y., Lu J., Zhou J., Runtime neural pruning, in: Advances in Neural Inform. Process. Sys., 2017, pp. 2181–2191.
[57]
Aghasi A., Abdi A., Nguyen N., Romberg J., Net-trim: Convex pruning of deep neural networks with performance guarantee, in: Advances in Neural Inform. Process. Sys., 2017, pp. 3177–3186.
[58]
Itakura F., Minimum prediction residual principle applied to speech recognition, IEEE Trans. Acoust. Speech Signal Process. 23 (1) (1975) 67–72,.
[59]
Haussler D., Convolution Kernels on Discrete Structures, Depart. Comput. Sci. University of California, Santa Cruz, 1999.
[60]
Aizerman A., Braverman E.M., Rozoner L., Theoretical foundations of the potential function method in pattern recognition learning, Autom. Remote Control 25 (1964) 821–837.
[61]
B.E. Boser, I.M. Guyon, V.N. Vapnik, A training algorithm for optimal margin classifiers, in: Annu. Workshop Computational Learning Theory, 1992, pp. 144–152, https://doi.org/10.1145/130385.130401.
[62]
Schölkopf B., Smola A., Müller K.-R., Nonlinear component analysis as a kernel eigenvalue problem, Neural Comput. 10 (5) (1998) 1299–1319,.
[63]
Iwana B.K., Frinken V., Riesen K., Uchida S., Efficient temporal pattern recognition by means of dissimilarity space embedding with discriminative prototypes, Pattern Recognit. 64 (2017) 268–276,.
[64]
B.J. Jain, S. Spiegel, Time series classification in dissimilarity spaces, in: Int. Workshop Adv. Anal. and Learning Temporal Data, 2015, pp. 71–76.
[65]
Chaovalitwongse W., Pardalos P., On the time series support vector machine using dynamic time warping kernel for brain activity classification, Cybern. Syst. Anal. 44 (1) (2008) 125–138,.
[66]
Shimodaira H., i. Noma K., Nakai M., Sagayama S., Dynamic time-alignment kernel in support vector machine, in: Advances in Neural Inform. Process. Syst., 2002, pp. 921–928.
[67]
C. Bahlmann, B. Haasdonk, H. Burkhardt, Online handwriting recognition with support vector machines-a kernel approach, in: Int. Workshop Frontiers in Handwriting Recognition, 2002, pp. 49–54, https://doi.org/10.1109/iwfhr.2002.1030883.
[68]
C.A. Ratanamahatana, E. Keogh, Everything you know about dynamic time warping is wrong, in: Workshop Mining Temporal and Sequential Data, 2004, pp. 53–63.
[69]
I. Guyon, L. Schomaker, R. Plamondon, M. Liberman, S. Janet, Unipen project of on-line data exchange and recognizer benchmarks, in: Int. Conf. Pattern Recognition, Vol. 2, 1994, pp. 29–33, https://doi.org/10.1109/icpr.1994.576870.
[70]
B. Bruno, F. Mastrogiovanni, A. Sgorbissa, T. Vernazza, R. Zaccaria, Analysis of human behavior recognition algorithms based on acceleration data, in: IEEE Int. Conf. Robotics and Automation, 2013, pp. 1602–1607, https://doi.org/10.1109/icra.2013.6630784.
[71]
N. Hammami, M. Sellam, Tree distribution classifier for automatic spoken Arabic digit recognition, in: Int. Conf. Internet Technology and Secured Trans., 2009, pp. 1–4, https://doi.org/10.1109/icitst.2009.5402575.
[72]
S.G. Wu, F.S. Bao, E.Y. Xu, Y.-X. Wang, Y.-F. Chang, Q.-L. Xiang, A leaf recognition algorithm for plant classification using probabilistic neural network, in: Int. Symp. Signal Process. and Inform. Technology, 2007, pp. 11–16, https://doi.org/10.1109/isspit.2007.4458016.
[73]
LeCun Y.A., Bottou L., Orr G.B., Müller K.-R., Efficient backprop, in: Neural Networks: Tricks of the Trade, Springer, 2012, pp. 9–48,.
[74]
S. Ioffe, C. Szegedy, Batch normalization: Accelerating deep network training by reducing internal covariate shift, in: Int. Conf. Machine Learning, Vol. 37, 2015, pp. 448–456.
[75]
Hu J., Lim S.G., Brown M.K., Writer independent on-line handwriting recognition using an HMM approach, Pattern Recognit. 33 (1) (2000) 133–147,.
[76]
J.-F. Hébert, M. Parizeau, N. Ghazzali, A new fuzzy geometric representation for online isolated character recognition, in: Int. Conf. Pattern Recognition, Vol. 2, 1998, pp. 1121–1123, https://doi.org/10.1109/icpr.1998.711891.
[77]
Kanna K.R., Sugumaran V., Vijayaram T., Karthikeyan C., Activities of daily life (ADL) recognition using wrist-worn accelerometer, Int. J. Eng. Technol. 4 (3) (2016) 1406–1413.
[78]
N. Hammami, M. Bedda, Improved tree model for arabic speech recognition, in: Int. Conf. Comp. Sci. and Inform. Technology, Vol. 5, 2010, pp. 521–526, https://doi.org/10.1109/iccsit.2010.5563892.
[79]
N. Hammami, M. Bedda, F. Nadir, The second-order derivatives of mfcc for improving spoken arabic digits recognition using tree distributions approximation model and HMMs, in: Int. Conf. Commun. and Inform. Technology, 2012, pp. 1–5, https://doi.org/10.1109/iccitechnol.2012.6285769.
[80]
X. Hu, L. Zhan, Y. Xue, W. Zhou, L. Zhang, Spoken arabic digits recognition based on wavelet neural networks, in: IEEE Int. Conf. Syst., Man, and Cybern., 2011, 1481–1485, https://doi.org/10.1109/icsmc.2011.6083880.
[81]
Singh K., Gupta I., Gupta S., SVM-BDT PNN and fourier moment technique for classification of leaf shape, Int. J. Signal Process. Image Process. Pattern Recognit. 3 (4) (2010) 67–78.
[82]
Chaki J., Parekh R., Bhattacharya S., Plant leaf recognition using texture and shape features with neural classifiers, Pattern Recognit. Lett. 58 (2015) 61–68,.
[83]
D.G. Tsolakidis, D.I. Kosmopoulos, G. Papadourakis, Plant leaf recognition using Zernike moments and histogram of oriented gradients, in: Hellenic Conf. on Artificial Intell., 2014, pp. 406–417, https://doi.org/10.1007/978-3-319-07064-3_33.
[84]
The Staff of the Benjamin Rose Hospital, Multidisciplinary studies of illness in aged persons, J. Chronic Dis. 9 (1) (1959) 55–62,.
[85]
Du J.-X., Wang X.-F., Zhang G.-J., Leaf shape based plant species recognition, Appl. Math. Comput. 185 (2) (2007) 883–893,.
[86]
Q.-P. Wang, J.-X. Du, C.-M. Zhai, Recognition of leaf image based on ring projection wavelet fractal feature, in: Int. Conf. Intell. Computing, 2010, pp. 240–246, https://doi.org/10.1007/978-3-642-14932-0_30.
[87]
Chaki J., Parekh R., Plant leaf recognition using shape based features and neural network classifiers, Int. J. Adv. Comput. Sci. Appl. 2 (10) (2019) 1,.
[88]
L. Ye, E. Keogh, Time series shapelets: a new primitive for data mining, in: ACM Int. Conf. Knowledge Discovery and Data Mining, 2009, pp. 947–956, https://doi.org/10.1145/1557019.1557122.
[89]
T. Beghin, J.S. Cope, P. Remagnino, S. Barman, Shape and texture based plant leaf classification, in: Int. Conf. Advanced Concepts for Intelligent Vision Syst., 2010, pp. 345–353, https://doi.org/10.1007/978-3-642-17691-3_32.
[90]
Sakoe H., Chiba S., Dynamic programming algorithm optimization for spoken word recognition, IEEE Trans. Acoust. Speech Signal Process. 26 (1) (1978) 43–49,.

Cited By

View all

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Knowledge-Based Systems
Knowledge-Based Systems  Volume 188, Issue C
Jan 2020
541 pages

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 05 January 2020

Author Tags

  1. Neural networks
  2. Dynamic time warping
  3. Temporal kernel
  4. Time series
  5. Dynamic programming

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 07 Mar 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Language of actionsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125947264:COnline publication date: 10-Mar-2025
  • (2024)Deep Learning for Time Series Classification and Extrinsic Regression: A Current SurveyACM Computing Surveys10.1145/364944856:9(1-45)Online publication date: 25-Apr-2024
  • (2024)An accurate slicing method for dynamic time warping algorithm and the segment-level early abandoning optimizationKnowledge-Based Systems10.1016/j.knosys.2024.112231300:COnline publication date: 18-Nov-2024
  • (2024)Test Time Augmentation as a Defense Against Adversarial Attacks on Online HandwritingDocument Analysis and Recognition - ICDAR 202410.1007/978-3-031-70536-6_10(156-172)Online publication date: 30-Aug-2024
  • (2023)A new intonation quality evaluation method based on self-supervised learningJournal of Intelligent & Fuzzy Systems: Applications in Engineering and Technology10.3233/JIFS-23016545:1(989-1000)Online publication date: 1-Jan-2023
  • (2023)A DTW Approach for Complex Data A Case Study with Network Data StreamsProceedings of the 38th ACM/SIGAPP Symposium on Applied Computing10.1145/3555776.3577638(402-409)Online publication date: 27-Mar-2023
  • (2023)Experimental analysis of similarity measurements for multivariate time series and its application to the stock marketApplied Intelligence10.1007/s10489-023-04874-053:21(25450-25466)Online publication date: 1-Nov-2023
  • (2023)IMGC-GNN: A multi-granularity coupled graph neural network recommendation method based on implicit relationshipsApplied Intelligence10.1007/s10489-022-04215-753:11(14668-14689)Online publication date: 1-Jun-2023
  • (2023)Improving stock trend prediction through financial time series classification and temporal correlation analysis based on aligning change pointSoft Computing - A Fusion of Foundations, Methodologies and Applications10.1007/s00500-022-07630-727:7(3655-3672)Online publication date: 1-Apr-2023
  • (2022)Multivariate time-series classification with hierarchical variational graph poolingNeural Networks10.1016/j.neunet.2022.07.032154:C(481-490)Online publication date: 1-Oct-2022
  • Show More Cited By

View Options

View options

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media