Abstract
Contrastive learning is extensively employed for time series representation learning. Existing contrastive learning methods often split long real-world time series records into short samples along the temporal dimension to mitigate the explosive growth in storage and enhance the effectiveness of representations. However, the segmentation in contrastive learning may result in the loss of long characteristics, such as periodicities and trends of the original time series. To address this issue, we propose LTCR to reconstruct these long temporal characteristics after the segmentation. There are two key components of LTCR. On the one hand, we propose a novel definition of positive and negative pairs for contrastive learning. We treat the samples separated by integer multiples of a long periodicity from the same record as positive pairs and the other samples from the same record as negative samples to capture the periodic similarity, where the periodicity is calculated by the analysis in the frequency domain. On the other hand, we apply relative position prediction in the representation space to mine the trend characteristics of the original records. Comprehensive experiments have been carried out on three real-world time-series datasets, AD, TDBrain, and NerveDamage, which have been widely used in the field of time series representation. The results have demonstrated that LTCR significantly enhances the effectiveness of current contrastive learning methods, working as a plug-and-play plugin. We release the code and datasets at https://anonymous.4open.science/r/LTCR-76D5.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Arsenault, C., et al.: Covid-19 and resilience of healthcare systems in ten countries. Nat. Med. 28(6), 1314–1324 (2022)
Caron, M., Bojanowski, P., Joulin, A., Douze, M.: Deep clustering for unsupervised learning of visual features. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) Computer Vision – ECCV 2018. LNCS, vol. 11218, pp. 139–156. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01264-9_9
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Chen, Z., Xu, J., Peng, T., Yang, C.: Graph convolutional network-based method for fault diagnosis using a hybrid of measurement and prior knowledge. IEEE Trans. Cybernet. 52(9), 9157–9169 (2021)
Ching, T., et al.: Opportunities and obstacles for deep learning in biology and medicine. J. R. Soc. Interface 15(141), 20170387 (2018)
Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: BERT: pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 (2018)
Dijk, H., van Wingen, G., Denys, D., Olbrich, S., Ruth, R., Arns, M.: The two decades brainclinics research archive for insights in neurophysiology (TDBRAIN) database. Sci. Data 9 (2022). https://doi.org/10.1038/s41597-022-01409-z
Doersch, C., Gupta, A., Efros, A.A.: Unsupervised visual representation learning by context prediction. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1422–1430 (2015)
Eldele, E., et al.: Time-series representation learning via temporal and contextual contrasting. arXiv preprint arXiv:2106.14112 (2021)
Escudero, J., Abásolo, D., Hornero, R., Espino, P., López, M.: Analysis of electroencephalograms in Alzheimer’s disease patients with multiscale entropy. Physiol. Meas. 27(11), 1091 (2006)
Fang, H., Wang, S., Zhou, M., Ding, J., Xie, P.: CERT: contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766 (2020)
Giorgi, J., Nitski, O., Wang, B., Bader, G.: DeCLUTR: deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659 (2020)
Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.: Internet of things (IoT): a vision, architectural elements, and future directions. Futur. Gener. Comput. Syst. 29(7), 1645–1660 (2013)
Gungor, V.C., Hancke, G.P.: Industrial wireless sensor networks: challenges, design principles, and technical approaches. IEEE Trans. Industr. Electron. 56(10), 4258–4265 (2009). https://doi.org/10.1109/TIE.2009.2015754
Gutmann, M., Hyvärinen, A.: Noise-contrastive estimation: a new estimation principle for unnormalized statistical models. In: Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 297–304. JMLR Workshop and Conference Proceedings (2010)
He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.: Momentum contrast for unsupervised visual representation learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738 (2020)
Kiyasseh, D., Zhu, T., Clifton, D.A.: CLOCS: contrastive learning of cardiac signals across space, time, and patients. In: International Conference on Machine Learning, pp. 5606–5615. PMLR (2021)
Larsson, G., Maire, M., Shakhnarovich, G.: Colorization as a proxy task for visual understanding. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6874–6883 (2017)
Van der Maaten, L., Hinton, G.: Visualizing data using t-SNE. J. Mach. Learn. Res. 9(11), 2579–2605 (2008)
Madakam, S., Lake, V., Lake, V., Lake, V., et al.: Internet of things (IoT): a literature review. J. Comput. Commun. 3(05), 164 (2015)
Noroozi, M., Favaro, P.: Unsupervised learning of visual representations by solving Jigsaw puzzles. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9910, pp. 69–84. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46466-4_5
Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)
Pöppelbaum, J., Chadha, G.S., Schwung, A.: Contrastive learning based self-supervised time-series analysis. Appl. Soft Comput. 117, 108397 (2022)
Qiu, J., Jammalamadaka, S.R., Ning, N.: Multivariate Bayesian structural time series model. J. Mach. Learn. Res. 19(68), 1–33 (2018)
Robinson, P.M.: Log-periodogram regression of time series with long range dependence. Ann. Statist. 23, 1048–1072 (1995)
Scott, S.L., Varian, H.R.: Bayesian Variable Selection for Nowcasting Economic Time Series. In: Economic Analysis of the Digital Economy, pp. 119–135. University of Chicago Press (2015)
Shaham, U., Svirsky, J., Katz, O., Talmon, R.: Discovery of single independent latent variable. Adv. Neural. Inf. Process. Syst. 35, 25251–25263 (2022)
Spencer, J., Bowden, R., Hadfield, S.: Medusa: universal feature learning via attentional multitasking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3800–3809 (2022)
Tang, S., et al.: Self-supervised graph neural networks for improved electroencephalographic seizure analysis. arXiv preprint arXiv:2104.08336 (2021)
Tian, Y., Krishnan, D., Isola, P.: Contrastive multiview coding. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020, Part XI. LNCS, vol. 12356, pp. 776–794. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58621-8_45
Tonekaboni, S., Eytan, D., Goldenberg, A.: Unsupervised representation learning for time series with temporal neighborhood coding. arXiv preprint arXiv:2106.00750 (2021)
Wang, F., et al.: Residual attention network for image classification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3156–3164 (2017)
Wang, Y., Han, Y., Wang, H., Zhang, X.: Contrast everything: a hierarchical contrastive framework for medical time-series. arXiv preprint arXiv:2310.14017 (2023)
Woo, G., Liu, C., Sahoo, D., Kumar, A., Hoi, S.: Cost: contrastive learning of disentangled seasonal-trend representations for time series forecasting. arXiv preprint arXiv:2202.01575 (2022)
Xiao, Z., Liang, S., Wang, J., Xiang, Y., Zhao, X., Song, J.: Long-time-series global land surface satellite leaf area index product derived from MODIS and AVHRR surface reflectance. IEEE Trans. Geosci. Remote Sens. 54(9), 5301–5318 (2016)
Xie, Z., Zhou, B., Cheng, X., Schoenfeld, E., Ye, F.: Passive and context-aware in-home vital signs monitoring using co-located UWB-depth sensor fusion. ACM Trans. Comput. Healthcare 3(4) (2022). https://doi.org/10.1145/3549941
Yue, Z., et al.: TS2Vec: towards universal representation of time series. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, pp. 8980–8987 (2022)
Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016, Part III. LNCS, vol. 9907, pp. 649–666. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46487-9_40
Zhang, X., Zhao, Z., Tsiligkaridis, T., Zitnik, M.: Self-supervised contrastive pre-training for time series via time-frequency consistency. Adv. Neural. Inf. Process. Syst. 35, 3988–4003 (2022)
Zia, H.B., Castro, I., Zubiaga, A., Tyson, G.: Improving zero-shot cross-lingual hate speech detection with pseudo-label fine-tuning of transformer language models. In: Proceedings of the International AAAI Conference on Web and Social Media, vol. 16, pp. 1435–1439 (2022)
Acknowledgments
This work was supported by the Project “Research and Demonstration of Anomalous Early Warning Methods for Structural Changes of Ancient Buildings Based on Normal Models (2022S172)” of Ningbo Municipal Public Welfare Science and Technology Project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
He, Y., Wu, Y., Zhang, J., Dong, Y. (2024). LTCR: Long Temporal Characteristic Reconstruction for Segmentation in Contrastive Learning. In: Bifet, A., Davis, J., Krilavičius, T., Kull, M., Ntoutsi, E., Žliobaitė, I. (eds) Machine Learning and Knowledge Discovery in Databases. Research Track. ECML PKDD 2024. Lecture Notes in Computer Science(), vol 14945. Springer, Cham. https://doi.org/10.1007/978-3-031-70362-1_21
Download citation
DOI: https://doi.org/10.1007/978-3-031-70362-1_21
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-70361-4
Online ISBN: 978-3-031-70362-1
eBook Packages: Computer ScienceComputer Science (R0)