Abstract
Time-Series data collected using body-worn sensors can be used to recognize activities of interest in various medical applications such as sleep studies. Recent advances in other domains, such as image recognition and natural language processing have shown that unlabeled data can still be useful when self-supervised techniques such as contrastive learning are used to generate meaningful feature space representations. Labeling data for Human Activity Recognition (HAR) and sleep disorder diagnosis (polysomnography) is difficult and requires trained professionals. In this work, we apply learned feature representation techniques to multimodal time-series data. By using signal-specific representations, based on self-supervised and supervised learning, the channels can be evaluated to determine if they are likely to contribute to correct classification. The learned representation embeddings are then used to process each channel into a new feature space that serves as input into a neural network. This results in a better understanding of the importance of each signal modality as well as the potential applicability of newer self-supervised techniques to time-series data.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Allocca, G., et al.: Validation of ‘Somnivore’, a machine learning algorithm for automated scoring and analysis of polysomnography data. Front. Neurosci. 13, 207 (2019)
Chen, T., Kornblith, S., Norouzi, M., Hinton, G.: A simple framework for contrastive learning of visual representations. In: International Conference on Machine Learning, pp. 1597–1607. PMLR (2020)
Choi, S.H., et al.: Real-time apnea-hypopnea event detection during sleep by convolutional neural networks. Comput. Biol. Med. 100, 123–131 (2018)
Ding, L., Peng, J., Song, L., Zhang, X.: Automatically detecting Apnea-Hypopnea snoring signal based on VGG19+ LSTM. Biomed. Sig. Process. Control 80, 104351 (2023)
Dwibedi, D., Aytar, Y., Tompson, J., Sermanet, P., Zisserman, A.: With a little help from my friends: nearest-neighbor contrastive learning of visual representations. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9588–9597 (2021)
Empatica: E4 wristband user’s manual, rev. 2.0. https://empatica.app.box.com/v/E4-User-Manual (2020). Accessed 09 Jun 2022
Hinkle, L.B., Atkinson, G., Metsis, V.: TWristAR - wristband activity recognition, January 2022, Online. https://doi.org/10.5281/zenodo.5911808
Hinkle, L.B., Metsis, V.: Model evaluation approaches for human activity recognition from time-series data. In: Tucker, A., Henriques Abreu, P., Cardoso, J., Pereira Rodrigues, P., Riaño, D. (eds.) AIME 2021. LNCS (LNAI), vol. 12721, pp. 209–215. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-77211-6_23
Hinkle, L.B., Metsis, V.: Unlabeled Empatica E4 Wristband Data (UE4W) Dataset, July 2022, Online. https://doi.org/10.5281/zenodo.6898244
Kokkalas, L., Korompili, G., Tatlas, N.A., Mitilineos, S.A., Potirakis, S.M.: Severe obstructive sleep apnea event detection from EEG recordings. In: Presented at 2nd International Electronic Conference on Applied Sciences, vol. 15, p. 31 (2021)
Korompili, G., et al.: PSG-audio, a scored polysomnography dataset with simultaneous audio recordings for sleep apnea studies. Sci. Data 8(1), 1–13 (2021)
Korompili, G., et al.: PSG-Audio, March 2022. https://doi.org/10.11922/sciencedb.00345
Lahat, D., Adali, T., Jutten, C.: Multimodal data fusion: an overview of methods, challenges, and prospects. Proc. IEEE 103(9), 1449–1477 (2015)
Levy, P., Pépin, J.L., Deschaux-Blanc, C., Paramelle, B., Brambilla, C.: Accuracy of oximetry for detection of respiratory disturbances in sleep apnea syndrome. Chest 109(2), 395–399 (1996)
McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)
Mohsenvand, M.N., Izadi, M.R., Maes, P.: Contrastive representation learning for electroencephalogram classification. In: Machine Learning for Health, pp. 238–253. PMLR (2020)
Mondal, A., Jain, V., Siddiqi, K.: Mini-batch similarity graphs for robust image classification (2012)
Pedregosa, F., et al.: Scikit-learn: machine learning in Python. J. Mach. Learn. Res. 12, 2825–2830 (2011)
Piorecky, M., et al.: Apnea detection in polysomnographic recordings using machine learning techniques. Diagnostics 11(12), 2302 (2021)
Qian, H., Tian, T., Miao, C.: What makes good contrastive learning on small-scale wearable-based tasks? arXiv preprint arXiv:2202.05998 (2022)
Sleeman IV, W.C., Kapoor, R., Ghosh, P.: Multimodal classification: current landscape, taxonomy and future directions. arXiv preprint arXiv:2109.09020 (2021)
Xu, H., Zhou, P., Tan, R., Li, M., Shen, G.: LIMU-BERT: unleashing the potential of unlabeled data for IMU sensing applications. In: Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pp. 220–233 (2021)
Zhao, A., Dong, J., Zhou, H.: Self-supervised learning from multi-sensor data for sleep recognition. IEEE Access 8, 93907–93921 (2020)
Zinchuk, A.V., et al.: Polysomnographic phenotypes and their cardiovascular implications in obstructive sleep apnoea. Thorax 73(5), 472–480 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 IFIP International Federation for Information Processing
About this paper
Cite this paper
Hinkle, L.B., Atkinson, G., Metsis, V. (2023). Fusion of Learned Representations for Multimodal Sensor Data Classification. In: Maglogiannis, I., Iliadis, L., MacIntyre, J., Dominguez, M. (eds) Artificial Intelligence Applications and Innovations. AIAI 2023. IFIP Advances in Information and Communication Technology, vol 675. Springer, Cham. https://doi.org/10.1007/978-3-031-34111-3_34
Download citation
DOI: https://doi.org/10.1007/978-3-031-34111-3_34
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-34110-6
Online ISBN: 978-3-031-34111-3
eBook Packages: Computer ScienceComputer Science (R0)