Abstract
Inertial Measurement Unit (IMU)-based Human Activity Recognition (HAR) systems that employ Generalized Zero-Shot Learning (GZSL) face significant challenges in accurately classifying activities that were not observed previously during training. These challenges stem primarily from the inherent difficulty of recognizing unseen classes without sacrificing the classification accuracy of observed classes in a GZSL setting. A novel deep neural network (DNN) architecture termed as the Joint Sequences (JS)-Siamese architecture is proposed to address these challenges using IMU and video data. The proposed architecture uses skeleton joint sequences to bridge the gap between IMU features and video data, thus effectively solving the domain shift problem. A Siamese DNN-based metric learning model is employed to handle the hubness problem by mapping similar samples in close proximity and dissimilar ones farther apart in a joint embedding space. Additionally, a Dynamic Calibration Ensemble (DCE) technique is introduced to address the classification bias towards the observed classes in GZSL, thereby ensuring balanced representation of both, observed and unseen classes. The proposed JS-Siamese DNN architecture is shown to yield significant performance improvement over attribute-based, word embedding-based and video embedding-based GZSL approaches for HAR proposed in the literature. Experimental evaluation on three IMU benchmark datasets, i.e., PAMAP2, DaLiAc and UTD-MHAD demonstrate the effectiveness of the proposed JS-Siamese DNN architecture for sensor-based HAR.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Al-Saad, M., Ramaswamy, L., Bhandarkar, S.: F4D: Factorized 4D Convolutional Neural Network for Efficient Video-level Representation Learning. In: Proc. ICAART (2024)
Cao, Z., Simon, T., Wei, S.E., Sheikh, Y.: Realtime multi-person 2d pose estimation using part affinity fields. In: Proc. IEEE Conf. CVPR. pp. 7291–7299 (2017)
Chao, W.L., Changpinyo, S., Gong, B., Sha, F.: An empirical study and analysis of generalized zero-shot learning for object recognition in the wild. In: Proc. ECCV. pp. 52–68. Springer (2016)
Chen, C., Jafari, R., Kehtarnavaz, N.: UTD-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and a wearable inertial sensor. In: Proc. IEEE ICIP. pp. 168–172. IEEE (2015). https://doi.org/10.1109/ICIP.2015.7350781
Cheng, H.T., Griss, M., Davis, P., Li, J., You, D.: Towards zero-shot learning for human activity recognition using semantic attribute sequence model. In: Proc. ACM UBICOMP. pp. 355–358 (2013). https://doi.org/10.1145/2493432.2493511
Cheng, H.T., Sun, F.T., Griss, M., Davis, P., Li, J., You, D.: Nuactiv: Recognizing unseen new activities using semantic attribute-based learning. In: Proc. MobiSys. pp. 361–374 (2013). https://doi.org/10.1145/2462456.2464438
Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)
Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., Krishnan, D.: Supervised contrastive learning. Proc. NIPS 33, 18661–18673 (2020)
Lampert, C.H., Nickisch, H., Harmeling, S.: Learning to detect unseen object classes by between-class attribute transfer. In: Proc. IEEE Conf. CVPR. pp. 951–958. IEEE (2009)
Leutheuser, H., Schuldhaus, D., Eskofier, B.M.: Hierarchical, multi-sensor based classification of daily life activities: comparison with state-of-the-art algorithms using a benchmark dataset. PLoS ONE 8(10), e75196 (2013). https://doi.org/10.1371/journal.pone.0075196
Madgwick, S., et al.: An efficient orientation filter for inertial and inertial/magnetic sensor arrays. Report x-io and University of Bristol (UK) 25, 113–118 (2010)
Matsuki, M., Lago, P., Inoue, S.: Characterizing word embeddings for zero-shot sensor-based human activity recognition. Sensors 19(22), 5043 (2019)
Ohashi, H., Al-Naser, M., Ahmed, S., Nakamura, K., Sato, T., Dengel, A.: Attributes’ importance for zero-shot pose-classification based on wearable sensors. Sensors 18(8), 2485 (2018)
Olinski, M., Gronowicz, A., Ceccarelli, M., Cafolla, D.: Human motion characterization using wireless inertial sensors. In: Proc. MTM Robotics. pp. 401–408. Springer (2017). https://doi.org/10.1007/978-3-319-45450-4_40
Reiss, A., Stricker, D.: Creating and benchmarking a new dataset for physical activity monitoring. In: Proc. Intl. Conf. Pervasive Tech. Related to Asst. Environ. pp. 1–8. New York, NY, USA (2012). https://doi.org/10.1145/2413097.2413148
Reiss, A., Stricker, D.: Introducing a new benchmarked dataset for activity monitoring. In: Intl. Symp. Wearable Comp. pp. 108–109. IEEE (2012). https://doi.org/10.1109/ISWC.2012.13
Soomro, K., Zamir, A.R., Shah, M.: UCF101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 (2012)
Tong, C., Ge, J., Lane, N.D.: Zero-shot learning for IMU-based activity recognition using video embeddings. Proc. ACM IMWUT, volume=5, number=4, pages=1–23, year=2021, publisher=ACM New York, NY, USA, https://doi.org/10.1145/3494995
Wang, W., Li, Q.: Generalized Zero-Shot Activity Recognition with Embedding-Based Method. ACM Trans. Sensor Networks 19(3), 1–25 (2023). https://doi.org/10.1145/3582690
Wang, W., Miao, C., Hao, S.: Zero-shot human activity recognition via nonlinear compatibility based method. In: Proc. Intl. Conf. Web Intell. pp. 322–330. New York, NY, USA (2017). https://doi.org/10.1145/3106426.3106526
Wu, T., Chen, Y., Gu, Y., Wang, J., Zhang, S., Zhechen, Z.: Multi-layer cross loss model for zero-shot human activity recognition. In: Proc. PAKDD Part I 24. pp. 210–221. Springer International Publishing, Cham (2020)
Yan, S., Xiong, Y., Lin, D.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proc. AAAI Conf. AI. vol. 32 (2018)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Al-Saad, M., Ramaswamy, L., Bhandarkar, S.M. (2025). JS-Siamese: Generalized Zero Shot Learning for IMU-based Human Activity Recognition. In: Antonacopoulos, A., Chaudhuri, S., Chellappa, R., Liu, CL., Bhattacharya, S., Pal, U. (eds) Pattern Recognition. ICPR 2024. Lecture Notes in Computer Science, vol 15315. Springer, Cham. https://doi.org/10.1007/978-3-031-78354-8_26
Download citation
DOI: https://doi.org/10.1007/978-3-031-78354-8_26
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-78353-1
Online ISBN: 978-3-031-78354-8
eBook Packages: Computer ScienceComputer Science (R0)