Abstract
The action recognition system based on deep learning has achieved great success in recent years. By this, the dynamic hand gesture recognition development provides the possibility for new human-computer interaction methods. However, most existing hand gesture recognition feature extraction models are difficult to balance the accuracy and speed of recognition. To improve the performance of the hand gesture recognition model, this paper analyzes the hand skeleton sequence, proposes Multi-skeletal Features (MSF), and constructs a dynamic hand gesture feature extraction model based on the MSF. The MSF includes the distance of the hand skeleton points and the angle of the skeleton connections, which can effectively reduce the angle of view and distance noise in the expression of skeleton features. The model has superior performance on the SHREC dataset and can achieve an accuracy of 96.43% in 14 gesture classifications. At the same time, we combine our method with a hand pose estimator to design a real-time sign language recognition (SLR) system.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Tran, D., Bourdev, L., Fergus, R., Torresani, L., Paluri, M.: Learning spatiotemporal features with 3D convolutional networks. In: IEEE International Conference on Computer Vision, pp. 4489–4497 (2015)
Hu, Z., Hu, Y., Liu, J., et al.: 3D separable convolutional neural network for dynamic hand gesture recognition. Neurocomputing 318, 151–161 (2018)
Xingjian, S.H I., Chen, Z., Wang, H., et al.: Convolutional LSTM network: a machine learning approach for precipitation nowcasting. In: Advances in neural information processing systems, pp. 802–810 (2015)
Zhang, X., Tie, Y., Qi, L.: Dynamic gesture recognition based on 3D separable convolutional LSTM networks. In: 2020 IEEE 11th International Conference on Software Engineering and Service Science (ICSESS), Beijing, China, pp. 180–183 (2020). https://doi.org/10.1109/ICSESS49938.2020.9237672
Wan, J., et al.: ChaLearn looking at people: IsoGD and ConGD large-scale RGB-D ges-ture recognition. IEEE Trans. Cybern. 52(5), 3422–3433 (2022). https://doi.org/10.1109/TCYB.2020.3012092
Yan, S., Xiong, Y.J., Lin, D.H.: Spatial temporal graph convolutional networks for skeleton-based action recognition. In: Proceeding of the National Conference on Artificial Intelligence (2018)
Jingyao, W., Naigong, Y., Firdaous, E.: Gesture recognition matching based on dynamic skeleton. In: 2021 33rd Chinese Control and Decision Conference (CCDC), Kunming, China, 2021, pp. 1680–1685 (2021). https://doi.org/10.1109/CCDC52312.2021.9601572
Song, Y.-F., Zhang, Z., Shan, C., Wang, L.: Stronger faster and more explainable: a graph convolutional baseline for skeleton-based action recognition. In: Proceedings of ACM International Conference on Multimedia, pp. 1625–1633 (2020)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)
Yang, F., et al.: Make skeleton-based action recognition model smaller, faster and better. arXiv:1907.09658 (2020)
Ioffe, S., Szegedy, C.: Batch normalization: accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015)
Zhang, F., Bazarevsky, V., Vakunov, A., et al.: Mediapipe hands: on-device real-time hand tracking arXiv:2006.10214 (2020)
Smedt, Q-D., Wannous, H., Vandeborre, J-P., Guerry, J., Saux, B-L., Filliat, D.: Shrec 2017 track: 3d hand gesture recognition using a depth and skeletal dataset. In: 10th Eurographics Workshop on 3D Object Retrieval (2017)
Shotton, J., et al.: Real-time human pose recognition in parts from single depth images. Commu. ACM 56(1), 116–124 (2011)
Ohn-Bar, E., Trivedi, M.M., Katz, I.: Joint angles similarities and HMM for skeleton-based gesture recognition. IEEE Trans. Multimedia 16(1), 37–48 (2014)
Lai, K., Yanushkevich, S.N.: CNN+RNN Depth and skeleton based dynamic hand gesture recognition. In: 2018 24th International Conference on Pattern Recognition (ICPR), Beijing, China, pp. 3451–3456 (2018). https://doi.org/10.1109/ICPR.2018.8545718
Nguyen, X.S., Brun, L., Lézoray, O., Bougleux, S.: Learning recurrent high-order statistics for skeleton-based hand gesture recognition. In: 2020 25th International Conference on Pattern Recognition (ICPR), Milan, Italy, pp. 975–982 (2021). https://doi.org/10.1109/ICPR48806.2021.9412036
Hou, J., Wang, G., Chen, X., Xue, J.-H., Zhu, R., Yang, H.: Spatial-temporal attention res-tcn for skeleton-based dynamic hand gesture recognition. In: Leal-Taixé, L., Roth, S. (eds.) ECCV 2018. LNCS, pp. 273–286. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-11024-6_18
Chakraborty, R., et al.: A statistical recurrent model on the manifold of symmetric positive definite matrices. In: Advances in Neural Information Processing Systems, NeurIPS, vol. 31, pp. 8897–8908 (2018)
Deng, Z., Gao, Q., Ju, Z., Yu, X.: Skeleton-based multifeatures and multistream network for real-time action recognition. IEEE S. J. 23(7), 7397–7409 (2023). https://doi.org/10.1109/JSEN.2023.3246133
Gao, Q., Liu, J., Ju, Z., Zhang, X.: Dual-Hand detection for human-robot interaction by a parallel network based on hand detection and body pose estimation. IEEE Trans. Industr. Electron. 66(12), 9663–9672 (2019). https://doi.org/10.1109/TIE.2019.2898624
Gao, Q., Chen, Y., Ju, Z., Yi, L.: Dynamic hand gesture recognition based on 3D hand pose estimation for human–robot interaction. IEEE Sens. J. 22(18), 17421–17430 (2022). https://doi.org/10.1109/JSEN.2021.3059685
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (62006204), the Guangdong Basic and Applied Basic Research Foundation (2022A1515011431), and Shenzhen Science and Technology Program (RCBS20210609104516043, JSGG20210802154004014).
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.
About this paper
Cite this paper
Wu, B., Deng, Z., Gao, Q. (2023). Dynamic Hand Gesture Recognition Based on Multi-skeletal Features for Sign Language Recognition System. In: Yang, H., et al. Intelligent Robotics and Applications. ICIRA 2023. Lecture Notes in Computer Science(), vol 14268. Springer, Singapore. https://doi.org/10.1007/978-981-99-6486-4_7
Download citation
DOI: https://doi.org/10.1007/978-981-99-6486-4_7
Published:
Publisher Name: Springer, Singapore
Print ISBN: 978-981-99-6485-7
Online ISBN: 978-981-99-6486-4
eBook Packages: Computer ScienceComputer Science (R0)