Can Same-right-and-different-left Gestures Be Recognized with Only Right-hand Signals?
Authors: 
Yidan Cao, Qingshan Wang, Qi Wang, Peng LiuAuthors Info & Claims
Article No.: 226, Pages 1 - 19
Abstract
Sign language serves as a bridge between the hearing-impaired and other people. Existing sensor-based approaches tend to only collect data from the dominant hand. Does this signal collection method affect the accuracy of gesture recognition, especially gestures where the dominant hand has the same movement while the non-dominant hand has different movements? The specific gestures are called same-right-and-different-left (SRDL) where the right hand is dominant. This article is the first to propose an SRDL-aware sign language recognition system. First, an SRDL discriminator based on an autoencoder and range classifier is designed to determine whether the gesture is SRDL. Second, an SRDL feature selector based on clustering relationship is presented. Multivariate variational mode decomposition and fast fourier transform are used to obtain the feature expression. Moreover, a clustering relationship algorithm is proposed to dynamically select features for every group of SRDL gestures in the feature expression. Finally, the experimental results show that the average word error rate is 14.3% and decreases by 8.5% and 12.1% compared with Signspeaker and MyoSign, respectively.
References
[1]
World Health Organization. 2021. World Report on Hearing. https://www.who.int/publications/i/item/9789240020481
[2]
Jiahui Hou, Xiang Yang Li, Peide Zhu, Zefan Wang, Yu Wang, Jianwei Qian, and Panlong Yang. 2019. SignSpeaker: A real-time, high-precision SmartWatch-based sign language translator. In Proceedings of the International Conference on Mobile Computing and Networking. 1–15.
[3]
Zhiwen Zheng, Qingshan Wang, Dejun Yang, Qi Wang, Wei Huang, and Yinlong Xu. 2022. L-Sign: Large-vocabulary sign gestures recognition system. IEEE Trans. Hum.-Mach. Syst. 52, 2 (2022), 290–301.
[4]
Qian Zhang, Dong Wang, Run Zhao, and Yinggang Yu. 2019. MyoSign: Enabling end-to-end sign language recognition with wearables. In Proceedings of the International Conference on Intelligent User Interfaces. 650–660.
[5]
Elyor Kodirov, Tao Xiang, and Shaogang Gong. 2017. Semantic autoencoder for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 4447–4456.
[6]
Hongwei Zhang, Wenbo Tang, Lingjie Zhang, Pengfei Li, and De Gu. 2019. Defect detection of yarn-dyed shirts based on denoising convolutional self-encoder. In Proceedings of the IEEE Data Driven Control and Learning Systems Conference (DDCLS’19). 1263–1268.
[7]
Wenrui Zhang. 2019. An image recognition algorithm based on self-encoding and convolutional neural network fusion. In Proceedings of the International Conference on Electronic Engineering and Informatics (EEI’19). 402–407.
[8]
Naveed ur Rehman and Hania Aftab. 2019. Multivariate variational mode decomposition. IEEE Trans. Signal Process. 67, 23 (2019), 6039–6052.
[9]
Kun Yang, Manjin Xu, Xiaotong Yang, and Yueming Chen. 2021. An EMG gesture recognition method based on multivariate variational mode decomposition. In Proceedings of the International Conference on Advanced Electronic Materials, Computers and Software Engineering (AEMCSE’21). 518–521.
[10]
Yingwei Zhang, Yiqiang Chen, Hanchao Yu, Xiaodong Yang, and Wang Lu. 2020. Learning effective spatial–temporal features for sEMG armband-based gesture recognition. IEEE Internet Things J. 7, 8 (2020), 6979–6992.
[11]
Misha Urooj Khan, Hareem Khan, Muhammad Muneeb, Zeeshan Abbasi, Usman Babar Abbasi, and Naveed Khan Baloch. 2021. Supervised machine learning based fast hand gesture recognition and classification using electromyography signals. In Proceedings of the International Conference on Applied and Engineering Mathematics (ICAEM’21). 81–86.
[12]
Chenlei Xie, Daqing Wang, Dun Hu, and Lifu Gao. 2021. Mechanomyography signals processing method using multivariate variational mode decomposition. In Proceedings of the International Symposium on Computational Intelligence and Design (ISCID’21). 278–281.
[13]
Fengji Ma, Junyi Chai, and Hai Wang. 2019. Two-dimensional compact variational mode decomposition-based low-light image enhancement. IEEE Access 7 (2019), 136299–136309.
[14]
Yanfei Guo and Zhousuo Zhang. 2021. Generalized variational mode decomposition: A multiscale and fixed-frequency decomposition algorithm. IEEE Trans. Instrument. Measure. 70 (2021), 1–13.
[15]
Ali Komaty, Abdel Ouahab Boudraa, Patrick Flandrin, Pierre Olivier Amblard, and Jacques-André Astolfi. 2021. On the behavior of MEMD in presence of multivariate fractional gaussian noise. IEEE Trans. Signal Process. 69 (2021), 2676–2688.
[16]
Ileana Diana Nicolae, Radu Florin Marinescu, Petre-Marian Nicolae, and Maria Diana Cristina. 2017. Limits and usability of fast Fourier, Discrete wavelet and wavelet packet transforms applied at signals from a primary winding of a locomotive transformer. In Proceedings of the International Conference on Electromechanical and Power Systems (SIELMEN’17). 462–467.
[17]
Denis Donnelly. 2006. The fast fourier and Hilbert-Huang transforms: A comparison. In Proceedings of the Multiconference on Computational Engineering in Systems Applications. 84–88.
[18]
Jun Wan, Stan Z. Li, Yibing Zhao, Shuai Zhou, Isabelle Guyon, and Sergio Escalera. 2016. ChaLearn looking at people RGB-D isolated and continuous datasets for gesture recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW’16). 761–769.
[19]
Zhou Ren, Junsong Yuan, Jingjing Meng, and Zhengyou Zhang. 2013. Robust part-based hand gesture recognition using kinect sensor. IEEE Trans. Multimedia 15, 5 (2013), 1110–1120.
[20]
Wan Yi Yeh, Teng Hui Tseng, Jun Wei Hsieh, and Chun Ming Tsai. 2016. Sign language recognition system via Kinect: Number and English alphabet. In Proceedings of the International Conference on Machine Learning and Cybernetics (ICMLC’16). 660–665.
[21]
Noriaki Fujishima and Tatsuhiro Ietsuka. 2016. Basic construction of a natural finger outline extraction system with a color glove. In Proceedings of the IEEE/ACIS International Conference on Computer and Information Science (ICIS’16). 1–6.
[22]
Hun Young Jung, Jong Hyeok Lee, Eunju Min, and Seung Hoon Na. 2019. Word reordering for translation into Korean Sign Language using syntactically guided classification. ACM Trans. Asian Low-Resour. Lang. Info. Process. 19, 2, Article 31 (2019), 20 pages. DOI:
[23]
Krishan Kumar. 2022. DEAF-BSL: Deep LEArning framework for British sign language recognition. ACM Trans. Asian Low-Resour. Lang. Info. Process. 21, 5, Article 101 (2022), 14 pages. DOI:
[24]
E. Rajalakshmi, R. Elakkiya, Alexey L. Prikhodko, M. G. Grif, Maxim A. Bakaev, Jatinderkumar R. Saini, Ketan Kotecha, and V. Subramaniyaswamy. 2022. Static and dynamic isolated Indian and Russian sign language recognition with spatial and temporal feature detection using hybrid neural network. ACM Trans. Asian Low-Resour. Lang. Info. Process. 22, 1 (2022), 1–23. DOI:
[25]
Oleg Makaussov, Mikhail Krassavin, Maxim Zhabinets, and Siamac Fazli. 2020. A low-cost, IMU-based real-time on device gesture recognition glove. In Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC’20). 3346–3351.
[26]
Derek W. Orbaugh Antillon, Christopher R. Walker, Samuel Rosset, and Iain A. Anderson. 2022. Glove-based hand gesture recognition for diver communication. IEEE Trans. Neural Netw. Learn. Syst. (2022), 1–13. DOI:
[27]
Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the CHI Conference on Human Factors in Computing Systems. 3847–3851.
[28]
Feng Duan, Xina Ren, and Yikang Yang. 2021. A gesture recognition system based on time domain features and linear discriminant analysis. IEEE Trans. Cogn. Dev. Syst. 13, 1 (2021), 200–208.
[29]
Biao Sun, Beida Song, Jiajun Lv, Peiyin Chen, Xinlin Sun, Chao Ma, and Zhongke Gao. 2022. A multi-scale feature extraction network based on channel-spatial attention for electromyographic signal classification. IEEE Trans. Cogn. Dev. Syst. 15, 2 (2022), 591–601. DOI:
[30]
Zhiwen Zheng, Qingshan Wang, Dazhu Deng, Qi Wang, and Wei Huang. 2022. CG-Recognizer: A biosignal-based continuous gesture recognition system. Biomed. Signal Process. Control 78 (2022), 103995–103995. DOI:
[31]
Zhibo Wang, Tengda Zhao, Jinxin Ma, Hongkai Chen, Kaixin Liu, Huajie Shao, Qian Wang, and Ju Ren. 2022. Hear sign language: A real-time end-to-end sign language recognition system. IEEE Trans. Mobile Comput. 21, 7 (2022), 2398–2410.
[32]
Rami N. Khushaba, Angkoon Phinyomark, Ali H. Al-Timemy, and Erik Scheme. 2020. Recursive multi-signal temporal fusions with attention mechanism improves EMG feature extraction. IEEE Trans. Artific. Intell. 1, 2 (2020), 139–150.
[33]
Ulysse Côté-Allard, Gabriel Gagnon Turcotte, Angkoon Phinyomark, Kyrre Glette, Erik Scheme, François Laviolette, and Benoit Gosselin. 2021. A transferable adaptive domain adversarial neural network for virtual reality augmented EMG-based gesture recognition. IEEE Trans. Neural Syst. Rehabil. Eng. 29 (2021), 546–555.
[34]
Chengcheng Hang, Rui Zhang, Zhiping Chen, Chenqi Li, and Zhijun Li. 2017. Dynamic gesture recognition method based on improved DTW algorithm. In Proceedings of the International Conference on Industrial Informatics—Computing Technology, Intelligent Technology, Industrial Information Integration (ICIICII’17). 71–74.
[35]
Jeroen F. Lichtenauer, Emile A. Hendriks, and Marcel J. T. Reinders. 2008. Sign language recognition by combining statistical DTW and independent classification. IEEE Trans. Pattern Anal. Mach. Intell. 30, 11 (2008), 2040–2046.
[36]
Xinshuo Li, Jianping Li, and Yuanyan Tang. 2014. Another general analytic construction for wavelet lowpassed filters. In Proceedings of the International Computer Conference on Wavelet Actiev Media Technology and Information Processing (ICCWAMTIP’14). 487–490.
[37]
Youssef Aiboud, Jamal El Mhamdi, Abdelilah Jilbab, and Hamza Sbaa. 2015. Review of ECG signal de-noising techniques. In Proceedings of the World Conference on Complex Systems (WCCS’15). 1–6.
[38]
Minhyuk Lee and Joonbum Bae. 2022. Real-time gesture recognition in the view of repeating characteristics of sign languages. IEEE Trans. Industr. Info. 18, 12 (2022), 8818–8828. DOI:
[39]
Yongqin Xian, Christoph H. Lampert, Bernt Schiele, and Zeynep Akata. 2019. Zero-shot learning—A comprehensive evaluation of the good, the bad and the ugly. IEEE Trans. Pattern Anal. Mach. Intell. 41, 9 (2019), 2251–2265.
[40]
Christoph H. Lampert, Hannes Nickisch, and Stefan Harmeling. 2009. Learning to detect unseen object classes by between-class attribute transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 951–958.
[41]
Shafin Rahman, Salman Khan, and Fatih Porikli. 2018. A unified approach for conventional zero-shot, generalized zero-shot, and few-shot learning. IEEE Trans. Image Process. 27, 11 (2018), 5652–5667.
[42]
Richard H. Bartels and G. W. Stewart. 1972. Solution of the matrix equation AX + XB = C [F4]. Commun. ACM 15, 9 (1972), 820–826. DOI:
[43]
Joveria Javed, Hashim Yasin, and Syed Faisal Ali. 2010. Human movement recognition using euclidean distance: A tricky approach. In Proceedings of the International Congress on Image and Signal Processing, Vol. 1. 317–321.
[44]
Minhyuk Lee and Joonbum Bae. 2020. Deep learning based real-time recognition of dynamic finger gestures using a data glove. IEEE Access 8 (2020), 219923–219933.
[45]
China Disabled Persons’ Federation and China Deaf Association. 2019. Chinese National Dictionary of Sign Language. Huaxia Publishing House.
[46]
Jiangtao Zhang, Qingshan Wang, Qi Wang, and Zhiwen Zheng. 2023. Multimodal fusion framework based on statistical attention and contrastive attention for sign language recognition. IEEE Trans. Mobile Comput. (2023), 1–13. DOI:
[47]
Fazla Rabbi Mashrur, Amit Dutta Roy, and Dabasish Kumar Saha. 2019. Automatic identification of arrhythmia from ECG using AlexNet convolutional neural network. In Proceedings of the International Conference on Electrical Information and Communication Technology (EICT’19). 1–5.
[48]
Runpeng Cui, Hu Liu, and Changshui Zhang. 2017. Recurrent convolutional neural networks for continuous sign language recognition by staged optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’17). 1610–1618.
Index Terms
- Can Same-right-and-different-left Gestures Be Recognized with Only Right-hand Signals?
Recommendations
Left and right hand distinction for multi-touch tabletop interactions
IUI '14: Proceedings of the 19th international conference on Intelligent User InterfacesIn multi-touch interactive systems, it is of great significance to distinguish which hand of the user is touching the surface in real time. Left-right hand distinction is essential for recognizing the multi-finger gestures and further fully exploring ...
Your left hand can do it too!: investigating intermanual, symmetric gesture transfer on touchscreens
CHI '13: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsThis work examines intermanual gesture transfer, i.e., learning a gesture with one hand and performing it with the other. Using a traditional retention and transfer paradigm from the motor learning literature, participants learned four gestures on a ...
Left/right hand segmentation in egocentric videos
AbstractWearable cameras allow people to record their daily activities from a user-centered (First Person Vision) perspective. Due to their favorable location, wearable cameras frequently capture the hands of the user, and may thus represent a ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
September 2023
226 pages
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 22 September 2023
Online AM: 25 August 2023
Accepted: 11 August 2023
Revised: 20 May 2023
Received: 13 September 2022
Published in TALLIP Volume 22, Issue 9
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- Anhui Provincial Natural Science Foundation of China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 190Total Downloads
- Downloads (Last 12 months)82
- Downloads (Last 6 weeks)7
Reflects downloads up to 12 Jan 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in