Cited By
View all- Shin JMiah AKabir MRahim MAl Shiam A(2024)A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data ModalitiesIEEE Access10.1109/ACCESS.2024.345643612(142606-142639)Online publication date: 2024
Hand Gesture Recognition for Blind Users by Tracking 3D Gesture Trajectory
Authors: 
Prerna Khanna, 
IV Ramakrishnan, Shubham Jain, 
Xiaojun Bi, 
Aruna BalasubramanianAuthors Info & Claims
Article No.: 405, Pages 1 - 15
Abstract
Hand gestures provide an alternate interaction modality for blind users and can be supported using commodity smartwatches without requiring specialized sensors. The enabling technology is an accurate gesture recognition algorithm, but almost all algorithms are designed for sighted users. Our study shows that blind user gestures are considerably different from sighted users, rendering current recognition algorithms unsuitable. Blind user gestures have high inter-user variance, making learning gesture patterns difficult without large-scale training data. Instead, we design a gesture recognition algorithm that works on a 3D representation of the gesture trajectory, capturing motion in free space. Our insight is to extract a micro-movement in the gesture that is user-invariant and use this micro-movement for gesture classification. To this end, we develop an ensemble classifier that combines image classification with geometric properties of the gesture. Our evaluation demonstrates a 92% classification accuracy, surpassing the next best state-of-the-art which has an accuracy of 82%.
Supplemental Material
- Download
- 39.92 MB
- Transcript
References
[1]
Diar Abdlkarim, Massimiliano Di Luca, Poppy Aves, Sang-Hoon Yeo, R Chris Miall, Peter Holland, and Joseph M Galea. 2022. A methodological framework to assess the accuracy of virtual reality hand-tracking systems: A case study with the oculus quest 2. BioRxiv (2022), 2022–02.
[2]
Ahmad Akl, Chen Feng, and Shahrokh Valaee. 2011. A novel accelerometer-based gesture recognition system. IEEE transactions on Signal Processing 59, 12 (2011), 6197–6205.
[3]
Sara Ashry, Reda Elbasiony, and Walid Gomaa. 2018. An LSTM-based descriptor for human activities recognition using IMU sensors. In Proceedings of the 15th International Conference on Informatics in Control, Automation and Robotics, ICINCO, Vol. 1. 494–501.
[4]
Shiri Azenkot and Nicole B Lee. 2013. Exploring the use of speech input by blind people on mobile devices. In Proceedings of the 15th international ACM SIGACCESS conference on computers and accessibility. 1–8.
[5]
Gibran Benitez-Garcia, Muhammad Haris, Yoshiyuki Tsuda, and Norimichi Ukita. 2020. Continuous finger gesture spotting and recognition based on similarities between start and end frames. IEEE Transactions on Intelligent Transportation Systems (2020).
[6]
K Abhijith Bhaskaran, Anoop G. Nair, K Deepak Ram, Krishnan Ananthanarayanan, and H.R. Nandi Vardhan. 2016. Smart gloves for hand gesture recognition: Sign language to speech conversion system. In 2016 International Conference on Robotics and Automation for Humanitarian Applications (RAHA). 1–6. https://doi.org/10.1109/RAHA.2016.7931887
[7]
Ting Kwok Chan, Ying Kin Yu, Ho Chuen Kam, and Kin Hong Wong. 2018. Robust hand gesture input using computer vision, inertial measurement unit (IMU) and flex sensors. In 2018 IEEE International Conference on Mechatronics, Robotics and Automation (ICMRA). IEEE, 95–99.
[8]
Feiyu Chen, Jia Deng, Zhibo Pang, Majid Baghaei Nejad, Huayong Yang, and Geng Yang. 2018. Finger angle-based hand gesture recognition for smart infrastructure using wearable wrist-worn camera. Applied Sciences 8, 3 (2018), 369.
[9]
Feiyu Chen, Honghao Lv, Zhibo Pang, Junhui Zhang, Yonghong Hou, Ying Gu, Huayong Yang, and Geng Yang. 2018. WristCam: A wearable sensor for hand trajectory gesture recognition and intelligent human–robot interaction. IEEE Sensors Journal 19, 19 (2018), 8441–8451.
[10]
Heng-Tze Cheng, An Mei Chen, Ashu Razdan, and Elliot Buller. 2011. Contactless gesture recognition system using proximity sensors. In 2011 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 149–150.
[11]
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition. 248–255. https://doi.org/10.1109/CVPR.2009.5206848
[12]
James Diebel 2006. Representing attitude: Euler angles, unit quaternions, and rotation vectors. Matrix 58, 15-16 (2006), 1–35.
[13]
Nem Khan Dim, Chaklam Silpasuwanchai, Sayan Sarcar, and Xiangshi Ren. 2016. Designing mid-air TV gestures for blind people using user-and choice-based elicitation approaches. In Proceedings of the 2016 ACM conference on designing interactive systems. 204–214.
[14]
Matthew Ernst, Travis Swan, Victor Cheung, and Audrey Girouard. 2017. Typhlex: Exploring deformable input for blind users controlling a mobile screen reader. IEEE Pervasive Computing 16, 4 (2017), 28–35.
[15]
Shirin Feiz and IV Ramakrishnan. 2019. Exploring feasibility of wrist gestures for non-visual interactions with wearables. In Proceedings of the 16th International Web for All Conference. 1–4.
[16]
Jun Gong, Xing-Dong Yang, and Pourang Irani. 2016. Wristwhirl: One-handed continuous smartwatch input using wrist gestures. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 861–872.
[17]
Pengfei Gui, Liqiong Tang, and Subhas Mukhopadhyay. 2015. MEMS based IMU for tilting measurement: Comparison of complementary and kalman filter based data fusion. In 2015 IEEE 10th conference on Industrial Electronics and Applications (ICIEA). IEEE, 2004–2009.
[18]
Steven Houben and Nicolai Marquardt. 2015. Watchconnect: A toolkit for prototyping smartwatch-centric cross-device applications. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 1247–1256.
[19]
Michael Xuelin Huang, Yang Li, Nazneen Nazneen, Alexander Chao, and Shumin Zhai. 2021. TapNet: The Design, Training, Implementation, and Applications of a Multi-Task Learning CNN for Off-Screen Mobile Input. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–11.
[20]
Yangjian Huang, Weichao Guo, Jianwei Liu, Jiayuan He, Haisheng Xia, Xinjun Sheng, Haitao Wang, Xuetao Feng, and Peter B Shull. 2015. Preliminary testing of a hand gesture recognition wristband based on emg and inertial sensor fusion. In Intelligent Robotics and Applications: 8th International Conference, ICIRA 2015, Portsmouth, UK, August 24-27, 2015, Proceedings, Part I 8. Springer, 359–367.
[21]
Dai Jiang, Yu Wu, and Andreas Demosthenous. 2020. Hand gesture recognition using three-dimensional electrical impedance tomography. IEEE Transactions on Circuits and Systems II: Express Briefs 67, 9 (2020), 1554–1558.
[22]
Weibin Jiang, Xuelin Ye, Ruiqi Chen, Feng Su, Mengru Lin, Yuhanxiao Ma, Yanxiang Zhu, and Shizhen Huang. 2021. Wearable on-device deep learning system for hand gesture recognition based on FPGA accelerator. Mathematical Biosciences and Engineering 18, 1 (2021), 132–153.
[23]
Pyeong-Gook Jung, Gukchan Lim, Seonghyok Kim, and Kyoungchul Kong. 2015. A wearable gesture recognition device for detecting muscular activities based on air-pressure sensors. IEEE Transactions on Industrial Informatics 11, 2 (2015), 485–494.
[24]
Shaun K Kane, Jacob O Wobbrock, and Richard E Ladner. 2011. Usable gestures for blind people: understanding preference and performance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 413–422.
[25]
Bryce Kellogg, Vamsi Talla, and Shyamnath Gollakota. 2014. Bringing gesture recognition to all devices. In 11th USENIX Symposium on Networked Systems Design and Implementation (NSDI 14). 303–316.
[26]
Brian Kemler. 2021. Our all-new talkback screen reader. https://blog.google/products/android/all-new-talkback
[27]
Prerna Khanna, Shirin Feiz, Jian Xu, IV Ramakrishnan, Shubham Jain, Xiaojun Bi, and Aruna Balasubramanian. 2023. AccessWear: Making Smartphone Applications Accessible to Blind Users. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking. 1–16.
[28]
Rebecca Killick, Paul Fearnhead, and Idris A Eckley. 2012. Optimal detection of changepoints with a linear computational cost. J. Amer. Statist. Assoc. 107, 500 (2012), 1590–1598.
[29]
Sven Kratz and Michael Rohs. 2010. The $3 recognizer: simple 3D gesture recognition on mobile devices. In Proceedings of the 15th international conference on Intelligent user interfaces. 419–420.
[30]
Daniel Lauss, Florian Eibensteiner, and Phillip Petz. 2022. A Deep Learning based Hand Gesture Recognition on a Low-power Microcontroller using IMU Sensors. In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA). IEEE, 733–736.
[31]
Hansheng Lei and Venu Govindaraju. 2004. Direct image matching by dynamic warping. In 2004 Conference on Computer Vision and Pattern Recognition Workshop. IEEE, 76–76.
[32]
Barbara Leporini, Maria Claudia Buzzi, and Marina Buzzi. 2012. Interacting with mobile devices via VoiceOver: usability and accessibility issues. In Proceedings of the 24th Australian Computer-Human Interaction Conference. 339–348.
[33]
Tzuu-Hseng S. Li, Min-Chi Kao, and Ping-Huan Kuo. 2016. Recognition System for Home-Service-Related Sign Language Using Entropy-Based K -Means Algorithm and ABC-Based HMM. IEEE Transactions on Systems, Man, and Cybernetics: Systems 46, 1 (2016), 150–162. https://doi.org/10.1109/TSMC.2015.2435702
[34]
Guanhong Liu, Yizheng Gu, Yiwen Yin, Chun Yu, Yuntao Wang, Haipeng Mi, and Yuanchun Shi. 2020. Keep the Phone in Your Pocket: Enabling Smartphone Operation with an IMU Ring for Visually Impaired People. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 2 (2020), 1–23.
[35]
Siyu Liu, Jian Chen, Cheng Wang, and Lin Lin. 2023. Ultrasonic positioning and IMU data fusion for pen-based 3D hand gesture recognition. Multimedia Tools and Applications (2023), 1–19.
[36]
Yuqi Luo, Jiang Liu, and Shigeru Shimamoto. 2021. Wearable air-writing recognition system employing dynamic time warping. In 2021 IEEE 18th Annual Consumer Communications & Networking Conference (CCNC). IEEE, 1–6.
[37]
Yuntao Ma, Yuxuan Liu, Ruiyang Jin, Xingyang Yuan, Raza Sekha, Samuel Wilson, and Ravi Vaidyanathan. 2017. Hand gesture recognition with convolutional neural networks for the multimodal UAV control. In 2017 Workshop on Research, Education and Development of Unmanned Aerial Systems (RED-UAS). IEEE, 198–203.
[38]
Ziyang Ma and Enhua Wu. 2014. Real-time and robust hand tracking with a single depth camera. The Visual Computer 30 (2014), 1133–1144.
[39]
Oleg Makaussov, Mikhail Krassavin, Maxim Zhabinets, and Siamac Fazli. 2020. A low-cost, IMU-based real-time on device gesture recognition glove. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC). IEEE, 3346–3351.
[40]
Meethu Malu, Pramod Chundury, and Leah Findlater. 2018. Exploring accessible smartwatch interactions for people with upper body motor impairments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–12.
[41]
Meinard Müller. 2007. Dynamic time warping. Information retrieval for music and motion (2007), 69–84.
[42]
Chaithanya Kumar Mummadi, Frederic Philips Peter Leo, Keshav Deep Verma, Shivaji Kasireddy, Philipp M Scholl, Jochen Kempfle, and Kristof Van Laerhoven. 2018. Real-time and embedded detection of hand gestures with an IMU-based glove. In Informatics, Vol. 5. MDPI, 28.
[43]
Lorenzo Porzi, Stefano Messelodi, Carla Mara Modena, and Elisa Ricci. 2013. A smart watch-based gesture recognition system for assisting people with visual impairments. In Proceedings of the 3rd ACM international workshop on Interactive multimedia on mobile & portable devices. 19–24.
[44]
Barbara Šepić, Abdurrahman Ghanem, and Stephan Vogel. 2015. BrailleEasy: one-handed braille keyboard for smartphones. In Assistive Technology. IOS Press, 1030–1035.
[45]
Ionut Cristian Severin, Dan Marius Dobrea, and Monica Claudia Dobrea. 2020. Head gesture recognition using a 6dof inertial IMU. International Journal of Computers Communications & Control 15, 3 (2020).
[46]
Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, 2015. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the 33rd annual ACM conference on human factors in computing systems. 3633–3642.
[47]
Sheng Shen, Mahanth Gowda, and Romit Roy Choudhury. 2018. Closing the gaps in inertial motion tracking. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking. 429–444.
[48]
Guangyi Shi, Yuexian Zou, Yufeng Jin, Xiaole Cui, and Wen J Li. 2009. Towards HMM based human motion recognition using MEMS inertial sensors. In 2008 IEEE International Conference on Robotics and Biomimetics. IEEE, 1762–1766.
[49]
Peter B Shull, Shuo Jiang, Yuhui Zhu, and Xiangyang Zhu. 2019. Hand gesture recognition and finger angle estimation via wrist-worn modified barometric pressure sensing. IEEE Transactions on Neural Systems and Rehabilitation Engineering 27, 4 (2019), 724–732.
[50]
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Miller. 2015. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision. 945–953.
[51]
Mingxing Tan and Quoc Le. 2019. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. PMLR, 6105–6114.
[52]
Edwin Valarezo Añazco, Seung Ju Han, Kangil Kim, Patricio Rivera Lopez, Tae-Seong Kim, and Sangmin Lee. 2021. Hand gesture recognition using single patchable six-axis inertial measurement unit via recurrent neural networks. Sensors 21, 4 (2021), 1404.
[53]
Tijana Vuletic, Alex Duffy, Laura Hay, Chris McTeague, Gerard Campbell, and Madeleine Grealy. 2019. Systematic literature review of hand gestures used in human computer interaction interfaces. International Journal of Human-Computer Studies 129 (2019), 74–94.
[54]
Danping Wang, Jina Wang, Yang Liu, and Xianming Meng. 2023. HMM-based IMU data processing for arm gesture classification and motion tracking. International Journal of Modelling, Identification and Control 42, 1 (2023), 54–63.
[55]
Zhengjie Wang, Yushan Hou, Kangkang Jiang, Wenwen Dou, Chengming Zhang, Zehua Huang, and Yinjing Guo. 2019. Hand gesture recognition based on active ultrasonic sensing of smartphone: a survey. IEEE Access 7 (2019), 111897–111922.
[56]
Eric W Weisstein. 2009. Euler angles. https://mathworld. wolfram. com/ (2009).
[57]
Hongyi Wen, Julian Ramos Rojas, and Anind K Dey. 2016. Serendipity: Finger gesture recognition using an off-the-shelf smartwatch. In Proceedings of the 2016 CHI conference on human factors in computing systems. 3847–3851.
[58]
Renqiang Xie and Juncheng Cao. 2016. Accelerometer-based hand gesture recognition by neural network and similarity matching. IEEE Sensors Journal 16, 11 (2016), 4537–4545.
[59]
Jian Xu, Syed Masum Billah, Roy Shilkrot, and Aruna Balasubramanian. 2019. DarkReader: bridging the gap between perception and reality of power consumption in smartphones for blind users. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility. 96–104.
[60]
Ruize Xu, Shengli Zhou, and Wen J Li. 2011. MEMS accelerometer based nonspecific-user hand gesture recognition. IEEE sensors journal 12, 5 (2011), 1166–1173.
[61]
Xuhai Xu, Jun Gong, Carolina Brum, Lilian Liang, Bongsoo Suh, Shivam Kumar Gupta, Yash Agarwal, Laurence Lindsey, Runchang Kang, Behrooz Shahsavari, 2022. Enabling hand gesture customization on wrist-worn devices. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–19.
[62]
Chan-Yun Yang, Pei-Yu Chen, Te-Jen Wen, and Gene Eu Jan. 2019. Imu consensus exception detection with dynamic time warping—a comparative approach. Sensors 19, 10 (2019), 2237.
[63]
Ying Yin and Randall Davis. 2014. Real-time continuous gesture recognition for natural human-computer interaction. In 2014 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 113–120.
[64]
Guan Yuan, Xiao Liu, Qiuyan Yan, Shaojie Qiao, Zhixiao Wang, and Li Yuan. 2020. Hand gesture recognition using deep feature fusion network based on wearable sensors. IEEE Sensors Journal 21, 1 (2020), 539–547.
[65]
Yu Zhang, Tao Gu, Chu Luo, Vassilis Kostakos, and Aruna Seneviratne. 2018. FinDroidHR: Smartwatch gesture input with optical heartrate monitor. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1–42.
[66]
Yang Zhang and Chris Harrison. 2015. Tomo: Wearable, low-cost electrical impedance tomography for hand gesture recognition. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. 167–173.
Index Terms
- Hand Gesture Recognition for Blind Users by Tracking 3D Gesture Trajectory
Recommendations
Multi-scenario gesture recognition using Kinect
CGAMES '12: Proceedings of the 2012 17th International Conference on Computer Games: AI, Animation, Mobile, Interactive Multimedia, Educational & Serious Games (CGAMES)Hand gesture recognition (HGR) is an important research topic because some situations require silent communication with sign languages. Computational HGR systems assist silent communication, and help people learn a sign language. In this article, a ...
A Method for Hand Gesture Recognition
CSNT '14: Proceedings of the 2014 Fourth International Conference on Communication Systems and Network TechnologiesIn this paper, we present a method for hand gesture recognition using Microsoft Kinect sensor. Kinect allows capturing dense, and three dimensional scans of an object in real time. We propose a combination of modelling and learning approach for hand ...
Pointing gesture recognition based on 3D-tracking of face, hands and head orientation
ICMI '03: Proceedings of the 5th international conference on Multimodal interfacesIn this paper, we present a system capable of visually detecting pointing gestures and estimating the 3D pointing direction in real-time. In order to acquire input features for gesture recognition, we track the positions of a person's face and hands on ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
May 2024
18961 pages
ISBN:9798400703300
DOI:10.1145/3613904
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 11 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Funding Sources
- NIH (National Institutes of Health)
- Google Inclusion Research Award
- Google Research Scholar Award
- NSF (National Science Foundation)
Conference
Acceptance Rates
Overall Acceptance Rate 6,199 of 26,314 submissions, 24%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 816Total Downloads
- Downloads (Last 12 months)816
- Downloads (Last 6 weeks)116
Reflects downloads up to 16 Dec 2024
Other Metrics
Citations
Cited By
View all- Shin JMiah AKabir MRahim MAl Shiam A(2024)A Methodological and Structural Review of Hand Gesture Recognition Across Diverse Data ModalitiesIEEE Access10.1109/ACCESS.2024.345643612(142606-142639)Online publication date: 2024
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderFull Text
View this article in Full Text.
Full TextHTML Format
View this article in HTML Format.
HTML Format