More Web Proxy on the site http://driver.im/

research-article

What is That in Your Hand?: Recognizing Grasped Objects via Forearm Electromyography Sensing

Authors:

Hongan WangAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 2, Issue 4

Article No.: 161, Pages 1 - 24

https://doi.org/10.1145/3287039

Published: 27 December 2018 Publication History

Abstract

Knowing the object in hand can offer essential contextual information revealing a user's fine-grained activities. In this paper, we investigate the feasibility, accuracy, and robustness of recognizing the uninstrumented object in a user's hand by sensing and decoding her forearm muscular activities via off-the-shelf electromyography (EMG) sensors. We present results from three studies to advance our fundamental understanding of the opportunities that EMG brings in object interaction recognition. In the first study, we investigated the influence of physical properties of objects such as shape, size, and weight on EMG signals. We also conducted a thorough exploration of the feature spaces and sensor positions which can provide a solid base to rely on for future designers and practitioners for such interactive technique. In the second study, we assessed the feasibility and accuracy of inferring the types of grasped objects via using forearm muscular activity as a cue. Our results indicate that the types of objects can be recognized with up to 94.2% accuracy by employing user-dependent training. In the third study, we investigated the robustness of this approach in a realistic office setting where users were allowed to interact with objects as they would naturally. Our approach achieved up to 82.5% accuracy in discriminating 15 types of objects, even when training and testing phrases were purposefully performed on different days to incorporate changes in EMG patterns over time. Overall, this work contributes a set of fundamental findings and guidelines on using EMG technologies for object-based activity tracking.

Supplementary Material

fan (fan.zip)

Supplemental movie, appendix, image and software files for, What is That in Your Hand? Recognizing Grasped Objects via Forearm Electromyography Sensing

Download
81.96 KB

References

[1]

Jake K Aggarwal and Michael S Ryoo. 2011. Human activity analysis: A review. ACM Computing Surveys (CSUR) 43, 3 (2011), 16.

Digital Library

[2]

Christoph Amma, Thomas Krings, Jonas Böer, and Tanja Schultz. 2015. Advancing muscle-computer interfaces with high-density electromyography. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 929--938.

Digital Library

[3]

Ling Bao and Stephen S Intille. 2004. Activity recognition from user-annotated acceleration data. In International Conference on Pervasive Computing. Springer, 1--17.

[4]

Eugen Berlin, Jun Liu, Kristof Van Laerhoven, and Bernt Schiele. 2010. Coming to grips with the objects we grasp: detecting interactions with efficient wrist-worn sensors. In Proceedings of the fourth international conference on Tangible, embedded, and embodied interaction. ACM, 57--64.

Digital Library

[5]

Inc BIOPAC Systems. 2017. MP System Hardware Guide. Retrieved March 31, 2017 from https://www.biopac.com/wp-content/uploads/MP-Hardware-Guide.pdf

[6]

Christopher M Bishop. 2006. Pattern Recognition and Machine Learning (Information Science and Statistics). 4 (2006), 049901.

Digital Library

[7]

Michael Buettner, Richa Prasad, Matthai Philipose, and David Wetherall. 2009. Recognizing daily activities with RFID-based sensors. In Proceedings of the 11th international conference on Ubiquitous computing. ACM, 51--60.

Digital Library

[8]

Baptiste Caramiaux, Marco Donnarumma, and Atau Tanaka. 2015. Understanding gesture expressivity through muscle sensing. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 6 (2015), 31.

Digital Library

[9]

Graeme S Chambers, Svetha Venkatesh, Geoff AW West, and Hung Hai Bui. 2002. Hierarchical recognition of intentional human gestures for sports video annotation. In Pattern Recognition, 2002. Proceedings. 16th International Conference on, Vol. 2. IEEE, 1082--1085.

[10]

Varun Chandola, Arindam Banerjee, and Vipin Kumar. 2009. Anomaly detection: A survey. ACM computing surveys (CSUR) 41, 3 (2009), 15.

Digital Library

[11]

Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: a library for support vector machines. ACM transactions on intelligent systems and technology (TIST) 2, 3 (2011), 27.

Digital Library

[12]

Xun Chen and Z Jane Wang. 2013. Pattern recognition of number gestures based on a wireless surface EMG system. Biomedical Signal Processing and Control 8, 2 (2013), 184--192.

[13]

Christian Cipriani, Christian Antfolk, Marco Controzzi, Göran Lundborg, Birgitta Rosén, Maria Chiara Carrozza, and Fredrik Sebelius. 2011. Online myoelectric control of a dexterous hand prosthesis by transradial amputees. IEEE Transactions on Neural Systems and Rehabilitation Engineering 19, 3 (2011), 260--270.

[14]

Catherine Disselhorst-Klug, Thomas Schmitz-Rode, and Günter Rau. 2009. Surface electromyography and muscle force: Limits in sEMG--force relationship and new approaches for applications. Clinical biomechanics 24, 3 (2009), 225--235.

[15]

Nadine Fligge, Holger Urbanek, and Patrick van der Smagt. 2013. Relation between object properties and EMG during reaching to grasp. Journal of Electromyography and Kinesiology 23, 2 (2013), 402--410.

[16]

Tobias Grosse-Puppendahl, Sebastian Herber, Raphael Wimmer, Frank Englert, Sebastian Beck, Julian von Wilmsdorff, Reiner Wichert, and Arjan Kuijper. 2014. Capacitive near-field communication for ubiquitous interaction and perception. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 231--242.

Digital Library

[17]

Faizan Haque, Mathieu Nancel, and Daniel Vogel. 2015. Myopoint: Pointing and clicking using forearm mounted electromyography and inertial motion sensors. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3653--3656.

Digital Library

[18]

Chris Harrison, Robert Xiao, and Scott Hudson. 2012. Acoustic barcodes: passive, durable and inexpensive notched identification tags. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 563--568.

Digital Library

[19]

Donny Huang, Xiaoyi Zhang, T Scott Saponas, James Fogarty, and Shyamnath Gollakota. 2015. Leveraging dual-observable input for fine-grained thumb interaction using forearm EMG. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 523--528.

Digital Library

[20]

Kari Kuutti. 1996. Activity theory as a potential framework for human-computer interaction research. Context and consciousness: Activity theory and human-computer interaction 17 (1996).

Digital Library

[21]

Jennifer R Kwapisz, Gary M Weiss, and Samuel A Moore. 2011. Activity recognition using cell phone accelerometers. ACM SigKDD Explorations Newsletter 12, 2 (2011), 74--82.

Digital Library

[22]

Gierad Laput, Robert Xiao, and Chris Harrison. 2016. Viband: High-fidelity bio-acoustic sensing using commodity smartwatch accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 321--333.

Digital Library

[23]

Gierad Laput, Chouchang Yang, Robert Xiao, Alanson Sample, and Chris Harrison. 2015. Em-sense: Touch recognition of uninstrumented, electrical and electromechanical objects. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 157--166.

Digital Library

[24]

Matthew L Lee and Anind K Dey. 2015. Sensor-based observations of daily living for aging in place. Personal and Ubiquitous Computing 19, 1 (2015), 27--43.

Digital Library

[25]

Hanchuan Li, Can Ye, and Alanson P Sample. 2015. IDSense: A human object interaction detection system based on passive UHF RFID. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2555--2564.

Digital Library

[26]

Minas V Liarokapis, Panagiotis K Artemiadis, and Kostas J Kyriakopoulos. 2013. Task discrimination from myoelectric activity: a learning scheme for EMG-based interfaces. In Rehabilitation Robotics (ICORR), 2013 IEEE International Conference on. IEEE, 1--6.

[27]

Takuya Maekawa, Yasue Kishino, Yasushi Sakurai, and Takayuki Suyama. 2011. Recognizing the use of portable electrical devices with hand-worn magnetic sensors. In International Conference on Pervasive Computing. Springer, 276--293.

Digital Library

[28]

Jess McIntosh, Charlie McNeill, Mike Fraser, Frederic Kerber, Markus Löchtefeld, and Antonio Krüger. 2016. EMPress: Practical hand gesture classification with wrist-mounted EMG and pressure sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 2332--2342.

Digital Library

[29]

Mohammadreza Asghari Oskoei, Huosheng Hu, et al. 2008. Support vector machine-based classification scheme for myoelectric control applied to upper limb. IEEE Trans. Biomed. Engineering 55, 8 (2008), 1956--1965.

[30]

Brandon Paulson, Danielle Cummings, and Tracy Hammond. 2011. Object interaction detection using hand posture cues in an office setting. International journal of human-computer studies 69, 1-2 (2011), 19--29.

Digital Library

[31]

Angkoon Phinyomark, Franck Quaine, Sylvie Charbonnier, Christine Serviere, Franck Tarpin-Bernard, and Yann Laurillau. 2013. EMG feature evaluation for improving myoelectric pattern recognition robustness. Expert Systems with applications 40, 12 (2013), 4832--4840.

Digital Library

[32]

Jun Rekimoto and Yuji Ayatsuka. 2000. CyberCode: designing augmented reality environments with visual tags. In Proceedings of DARE 2000 on Designing augmented reality environments. ACM, 1--10.

Digital Library

[33]

Xiaofeng Ren and Matthai Philipose. 2009. Egocentric recognition of handled objects: Benchmark and analysis. In Computer Vision and Pattern Recognition Workshops, 2009. CVPR Workshops 2009. IEEE Computer Society Conference on. IEEE, 1--8.

[34]

T Scott Saponas, Desney S Tan, Dan Morris, and Ravin Balakrishnan. 2008. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 515--524.

Digital Library

[35]

T Scott Saponas, Desney S Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A Landay. 2009. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. ACM, 167--176.

Digital Library

[36]

T Scott Saponas, Desney S Tan, Dan Morris, Jim Turner, and James A Landay. 2010. Making muscle-computer interfaces more practical. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 851--854.

Digital Library

[37]

Georg Schlesinger. 1919. Der mechanische aufbau der künstlichen glieder. In Ersatzglieder und Arbeitshilfen. Springer, 321--661.

[38]

Isabel Suarez, Andreas Jahn, Christoph Anderson, and Klaus David. 2015. Improved activity recognition by using enriched acceleration data. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. ACM, 1011--1015.

Digital Library

[39]

Marian Theiss, Philipp M Scholl, and Kristof Van Laerhoven. 2016. Predicting grasps with a wearable inertial and EMG sensing unit for low-power detection of in-hand objects. In Proceedings of the 7th Augmented Human International Conference 2016. ACM, 4.

Digital Library

[40]

Radu-Daniel Vatavu and Ionuţ Alexandru Zaiţi. 2013. Automatic recognition of object size and shape via user-dependent measurements of the grasping hand. International Journal of Human-Computer Studies 71, 5 (2013), 590--607.

Digital Library

[41]

Jamie A Ward, Paul Lukowicz, Gerhard Troster, and Thad E Starner. 2006. Activity recognition of assembly tasks using body-worn microphones and accelerometers. IEEE transactions on pattern analysis and machine intelligence 28, 10 (2006), 1553--1567.

Digital Library

[42]

Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. Radarcat: Radar categorization for input & interaction. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 833--841.

Digital Library

[43]

Mahyar Zardoshti-Kermani, Bruce C Wheeler, Kambiz Badie, and Reza M Hashemi. 1995. EMG feature evaluation for movement control of upper extremity prostheses. IEEE Transactions on Rehabilitation Engineering 3, 4 (1995), 324--333.

Cited By

Li KIravantchi YZhu YPark HSample A(2025)HandSAW: Wearable Hand-based Event Recognition via On-Body Surface Acoustic WavesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/37122769:1(1-29)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3712276
Chaffangeon Caillet AGoguey ANigay L(2025)Microgesture ＋ GraspInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103398195:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103398
Chen WLin SPeng ZParizi FHeo SPatel SMatusik WZhao WStankovic J(2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643547
Show More Cited By

Index Terms

What is That in Your Hand?: Recognizing Grasped Objects via Forearm Electromyography Sensing
1. Human-centered computing
  1. Ubiquitous and mobile computing
    1. Empirical studies in ubiquitous and mobile computing

Recommendations

New dynamic muscle fatigue model to limit musculo-skeletal disorder
VRIC '16: Proceedings of the 2016 Virtual Reality International Conference

Muscle fatigue is one of the reasons leading to Musculo-Skeletal Disorder (MSD). Automation in industries reduced the human effort, but still there are many industries in which human have to do complex and repetitive tasks manually. The society/...
The Swing Control of Knee Exoskeleton Based on Admittance Model and Nonlinear Oscillator
Abstract
It is important to control exoskeleton to act synchronously with the user for movement assistance. A knee exoskeleton was constructed for gait rehabilitation training, and a control method based on adaptive frequency oscillator (AFO) was presented ...
Accuracy of Pedometry on a Head-mounted Display
CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

The accuracy of pedometry varies depending on where an inertial sensor is located on the body. Motivated by the increasing popularity of wearable computing, this paper investigates the accuracy with which pedometry can be achieved on a head-mounted ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 2, Issue 4

December 2018

1169 pages

EISSN:2474-9567

DOI:10.1145/3301777

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 December 2018

Accepted: 01 October 2018

Revised: 01 August 2018

Received: 01 May 2018

Published in IMWUT Volume 2, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

25
Total Citations
View Citations
734
Total Downloads

Downloads (Last 12 months)78
Downloads (Last 6 weeks)4

Reflects downloads up to 03 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Li KIravantchi YZhu YPark HSample A(2025)HandSAW: Wearable Hand-based Event Recognition via On-Body Surface Acoustic WavesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/37122769:1(1-29)Online publication date: 3-Mar-2025
https://dl.acm.org/doi/10.1145/3712276
Chaffangeon Caillet AGoguey ANigay L(2025)Microgesture ＋ GraspInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2024.103398195:COnline publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1016/j.ijhcs.2024.103398
Chen WLin SPeng ZParizi FHeo SPatel SMatusik WZhao WStankovic J(2024)ViObjectProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435478:1(1-26)Online publication date: 6-Mar-2024
https://dl.acm.org/doi/10.1145/3643547
Lee CZhang RAgarwal DYu TGunda VLopez OKim JYin SDong BLi KSakashita MGuimbretiere FZhang C(2024)EchoWrist: Continuous Hand Pose Tracking and Hand-Object Interaction Recognition Using Low-Power Active Acoustic Sensing On a WristbandProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642910(1-21)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642910
Gupta RKavanal VJoshi A(2024)Deep Learning Models for Recognition of Hand Gestures in Basketball Sports2024 1st International Conference on Cognitive, Green and Ubiquitous Computing (IC-CGU)10.1109/IC-CGU58078.2024.10530848(1-6)Online publication date: 1-Mar-2024
https://doi.org/10.1109/IC-CGU58078.2024.10530848
Zhang YLi TYuan YXu FYang FSun FLi Y(2023)Demand-Driven Urban Facility Visit PredictionACM Transactions on Intelligent Systems and Technology10.1145/3625233Online publication date: 9-Nov-2023
https://dl.acm.org/doi/10.1145/3625233
Shen ZDu WZhao XZou J(2023)Retrieving similar trajectories from cellular data of multiple carriers at city scaleACM Transactions on Sensor Networks10.1145/3613245Online publication date: 3-Aug-2023
https://dl.acm.org/doi/10.1145/3613245
Yin ZLiyanage MOttun APaul SZuniga ANurmi PFlores H(2023)HIPPOProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35703446:4(1-30)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3570344
He SMa WDong HXiao LJiang T(2023)C-CubeProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35703426:4(1-26)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3570342
Sharma ASalchow-Hömmen CMollyn VNittala AHedderich MKoelle MSeel TSteimle J(2023)SparseIMU: Computational Design of Sparse IMU Layouts for Sensing Fine-grained Finger MicrogesturesACM Transactions on Computer-Human Interaction10.1145/356989430:3(1-40)Online publication date: 10-Jun-2023
https://dl.acm.org/doi/10.1145/3569894
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents