More Web Proxy on the site http://driver.im/

research-article

Public Access

SeleCon: Scalable IoT Device Selection and Control Using Hand Gestures

Authors:

Moustafa Alzantot,

Mani SrivastavaAuthors Info & Claims

IoTDI '17: Proceedings of the Second International Conference on Internet-of-Things Design and Implementation

Pages 47 - 58

https://doi.org/10.1145/3054977.3054981

Published: 18 April 2017 Publication History

Abstract

Although different interaction modalities have been proposed in the field of human-computer interface (HCI), only a few of these techniques could reach the end users because of scalability and usability issues. Given the popularity and the growing number of IoT devices, selecting one out of many devices becomes a hurdle in a typical smarthome environment. Therefore, an easy-to-learn, scalable, and non-intrusive interaction modality has to be explored. In this paper, we propose a pointing approach to interact with devices, as pointing is arguably a natural way for device selection. We introduce SeleCon for device selection and control which uses an ultra-wideband (UWB) equipped smartwatch. To interact with a device in our system, people can point to the device to select it then draw a hand gesture in the air to specify a control action. To this end, SeleCon employs inertial sensors for pointing gesture detection and a UWB transceiver for identifying the selected device from ranging measurements. Furthermore, SeleCon supports an alphabet of gestures that can be used for controlling the selected devices. We performed our experiment in a 9m-by-10m lab space with eight deployed devices. The results demonstrate that SeleCon can achieve 84.5% accuracy for device selection and 97% accuracy for hand gesture recognition. We also show that SeleCon is power efficient to sustain daily use by turning off the UWB transceiver, when a user's wrist is stationary.

References

[1]

Amazon Echo. http://amazon.com/echo.

[2]

Google Home. https://madeby.google.com/home/.

[3]

Microsoft Kinect. https://developer.microsoft.com/en-us/windows/kinect.

[4]

Nintendo Wii. http://www.nintendo.com/wii.

[5]

OptiTrack Motion Capture System. http://www.optitrack.com.

[6]

Fadel Adib, Zachary Kabelac, Dina Katabi, and Robert C. Miller. 2014. 3D Tracking via Body Radio Reflections. In Proceedings of the 11th USENIX Conference on Networked Systems Design and Implementation (NSDI'14). USENIX Association, Berkeley, CA, USA, 317--329. http://dl.acm.org/citation.cfm?id=2616448.2616478

Digital Library

[7]

Fadel Adib and Dina Katabi. 2013. See Through Walls with WiFi!. In Proceedings of the ACM SIGCOMM 2013 Conference on SIGCOMM (SIGCOMM '13). ACM, New York, NY, USA, 75--86.

Digital Library

[8]

Sandip Agrawal, Ionut Constandache, Shravan Gaonkar, Romit Roy Choudhury, Kevin Caves, and Frank DeRuyter. 2011. Using Mobile Phones to Write in Air. In Proceedings of the 9th International Conference on Mobile Systems, Applications, and Services (MobiSys '11). ACM, New York, NY, USA, 15--28.

Digital Library

[9]

Kamran Ali, Alex X. Liu, Wei Wang, and Muhammad Shahzad. 2015. Keystroke Recognition Using WiFi Signals. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (MobiCom '15). ACM, New York, NY, USA, 90--102.

Digital Library

[10]

Fernando Alonso-Martín, Aívaro Castro-González, Francisco Javier Fernandez de Gorostiza Luengo, and Miguel Ángel Salichs. 2015. Augmented Robotics Dialog System for Enhancing Human--Robot Interaction. Sensors 15, 7 (2015), 15799--15829.

[11]

L. Ardser, P. Bissig, P. Brandes, and R. Wattenhofer. 2016. Recognizing text using motion data from a smartwatch. In 2016 IEEE International Conference on Pervasive Computing and Communication Workshops (PerCom Workshops). 1--6.

[12]

Parvin Asadzadeh, Lars Kulik, and Egemen Tanin. 2012. Gesture recognition using RFID technology. Personal and Ubiquitous Computing 16, 3 (2012), 225--234.

Digital Library

[13]

Cheng Bo, Xuesi Jian, Xiang-Yang Li, Xufei Mao, Yu Wang, and Fan Li. 2013. You're driving and texting: detecting drivers using personal smart phones by leveraging inertial sensors. In Proceedings of the 19th annual international conference on Mobile computing & networking. ACM, 199--202.

Digital Library

[14]

Kevin Bouchard, Abdenour Bouzouane, and Bruno Bouchard. 2014. Gesture Recognition in Smart Home Using Passive RFID Technology. In Proceedings of the 7th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '14). ACM, New York, NY, USA, Article 12, 8 pages.

Digital Library

[15]

Feng-Sheng Chen, Chih-Ming Fu, and Chung-Lin Huang. 2003. Hand gesture recognition using a real-time tracking method and hidden Markov models. Image and vision computing 21, 8 (2003), 745--758.

[16]

Gabe Cohn, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Humantenna: using the body as an antenna for real-time whole-body interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1901--1910.

Digital Library

[17]

Sinan Gezici, Zhi Tian, Georgios B Giannakis, Hisashi Kobayashi, Andreas F Molisch, H Vincent Poor, and Zafer Sahinoglu. 2005. Localization via ultra-wideband radios: a look at positioning aspects for future sensor networks. IEEE signal processing magazine 22, 4 (2005), 70--84.

[18]

SS Ghassemzadeh, LJ Greenstein, A Kavcic, T Sveinsson, and V Tarokh. 2003. UWB indoor path loss model for residential and commercial buildings. In Vehicular Technology Conference, 2003. VTC 2003-Fall. 2003 IEEE 58th, Vol. 5. IEEE, 3115--3119.

[19]

Sidhant Gupta, Daniel Morris, Shwetak Patel, and Desney Tan. 2012. Sound-Wave: Using the Doppler Effect to Sense Gestures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, New York, NY, USA, 1911--1914.

Digital Library

[20]

Chunmei Han, Kaishun Wu, Yuxi Wang, and Lionel M Ni. 2014. WiFall: Device-free fall detection by wireless networks. In IEEE INFOCOM 2014-IEEE Conference on Computer Communications. IEEE, 271--279.

[21]

Jun Han, Emmanuel Owusu, Le T Nguyen, Adrian Perrig, and Joy Zhang. 2012. Accomplice: Location inference using accelerometers on smartphones. In 2012 Fourth International Conference on Communication Systems and Networks (COMSNETS 2012). IEEE, 1--9.

[22]

Bo-Jhang Ho, Hsin-Liu Cindy Kao, Nan-Chen Chen, Chuang-Wen You, Hao-Hua Chu, and Ming-Syan Chen. 2011. HeatProbe: a thermal-based power meter for accounting disaggregated electricity usage. In Proceedings of the 13th international conference on Ubiquitous computing. ACM, 55--64.

Digital Library

[23]

DecaWave DW1000 IR-UWB. http://www.decawave.com/products/dw1000.

[24]

Pan Jing and Guan Ye-peng. 2013. Human-computer interaction using pointing gesture based on an adaptive virtual touch screen. International Journal of Signal Processing, Image Processing and Pattern Recognition 6, 4 (2013), 81--92.

[25]

Holger Junker, Paul Lukowicz, and Gerhard Tröster. 2004. Continuous Recognition of Arm Activities With Body-Worn Inertial Sensors. In ISWC 188--189.

Digital Library

[26]

Hamed Ketabdar, Peyman Moghadam, Babak Naderi, and Mehran Roshandel. 2012. Magnetic Signatures in Air for Mobile Devices. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services Companion (MobileHCI '12). ACM, New York, NY, USA, 185--188.

Digital Library

[27]

Youngwook Kim and Hao Ling. 2009. Human activity classification based on micro-Doppler signatures using a support vector machine. IEEE Transactions on Geoscience and Remote Sensing 47, 5 (2009), 1328--1337.

[28]

Eric Larson, Gabe Cohn, Sidhant Gupta, Xiaofeng Ren, Beverly Harrison, Dieter Fox, and Shwetak N. Patel. HeatWave: thermal imaging for surface user interaction. In In Proc of CHI 2011. 2565--2574.

Digital Library

[29]

Won Oh Lee, Yeong Gon Kim, Hyung Gil Hong, and Kang Ryoung Park. 2014. Face recognition system for set-top box-based intelligent TV. Sensors 14, 11 (2014), 21726--21749.

[30]

Hong Li, Wei Yang, Jianxin Wang, Yang Xu, and Liusheng Huang. 2016. WiFinger: Talk to Your Smart Devices with Finger-grained Gesture. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 250--261.

Digital Library

[31]

Asier Lopez-Basterretxea, Amaia Mendez-Zorrilla, and Begoña Garcia-Zapirain. 2015. Eye/head tracking technology to improve HCI with iPad applications. Sensors 15, 2 (2015), 2244--2264.

[32]

Philip Marquardt, Arunabh Verma, Henry Carter, and Patrick Traynor. 2011. (sp) iPhone: decoding vibrations from nearby keyboards using mobile phone accelerometers. In Proceedings of the 18th ACM conference on Computer and communications security. ACM, 551--562.

Digital Library

[33]

Brian D Mayton, Nan Zhao, Matt Aldrich, Nicholas Gillian, and Joseph A Paradiso. 2013. WristQue: A personal sensor wristband. In 2013 IEEE International Conference on Body Sensor Networks. IEEE, 1--6.

[34]

Brice Michoud, Erwan Guillou, and Saïda Bouakaz. 2007. Real-time and markerless 3D human motion capture using multiple views. In Human Motion--Understanding, Modeling, Capture and Animation. Springer, 88--103.

[35]

Rajalakshmi Nandakumar, Bryce Kellogg, and Shyamnath Gollakota. 2014. Wi-fi gesture recognition on existing devices. arXiv preprint arXiv:1411.5394 (2014).

[36]

Emmanuel Owusu, Jun Han, Sauvik Das, Adrian Perrig, and Joy Zhang. 2012. ACCessory: password inference using accelerometers on smartphones. In Proceedings of the Twelfth Workshop on Mobile Computing Systems & Applications. ACM, 9.

Digital Library

[37]

Taiwoo Park, Jinwon Lee, Inseok Hwang, Chungkuk Yoo, Lama Nachman, and Junehwa Song. 2011. E-Gesture: A Collaborative Architecture for Energy-efficient Gesture Recognition with Hand-worn Sensor and Mobile Devices. In Proceedings of the 9th ACM Conference on Embedded Networked Sensor Systems (SenSys '11). ACM, New York, NY, USA, 260--273.

Digital Library

[38]

Lorenzo Porzi, Stefano Messelodi, Carla Mara Modena, and Elisa Ricci. 2013. A smart watch-based gesture recognition system for assisting people with visual impairments. In Proceedings of the 3rd ACM international workshop on Interactive multimedia on mobile & portable devices. ACM, 19--24.

Digital Library

[39]

Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. ACM, 27--38.

Digital Library

[40]

James M. Rehg and Takeo Kanade. 1994. Visual Tracking of High DOF Articulated Structures: An Application to Human Hand Tracking. In Proceedings of the Third European Conference-Volume II on Computer Vision - Volume II (ECCV '94). Springer-Verlag, London, UK, UK, 35--46. http://dl.acm.org/citation.cfm?id=645308.649148

[41]

Jun Rekimoto. 2001. Gesturewrist and gesturepad: Unobtrusive wearable interaction devices. In Wearable Computers, 2001. Proceedings. Fifth International Symposium on. IEEE, 21--27.

Digital Library

[42]

Sheng Shen, He Wang, and Romit Roy Choudhury. 2016. I Am a Smartwatch and I Can Track My User's Arm. In Proceedings of the 14th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '16). ACM, New York, NY, USA, 85--96.

Digital Library

[43]

Thad Starner and Alex Pentland. 1997. Real-time american sign language recognition from video using hidden markov models. In Motion-Based Recognition. Springer, 227--243.

[44]

Guanhua Wang, Yongpan Zou, Zimu Zhou, Kaishun Wu, and Lionel M Ni. 2014. We can hear you with wi-fi!. In Proceedings of the 20th annual international conference on Mobile computing and networking. ACM, 593--604.

Digital Library

[45]

Yan Wang, Jian Liu, Yingying Chen, Marco Gruteser, Jie Yang, and Hongbo Liu. 2014. E-eyes: Device-free Location-oriented Activity Identification Using Fine-grained WiFi Signatures. In Proceedings of the 20th Annual International Conference on Mobile Computing and Networking (MobiCom '14). ACM, New York, NY, USA, 617--628.

Digital Library

[46]

Hongyi Wen, Julian Ramos Rojas, and Anind K. Dey. 2016. Serendipity: Finger Gesture Recognition Using an Off-the-Shelf Smartwatch. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 3847--3851.

Digital Library

[47]

Wei Xi, Jizhong Zhao, Xiang-Yang Li, Kun Zhao, Shaojie Tang, Xue Liu, and Zhiping Jiang. 2014. Electronic frog eye: Counting crowd using wifi. In IEEE INFOCOM 2014-IEEE Conference on Computer Communications. IEEE, 361--369.

[48]

Chao Xu, Parth H Pathak, and Prasant Mohapatra. 2015. Finger-writing with smartwatch: A case for finger and hand gesture recognition using smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications. ACM, 9--14.

Digital Library

[49]

Xu Zhang, Xiang Chen, Wen-hui Wang, Ji-hai Yang, Vuokko Lantz, and Kong-qiao Wang. 2009. Hand Gesture Recognition and Virtual Game Control Based on 3D Accelerometer and EMG Sensors. In Proceedings of the 14th International Conference on Intelligent User Interfaces (IUI '09). ACM, New York, NY, USA, 401--406.

Digital Library

Cited By

Kim MGlenn AVeluri BLee YGebre EBagaria APatel SGollakota S(2024)IRIS: Wireless ring for vision-based smart home interactionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676327(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676327
Kida YChiba TKinoshita TMikami AYakoh TOgi T(2024)IoT Avatar: Turning Various Objects into AvatarsAdvances in Network-Based Information Systems10.1007/978-3-031-72325-4_39(398-405)Online publication date: 20-Sep-2024
https://doi.org/10.1007/978-3-031-72325-4_39
Gomes PCastro MNascimento T(2023)Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature ReviewProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638082(1-11)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638082
Show More Cited By

Recommendations

Evaluating the placement of arm-worn devices for recognizing variations of dynamic hand gestures

Dynamic hand gestures have become increasingly popular as touch-free input modality for interactive systems. There exists a variety of arm-worn devices for the recognition of hand gestures, which differ not only in their capabilities, but also in their ...
Mirrortablet: exploring a low-cost mobile system for capturing unmediated hand gestures in remote collaboration
MUM '17: Proceedings of the 16th International Conference on Mobile and Ubiquitous Multimedia

Direct and natural images of hand gestures have been shown to benefit remote collaboration on physical tasks in several settings, including ad-hoc ones. However, to capture such unmediated hand gestures within collaborative tasks, existing approaches ...
Comparing the Placement of Two Arm-Worn Devices for Recognizing Dynamic Hand Gestures
MoMM '16: Proceedings of the 14th International Conference on Advances in Mobile Computing and Multi Media

Dynamic hand gestures have become increasingly popular as an input modality for interactive systems. There exists a variety of arm-worn devices for the recognition of hand gestures, which differ not only in their capabilities, but also in the arm ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

IoTDI '17: Proceedings of the Second International Conference on Internet-of-Things Design and Implementation

April 2017

353 pages

ISBN:9781450349666

DOI:10.1145/3054977

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

IoTDI '17

Sponsor:

SIGBED

IoTDI '17: International Conference on Internet-of-Things Design and Implementation

April 18 - 21, 2017

PA, Pittsburgh, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
1,127
Total Downloads

Downloads (Last 12 months)159
Downloads (Last 6 weeks)23

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kim MGlenn AVeluri BLee YGebre EBagaria APatel SGollakota S(2024)IRIS: Wireless ring for vision-based smart home interactionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676327(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676327
Kida YChiba TKinoshita TMikami AYakoh TOgi T(2024)IoT Avatar: Turning Various Objects into AvatarsAdvances in Network-Based Information Systems10.1007/978-3-031-72325-4_39(398-405)Online publication date: 20-Sep-2024
https://doi.org/10.1007/978-3-031-72325-4_39
Gomes PCastro MNascimento T(2023)Gesture Recognition Methods Using Sensors Integrated into Smartwatches: Results of a Systematic Literature ReviewProceedings of the XXII Brazilian Symposium on Human Factors in Computing Systems10.1145/3638067.3638082(1-11)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3638067.3638082
Cai RJanaka NZhao SSun M(2023)ParaGlassMenu: Towards Social-Friendly Subtle Interactions in ConversationsProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581065(1-21)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581065
Qin YYu CYao WYao JLiang CWeng YYan YShi Y(2023)Selecting Real-World Objects via User-Perspective Phone OcclusionProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3580696(1-13)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3580696
Song ZDudley JKristensson P(2023)HotGestures: Complementing Command Selection and Use with Delimiter-Free Gesture-Based Shortcuts in Virtual RealityIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332025729:11(4600-4610)Online publication date: 1-Nov-2023
https://dl.acm.org/doi/10.1109/TVCG.2023.3320257
Lang MStrobel CWeckesser FLanglois DKasneci EKozlíková BKrone M(2023)A multimodal smartwatch-based interaction concept for immersive environmentsComputers and Graphics10.1016/j.cag.2023.10.010117:C(85-95)Online publication date: 1-Dec-2023
https://dl.acm.org/doi/10.1016/j.cag.2023.10.010
Cocha Toabanda EErazo MYoo S(2023)Gaze Tracking: A Survey of Devices, Libraries and ApplicationsModelling and Development of Intelligent Systems10.1007/978-3-031-27034-5_2(18-41)Online publication date: 26-Feb-2023
https://doi.org/10.1007/978-3-031-27034-5_2
Zhou HHuang JFan HRen GGao YDong W(2022)VSLink: A Fast and Pervasive Approach to Physical Cyber Space Interaction via Visual SLAM2022 18th International Conference on Mobility, Sensing and Networking (MSN)10.1109/MSN57253.2022.00054(271-278)Online publication date: Dec-2022
https://doi.org/10.1109/MSN57253.2022.00054
Li ABodanese EPoslad SHou TWu KLuo F(2022)A Trajectory-Based Gesture Recognition in Smart Homes Based on the Ultrawideband Communication SystemIEEE Internet of Things Journal10.1109/JIOT.2022.31850849:22(22861-22873)Online publication date: 15-Nov-2022
https://doi.org/10.1109/JIOT.2022.3185084
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents