[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article
Open access

BLEselect: Gestural IoT Device Selection via Bluetooth Angle of Arrival Estimation from Smart Glasses

Published: 11 January 2023 Publication History

Abstract

Spontaneous selection of IoT devices from the head-mounted device is key for user-centered pervasive interaction. BLEselect enables users to select an unmodified Bluetooth 5.1 compatible IoT device by nodding at, pointing at, or drawing a circle in the air around it. We designed a compact antenna array that fits on a pair of smart glasses to estimate the Angle of Arrival (AoA) of IoT and wrist-worn devices' advertising signals. We then developed a sensing pipeline that supports all three selection gestures with lightweight machine learning models, which are trained in real-time for both hand gestures. Extensive characterizations and evaluations show that our system is accurate, natural, low-power, and privacy-preserving. Despite the small effective size of the antenna array, our system achieves a higher than 90% selection accuracy within a 3 meters distance in front of the user. In a user study that mimics real-life usage cases, the overall selection accuracy is 96.7% for a diverse set of 22 participants in terms of age, technology savviness, and body structures.

Supplementary Material

zhang-2 (zhang-2.zip)
Supplemental movie, appendix, image and software files for, BLEselect: Gestural IoT Device Selection via Bluetooth Angle of Arrival Estimation from Smart Glasses

References

[1]
2018. Angle of Arrival BoosterPack. https://dev.ti.com/tirex/explore/node?devtools=BOOSTXL-AOA&node=AHckEvhg0Y3xs5rlangU2w__FUz-xrs__LATEST
[2]
2022. Human head. https://en.wikipedia.org/w/index.php?title=Human_head&oldid=1075721879 Page Version ID: 1075721879.
[3]
Abdul Rafey Aftab, Michael von der Beeck, and Michael Feld. 2020. You Have a Point There: Object Selection Inside an Automobile Using Gaze, Head Pose and Finger Pointing. In Proceedings of the 2020 International Conference on Multimodal Interaction (ICMI '20). Association for Computing Machinery, New York, NY, USA, 595--603. https://doi.org/10/gk88qb
[4]
Karan Ahuja, Sujeath Pareddy, Robert Xiao, Mayank Goel, and Chris Harrison. 2019. LightAnchors: Appropriating Point Lights for Spatially-Anchored Augmented Reality Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST '19). Association for Computing Machinery, New York, NY, USA, 189--196. https://doi.org/10.1145/3332165.3347884
[5]
Ashwin Ashok, Chenren Xu, Tam Vu, Marco Gruteser, Rich Howard, Yanyong Zhang, Narayan Mandayam, Wenjia Yuan, and Kristin Dana. 2016. What Am I Looking At? Low-Power Radio-Optical Beacons for In-View Recognition on Smart-Glass. IEEE Transactions on Mobile Computing 15, 12 (Dec. 2016), 3185--3199. https://doi.org/10.1109/TMC.2016.2522967 Conference Name: IEEE Transactions on Mobile Computing.
[6]
Md Tanvir Islam Aumi, Sidhant Gupta, Mayank Goel, Eric Larson, and Shwetak Patel. 2013. DopLink: Using the Doppler Effect for Multi-device Interaction. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '13). ACM, New York, NY, USA, 583--586. https://doi.org/10/gfsvhm
[7]
Zonglong Bai, Liming Shi, Jesper Rindom Jensen, Jinwei Sun, and Mads Grææsbøll Christensen. 2021. Acoustic DOA estimation using space alternating sparse Bayesian learning. EURASIP Journal on Audio, Speech, and Music Processing 2021, 1 (April 2021), 14. https://doi.org/10.1186/s13636-021-00200-z
[8]
Andreas Biri, Neal Jackson, Lothar Thiele, Pat Pannuto, and Prabal Dutta. 2020. SociTrack: infrastructure-free interaction tracking through mobile sensor networks. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. ACM, London United Kingdom, 1--14. https://doi.org/10.1145/3372224.3419190
[9]
Leo Botler, Michael Spörk, Konrad Diwold, and Kay Römer. 2020. Direction Finding with UWB and BLE: A Comparative Study. In 2020 IEEE 17th International Conference on Mobile Ad Hoc and Sensor Systems (MASS). 44--52. https://doi.org/10.1109/MASS50613.2020.00016 ISSN: 2155-6814.
[10]
Kaifei Chen, Jonathan Fürst, John Kolb, Hyung-Sin Kim, Xin Jin, David E. Culler, and Randy H. Katz. 2018. SnapLink: Fast and Accurate Vision-Based Appliance Control in Large Commercial Buildings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4 (Jan. 2018), 129:1--129:27. https://doi.org/10/gfhwdh
[11]
Li-Xuan Chuo, Zhihong Luo, Dennis Sylvester, David Blaauw, and Hun-Seok Kim. 2017. RF-Echo: A Non-Line-of-Sight Indoor Localization System Using a Low-Power Active RF Reflector ASIC Tag. In Proceedings of the 23rd Annual International Conference on Mobile Computing and Networking (MobiCom '17). Association for Computing Machinery, New York, NY, USA, 222--234. https://doi.org/10.1145/3117811.3117840
[12]
Marco Cominelli, Paul Patras, and Francesco Gringoli. 2019. Dead on Arrival: An Empirical Study of The Bluetooth 5.1 Positioning System. In Proceedings of the 13th International Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization (WiNTECH '19). Association for Computing Machinery, New York, NY, USA, 13--20. https://doi.org/10/gg9m69
[13]
Adrian A. de Freitas, Michael Nebeling, Xiang 'Anthony' Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K. Dey. 2016. Snap-To-It: A User-Inspired Platform for Opportunistic Device Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5909--5920. https://doi.org/10/gg4nkk
[14]
Ziheng Ding, Jingfeng Chen, Chong He, and Ronghong Jin. 2022. Elevation and Azimuth Direction Finding by Two-Element Pattern Reconfigurable Antenna Array. IEEE Transactions on Antennas and Propagation 70, 3 (March 2022), 2261--2270. https://doi.org/10.1109/TAP.2021.3118820 Conference Name: IEEE Transactions on Antennas and Propagation.
[15]
Egils Ginters and Jorge Martin-Gutierrez. 2013. Low Cost Augmented Reality and RFID Application for Logistics Items Visualization. Procedia Computer Science 26 (Jan. 2013), 3--13. https://doi.org/10.1016/j.procs.2013.12.002
[16]
Cory Hekimian-Williams, Brandon Grant, Xiuwen Liu, Zhenghao Zhang, and Piyush Kumar. 2010. Accurate localization of RFID tags using phase difference. In 2010 IEEE International Conference on RFID (IEEE RFID 2010). 89--96. https://doi.org/10.1109/RFID.2010.5467268 ISSN: 2374-0221.
[17]
Jie Hua, Sangsu Lee, Gruia-Catalin Roman, and Christine Julien. 2021. ArcIoT: Enabling Intuitive Device Control in the Internet of Things through Augmented Reality. In 2021 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). IEEE, 558--564. https://doi.org/10/gpnd3w
[18]
Dongsik Jo and Gerard Jounghyun Kim. 2016. ARIoT: scalable augmented reality framework for interacting with Internet of Things appliances everywhere. IEEE Transactions on Consumer Electronics 62, 3 (Aug. 2016), 334--340. https://doi.org/10.1109/TCE.2016.7613201 Conference Name: IEEE Transactions on Consumer Electronics.
[19]
Kyu-Han Kim. 2019. When IoT met Augmented Reality: Visualizing the Source of the Wireless Signal in AR View. In Proceedings of the 17th Annual International Conference on Mobile Systems, Applications, and Services. ACM, Seoul Republic of Korea, 117--129. https://doi.org/10/gkfcrs
[20]
Quan Kong, Takuya Maekawa, Taiki Miyanishi, and Takayuki Suyama. 2016. Selecting Home Appliances with Smart Glass Based on Contextual Information. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '16). ACM, New York, NY, USA, 97--108. https://doi.org/10/gnc4j4
[21]
Manikanta Kotaru, Kiran Joshi, Dinesh Bharadia, and Sachin Katti. 2015. SpotFi: Decimeter Level Localization Using WiFi. ACM SIGCOMM Computer Communication Review 45, 4 (Aug. 2015), 269--282. https://doi.org/10.1145/2829988.2787487
[22]
Hanchuan Li, Eric Whitmire, Alex Mariakakis, Victor Chan, Alanson P. Sample, and Shwetak N. Patel. 2019. IDCam: Precise Item Identification for AR Enhanced Object Interactions. In 2019 IEEE International Conference on RFID (RFID). 1--7. https://doi.org/10.1109/RFID.2019.8719279 ISSN: 2573-7635.
[23]
Robert LiKamWa, Zhen Wang, Aaron Carroll, Felix Xiaozhu Lin, and Lin Zhong. 2014. Draining our glass: an energy and heat characterization of Google Glass. In Proceedings of 5th Asia-Pacific Workshop on Systems (APSys '14). Association for Computing Machinery, New York, NY, USA, 1--7. https://doi.org/10.1145/2637166.2637230
[24]
Sikun Lin, Hao Fei Cheng, Weikai Li, Zhanpeng Huang, Pan Hui, and Christoph Peylo. 2017. Ubii: Physical World Interaction Through Augmented Reality. IEEE Transactions on Mobile Computing 16, 3 (March 2017), 872--885. https://doi.org/10.1109/tmc.2016.2567378 Conference Name: IEEE Transactions on Mobile Computing.
[25]
Xuefeng Liu, Tianye Yang, Shaojie Tang, Peng Guo, and Jianwei Niu. 2020. From relative azimuth to absolute location: pushing the limit of PIR sensor based localization. In Proceedings of the 26th Annual International Conference on Mobile Computing and Networking. Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3372224.3380878
[26]
R. Mayrhofer and H. Gellersen. 2009. Shake Well Before Use: Intuitive and Secure Pairing of Mobile Devices. IEEE Transactions on Mobile Computing 8, 6 (June 2009), 792--806. https://doi.org/10/bqk9kd
[27]
Georg Oberholzer, Philipp Sommer, and Roger Wattenhofer. 2011. SpiderBat: Augmenting wireless sensor networks with distance and angle information. In Proceedings of the 10th ACM/IEEE International Conference on Information Processing in Sensor Networks. 211--222.
[28]
Michael Parker. 2010. Chapter 18 - Synthetic Array Radar. In Digital Signal Processing 101, Michael Parker (Ed.). Newnes, Boston, 213--222. https://doi.org/10.1016/B978-1-85617-921-8.00022-5
[29]
Shwetak N. Patel and Gregory D. Abowd. 2003. A 2-Way Laser-Assisted Selection Scheme for Handhelds in a Physical Environment. In UbiComp 2003: Ubiquitous Computing (Lecture Notes in Computer Science). Springer, Berlin, Heidelberg, 200--207. https://doi.org/10.1007/978-3-540-39653-6_16
[30]
Giovanni Pau, Fabio Arena, Yonas Engida Gebremariam, and Ilsun You. 2021. Bluetooth 5.1: An Analysis of Direction Finding Capability for High-Precision Location Services. Sensors 21, 11 (May 2021), 3589. https://doi.org/10.3390/s21113589
[31]
Chunyi Peng, Guobin Shen, Yongguang Zhang, and Songwu Lu. 2009. Point&Connect: Intention-based Device Pairing for Mobile Phone Users. In Proceedings of the 7th International Conference on Mobile Systems, Applications, and Services (MobiSys '09). ACM, New York, NY, USA, 137--150. https://doi.org/10/b6rcdg
[32]
Felix Putze, Dennis Weiß, Lisa-Marie Vortmann, and Tanja Schultz. 2019. Augmented Reality Interface for Smart Home Control using SSVEP-BCI and Eye Gaze. In 2019 IEEE International Conference on Systems, Man and Cybernetics (SMC). 2812--2817. https://doi.org/10.1109/SMC.2019.8914390 ISSN: 2577-1655.
[33]
Jonathan Rosales, Sourabh Deshpande, and Sam Anand. 2021. IIoT based Augmented Reality for Factory Data Collection and Visualization. Procedia Manufacturing 53 (Jan. 2021), 618--627. https://doi.org/10.1016/j.promfg.2021.06.062
[34]
Ludwig Sidenmark and Hans Gellersen. 2019. Eye&Head: Synergetic Eye and Head Movement for Gaze Pointing and Selection. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST '19). Association for Computing Machinery, New York, NY, USA, 1161--1174. https://doi.org/10.1145/3332165.3347921
[35]
Elahe Soltanaghaei, Adwait Dongare, Akarsh Prabhakara, Swarun Kumar, Anthony Rowe, and Kamin Whitehouse. 2021. TagFi: Locating Ultra-Low Power WiFi Tags Using Unmodified WiFi Infrastructure. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (March 2021), 1--29. https://doi.org/10/gjnjgp
[36]
Elahe Soltanaghaei, Akarsh Prabhakara, Artur Balanuta, Matthew Anderson, Jan M. Rabaey, Swarun Kumar, and Anthony Rowe. 2021. Millimetro: mmWave retro-reflective tags for accurate, long range localization. In Proceedings of the 27th Annual International Conference on Mobile Computing and Networking. ACM, New Orleans Louisiana, 69--82. https://doi.org/10/gjg3rb
[37]
Zheng Sun, Aveek Purohit, Raja Bose, and Pei Zhang. 2013. Spartacus: Spatially-aware Interaction for Mobile Devices Through Energy-efficient Audio Sensing. In Proceeding of the 11th Annual International Conference on Mobile Systems, Applications, and Services (MobiSys '13). ACM, New York, NY, USA, 263--276. https://doi.org/10/gk5cwm
[38]
David Verweij, Augusto Esteves, Vassilis-Javed Khan, and Saskia Bakker. 2017. Smart Home Control Using Motion Matching and Smart Watches. In Proceedings of the 2017 ACM International Conference on Interactive Surfaces and Spaces (ISS '17). ACM, New York, NY, USA, 466--468. https://doi.org/10/gk5cwj
[39]
Jue Wang and Dina Katabi. 2013. Dude, where's my card? RFID positioning that works with multipath and non-line of sight. In Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM (SIGCOMM '13). Association for Computing Machinery, New York, NY, USA, 51--62. https://doi.org/10.1145/2486001.2486029
[40]
Jue Wang, Deepak Vasisht, and Dina Katabi. 2014. RF-IDraw: virtual touch screen in the air using RF signals. In Proceedings of the 2014 ACM conference on SIGCOMM. ACM, Chicago Illinois USA, 235--246. https://doi.org/10.1145/2619239.2626330
[41]
Weiguo Wang, Jinming Li, Yuan He, and Yunhao Liu. 2020. Symphony: localizing multiple acoustic sources with a single microphone array. In Proceedings of the 18th Conference on Embedded Networked Sensor Systems. Association for Computing Machinery, New York, NY, USA, 82--94. https://doi.org/10.1145/3384419.3430724
[42]
Yuntao Wang, Jiexin Ding, Ishan Chatterjee, Farshid Salemi Parizi, Yuzhou Zhuang, Yukang Yan, Shwetak Patel, and Yuanchun Shi. 2022. FaceOri: Tracking Head Position and Orientation Using Ultrasonic Ranging on Earphones. In CHI Conference on Human Factors in Computing Systems (CHI '22). Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3491102.3517698
[43]
Robert Xiao, Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, New York, NY, USA, 4000--4008. https://doi.org/10/gg94n6
[44]
Jie Xiong and Kyle Jamieson. 2013. Arraytrack: A fine-grained indoor location system. In 10th ${$USENIX$}$ Symposium on Networked Systems Design and Implementation (${$NSDI$}$ 13). 71--84.
[45]
Yukang Yan, Yingtian Shi, Chun Yu, and Yuanchun Shi. 2020. HeadCross: Exploring Head-Based Crossing Selection on Head-Mounted Displays. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 1 (March 2020), 35:1--35:22. https://doi.org/10.1145/3380983
[46]
Jackie (Junrui) Yang and James A. Landay. 2019. InfoLED: Augmenting LED Indicator Lights for Device Positioning and Communication. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (UIST '19). Association for Computing Machinery, New York, NY, USA, 175--187. https://doi.org/10.1145/3332165.3347954
[47]
Lichen Yao. 2018. Bluetooth Direction Finding. (2018).
[48]
Ben Zhang, Yu-Hsiang Chen, Claire Tuna, Achal Dave, Yang Li, Edward Lee, and Björn Hartmann. 2014. HOBS: Head Orientation-based Selection in Physical Spaces. In Proceedings of the 2Nd ACM Symposium on Spatial User Interaction (SUI '14). ACM, New York, NY, USA, 17--25. https://doi.org/10/gk5cwk
[49]
Tengxiang Zhang, Xin Yi, Ruolin Wang, Jiayuan Gao, Yuntao Wang, Chun Yu, Simin Li, and Yuanchun Shi. 2019. Facilitating Temporal Synchronous Target Selection through User Behavior Modeling. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (Dec. 2019), 1--24. https://doi.org/10/gpnbc5
[50]
Tengxiang Zhang, Xin Yi, Ruolin Wang, Yuntao Wang, Chun Yu, Yiqin Lu, and Yuanchun Shi. 2018. Tap-to-Pair: Associating Wireless Devices with Synchronous Tapping. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 4 (Dec. 2018), 201:1--201:21. https://doi.org/10/gg7k4k
[51]
Tengxiang Zhang, Xin Zeng, Yinshuai Zhang, Ke Sun, Yuntao Wang, and Yiqiang Chen. 2020. ThermalRing: Gesture and Tag Inputs Enabled by a Thermal Imaging Smart Ring. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (CHI '20). Association for Computing Machinery, Honolulu, HI, USA, 1--13. https://doi.org/10.1145/3313831.3376323 00000.

Cited By

View all
  • (2025)Applying MLP-Mixer and gMLP to Human Activity RecognitionSensors10.3390/s2502031125:2(311)Online publication date: 7-Jan-2025
  • (2024)Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and ChallengesSensors10.3390/s2415504524:15(5045)Online publication date: 4-Aug-2024
  • (2024)Wearable Sensors as a Preoperative Assessment Tool: A ReviewSensors10.3390/s2402048224:2(482)Online publication date: 12-Jan-2024
  • Show More Cited By

Index Terms

  1. BLEselect: Gestural IoT Device Selection via Bluetooth Angle of Arrival Estimation from Smart Glasses

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
      Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 6, Issue 4
      December 2022
      1534 pages
      EISSN:2474-9567
      DOI:10.1145/3580286
      Issue’s Table of Contents
      Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 11 January 2023
      Published in IMWUT Volume 6, Issue 4

      Check for updates

      Author Tags

      1. IoT
      2. angle of arrival
      3. device selection
      4. gesture

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)397
      • Downloads (Last 6 weeks)51
      Reflects downloads up to 20 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Applying MLP-Mixer and gMLP to Human Activity RecognitionSensors10.3390/s2502031125:2(311)Online publication date: 7-Jan-2025
      • (2024)Large Language Models for Wearable Sensor-Based Human Activity Recognition, Health Monitoring, and Behavioral Modeling: A Survey of Early Trends, Datasets, and ChallengesSensors10.3390/s2415504524:15(5045)Online publication date: 4-Aug-2024
      • (2024)Wearable Sensors as a Preoperative Assessment Tool: A ReviewSensors10.3390/s2402048224:2(482)Online publication date: 12-Jan-2024
      • (2024)Unified Domain Adaptation for Specialized Indoor Scene Inpainting Using a Pre-Trained ModelElectronics10.3390/electronics1324497013:24(4970)Online publication date: 17-Dec-2024
      • (2024)SemiCMT: Contrastive Cross-Modal Knowledge Transfer for IoT Sensing with Semi-Paired Multi-Modal SignalsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997798:4(1-30)Online publication date: 21-Nov-2024
      • (2024)ContrastSense: Domain-invariant Contrastive Learning for In-the-Wild Wearable SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997448:4(1-32)Online publication date: 21-Nov-2024
      • (2024)GOAT: A Generalized Cross-Dataset Activity Recognition Framework with Natural Language SupervisionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36997368:4(1-28)Online publication date: 21-Nov-2024
      • (2024)RaDro: Indoor Drone Tracking Using Millimeter Wave RadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36785498:3(1-23)Online publication date: 9-Sep-2024
      • (2024)MLP-HAR: Boosting Performance and Efficiency of HAR Models on Edge Devices with Purely Fully Connected LayersProceedings of the 2024 ACM International Symposium on Wearable Computers10.1145/3675095.3676624(133-139)Online publication date: 5-Oct-2024
      • (2024)Predicting Multi-dimensional Surgical Outcomes with Multi-modal Mobile SensingProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36596288:2(1-30)Online publication date: 15-May-2024
      • Show More Cited By

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Login options

      Full Access

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media