More Web Proxy on the site http://driver.im/

research-article

Open access

Minuet: Multimodal Interaction with an Internet of Things

Authors:

Xiang 'Anthony' ChenAuthors Info & Claims

SUI '19: Symposium on Spatial User Interaction

Article No.: 2, Pages 1 - 10

https://doi.org/10.1145/3357251.3357581

Published: 19 October 2019 Publication History

All formats PDF

Abstract

A large number of Internet-of-Things (IoT) devices will soon populate our physical environments. Yet, IoT devices’ reliance on mobile applications and voice-only assistants as the primary interface limits their scalability and expressiveness. Building off of the classic ‘Put-That-There’ system, we contribute an exploration of the design space of voice + gesture interaction with spatially-distributed IoT devices. Our design space decomposes users’ IoT commands into two components—selection and interaction. We articulate how the permutations of voice and freehand gesture for these two components can complementarily afford interaction possibilities that go beyond current approaches. We instantiate this design space as a proof-of-concept sensing platform and demonstrate a series of novel IoT interaction scenarios, such as making ‘dumb’ objects smart, commanding robotic appliances, and resolving ambiguous pointing at cluttered devices.

Supplementary Material

MP4 File (a2-kang.mp4)

Download
527.06 MB

References

[1]

Amr Alanwar, Moustafa Alzantot, Bo-Jhang Ho, Paul Martin, and Mani Srivastava. 2017. SeleCon: Scalable IoT Device Selection and Control Using Hand Gestures. In Proceedings of the Second International Conference on Internet-of-Things Design and Implementation(IoTDI ’17). ACM, New York, NY, USA, 47–58. https://doi.org/10.1145/3054977.3054981

Digital Library

[2]

Amazon. 2019. Amazon Alexa. https://developer.amazon.com/alexa

[3]

Apple. 2019. HomePod. https://www.apple.com/homepod/

[4]

Ferran Argelaguet and Carlos Andujar. 2013. A survey of 3D object selection techniques for virtual environments. Computers & Graphics 37, 3 (2013), 121–136.

Digital Library

[5]

Till Ballendat, Nicolai Marquardt, and Saul Greenberg. 2010. Proxemic Interaction: Designing for a Proximity and Orientation-aware Environment. In ACM International Conference on Interactive Tabletops and Surfaces(ITS ’10). ACM, New York, NY, USA, 121–130. https://doi.org/10.1145/1936652.1936676

Digital Library

[6]

Michael Beigl. 1999. Point & Click - Interaction in Smart Environments. In Proceedings of the 1st International Symposium on Handheld and Ubiquitous Computing(HUC ’99). Springer-Verlag, London, UK, UK, 311–313. http://dl.acm.org/citation.cfm?id=647985.743710

Digital Library

[7]

Hugh Beyer and Karen Holtzblatt. 1997. Contextual design: defining customer-centered systems. Elsevier.

[8]

Richard A. Bolt. 1980. ‘Put-that-there’: Voice and Gesture at the Graphics Interface. In Proceedings of the 7th Annual Conference on Computer Graphics and Interactive Techniques(SIGGRAPH ’80). ACM, New York, NY, USA, 262–270. https://doi.org/10.1145/800250.807503

[9]

Kaifei Chen, Jonathan Fürst, John Kolb, Hyung-Sin Kim, Xin Jin, David E. Culler, and Randy H. Katz. 2018. SnapLink: Fast and Accurate Vision-Based Appliance Control in Large Commercial Buildings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 4, Article 129 (Jan. 2018), 27 pages. https://doi.org/10.1145/3161173

Digital Library

[10]

Xiang ‘Anthony’ Chen and Yang Li. 2017. Improv: An Input Framework for Improvising Cross-Device Interaction by Demonstration. ACM Trans. Comput.-Hum. Interact. 24, 2, Article 15 (April 2017), 21 pages. https://doi.org/10.1145/3057862

Digital Library

[11]

PR Cohen, M Darlymple, FCN Pereira, JW Sullivan, RA Gargan Jr, JL Schlossberg, and SW Tyler. [n.d.]. Synergic use of direct manipulation and natural language. In Proc. Conf. human Factors in Computing Systems (CHI’89). 227–233.

[12]

Philip R. Cohen, Michael Johnston, David McGee, Sharon Oviatt, Jay Pittman, Ira Smith, Liang Chen, and Josh Clow. 1997. QuickSet: Multimodal Interaction for Distributed Applications. In Proceedings of the Fifth ACM International Conference on Multimedia(MULTIMEDIA ’97). ACM, New York, NY, USA, 31–40. https://doi.org/10.1145/266180.266328

Digital Library

[13]

Adrian A. de Freitas, Michael Nebeling, Xiang ’Anthony’ Chen, Junrui Yang, Akshaye Shreenithi Kirupa Karthikeyan Ranithangam, and Anind K. Dey. 2016. Snap-To-It: A User-Inspired Platform for Opportunistic Device Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(CHI ’16). ACM, New York, NY, USA, 5909–5920. https://doi.org/10.1145/2858036.2858177

Digital Library

[14]

Google. 2019. Google Home - Smart Speaker & Home Assistant - Google Store. https://store.google.com/us/product/google_home

[15]

Google Cloud. 2019a. Cloud Natural Language. https://cloud.google.com/natural-language/

[16]

Google Cloud. 2019b. Cloud Speech-to-Text - Speech Recognition Cloud. https://cloud.google.com/speech-to-text/

[17]

Boris Gromov, Luca M Gambardella, and Gianni A Di Caro. 2016. Wearable multi-modal interface for human multi-robot interaction. In Safety, Security, and Rescue Robotics (SSRR), 2016 IEEE International Symposium on. IEEE, 240–245.

[18]

Boris Gromov, Luca M Gambardella, and Alessandro Giusti. 2018. Robot Identification and Localization with Pointing Gestures. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 3921–3928.

[19]

Valentin Heun, Shunichi Kasahara, and Pattie Maes. 2013. Smarter objects: using AR technology to program physical objects and their interactions. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. ACM, 961–966.

[20]

Ke Huo, Yuanzhi Cao, Sang Ho Yoon, Zhuangying Xu, Guiming Chen, and Karthik Ramani. 2018. Scenariot: Spatially Mapping Smart Things Within Augmented Reality Scenes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, Article 219, 13 pages. https://doi.org/10.1145/3173574.3173793

Digital Library

[21]

Antonio Ramón Jiménez and Fernando Seco. 2016. Comparing Decawave and Bespoon UWB location systems: Indoor/outdoor performance analysis. In IPIN. 1–8.

[22]

Wendy Ju, Brian A. Lee, and Scott R. Klemmer. 2008. Range: Exploring Implicit Interaction Through Electronic Whiteboard Design. In Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work(CSCW ’08). ACM, New York, NY, USA, 17–26. https://doi.org/10.1145/1460563.1460569

Digital Library

[23]

Naohiko Kohtake, Jun Rekimoto, and Yuichiro Anzai. 2001. InfoPoint: A Device That Provides a Uniform User Interface to Allow Appliances to Work Together over a Network. Personal Ubiquitous Comput. 5, 4 (Jan. 2001), 264–274. https://doi.org/10.1007/s007790170005

Digital Library

[24]

Gierad Laput, Robert Xiao, and Chris Harrison. 2016. ViBand: High-Fidelity Bio-Acoustic Sensing Using Commodity Smartwatch Accelerometers. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology(UIST ’16). ACM, New York, NY, USA, 321–333. https://doi.org/10.1145/2984511.2984582

Digital Library

[25]

Gierad P. Laput, Mira Dontcheva, Gregg Wilensky, Walter Chang, Aseem Agarwala, Jason Linder, and Eytan Adar. 2013. PixelTone: A Multimodal Interface for Image Editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’13). ACM, New York, NY, USA, 2185–2194. https://doi.org/10.1145/2470654.2481301

Digital Library

[26]

David Ledo, Saul Greenberg, Nicolai Marquardt, and Sebastian Boring. 2015. Proxemic-Aware Controls: Designing Remote Controls for Ubiquitous Computing Ecologies. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services(MobileHCI ’15). ACM, New York, NY, USA, 187–198. https://doi.org/10.1145/2785830.2785871

Digital Library

[27]

Andy Liaw, Matthew Wiener, 2002. Classification and regression by randomForest. R news 2, 3 (2002), 18–22.

[28]

Nicolai Marquardt, Till Ballendat, Sebastian Boring, Saul Greenberg, and Ken Hinckley. 2012. Gradual Engagement: Facilitating Information Exchange Between Digital Devices As a Function of Proximity. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces(ITS ’12). ACM, New York, NY, USA, 31–40. https://doi.org/10.1145/2396636.2396642

Digital Library

[29]

B. D. Mayton, N. Zhao, M. Aldrich, N. Gillian, and J. A. Paradiso. 2013. WristQue: A personal sensor wristband. In 2013 IEEE International Conference on Body Sensor Networks. 1–6. https://doi.org/10.1109/BSN.2013.6575483

[30]

Laurence Nigay and Joëlle Coutaz. 1993. A design space for multimodal systems: concurrent processing and data fusion. In Proceedings of the INTERACT’93 and CHI’93 conference on Human factors in computing systems. ACM, 172–178.

Digital Library

[31]

Sharon Oviatt. 1999a. Mutual Disambiguation of Recognition Errors in a Multimodel Architecture. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems(CHI ’99). ACM, New York, NY, USA, 576–583. https://doi.org/10.1145/302979.303163

Digital Library

[32]

Sharon Oviatt. 1999b. Ten Myths of Multimodal Interaction. Commun. ACM 42, 11 (Nov. 1999), 74–81. https://doi.org/10.1145/319382.319398

Digital Library

[33]

Sharon Oviatt, Phil Cohen, Lizhong Wu, Lisbeth Duncan, Bernhard Suhm, Josh Bers, Thomas Holzman, Terry Winograd, James Landay, Jim Larson, 2000. Designing the user interface for multimodal speech and pen-based gesture applications: state-of-the-art systems and future research directions. Human-computer interaction 15, 4 (2000), 263–322.

[34]

Sharon Oviatt, Antonella DeAngeli, and Karen Kuhn. 1997. Integration and Synchronization of Input Modes During Multimodal Human-computer Interaction. In Referring Phenomena in a Multimedia Context and Their Computational Treatment(ReferringPhenomena ’97). Association for Computational Linguistics, Stroudsburg, PA, USA, 1–13. http://dl.acm.org/citation.cfm?id=1621585.1621587

[35]

Shwetak N Patel, Jun Rekimoto, and Gregory D Abowd. 2006. icam: Precise at-a-distance interaction in the physical environment. In International Conference on Pervasive Computing. Springer, 272–287.

Digital Library

[36]

Trevor Pering, Yaw Anokwa, and Roy Want. 2007. Gesture Connect: Facilitating Tangible Interaction with a Flick of the Wrist. In Proceedings of the 1st International Conference on Tangible and Embedded Interaction(TEI ’07). ACM, New York, NY, USA, 259–262. https://doi.org/10.1145/1226969.1227022

Digital Library

[37]

Jun Rekimoto and Katashi Nagao. 1995. The World Through the Computer: Computer Augmented Interaction with Real World Environments. In Proceedings of the 8th Annual ACM Symposium on User Interface and Software Technology(UIST ’95). ACM, New York, NY, USA, 29–36. https://doi.org/10.1145/215585.215639

Digital Library

[38]

Bill Schilit, Norman Adams, and Roy Want. 1994. Context-aware computing applications. In Mobile Computing Systems and Applications, 1994. Proceedings., Workshop on. IEEE, 85–90.

Digital Library

[39]

EDWARD TSE, SAUL GREENBERG, CHIA SHEN, and CLIFTON FORLINES. 2007. Multimodal Multiplayer Tabletop Gaming. Comput. Entertain. 5, 2, Article 12 (April 2007). https://doi.org/10.1145/1279540.1279552

[40]

Eduardo Velloso, Markus Wirth, Christian Weichel, Augusto Esteves, and Hans Gellersen. 2016. AmbiGaze: Direct Control of Ambient Devices by Gaze. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems(DIS ’16). ACM, New York, NY, USA, 812–817. https://doi.org/10.1145/2901790.2901867

Digital Library

[41]

Robert Xiao, Gierad Laput, Yang Zhang, and Chris Harrison. 2017. Deus EM Machina: On-Touch Contextual Functionality for Smart IoT Appliances. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems(CHI ’17). ACM, New York, NY, USA, 4000–4008. https://doi.org/10.1145/3025453.3025828

Digital Library

[42]

Ben Zhang, Yu-Hsiang Chen, Claire Tuna, Achal Dave, Yang Li, Edward Lee, and Björn Hartmann. 2014. HOBS: Head Orientation-based Selection in Physical Spaces. In Proceedings of the 2Nd ACM Symposium on Spatial User Interaction(SUI ’14). ACM, New York, NY, USA, 17–25. https://doi.org/10.1145/2659766.2659773

Digital Library

[43]

Yang Zhang, Chouchang (Jack) Yang, Scott E. Hudson, Chris Harrison, and Alanson Sample. 2018. Wall++: Room-Scale Interactive and Context-Aware Sensing. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems(CHI ’18). ACM, New York, NY, USA, Article 273, 15 pages. https://doi.org/10.1145/3173574.3173847

Digital Library

Cited By

Kim MGlenn AVeluri BLee YGebre EBagaria APatel SGollakota S(2024)IRIS: Wireless ring for vision-based smart home interactionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676327(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676327
Wu LLafreniere BGrossman TWhite TSantosa S(2024)Body Language for VUIs: Exploring Gestures to Enhance Interactions with Voice User InterfacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660691(133-150)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660691
Rodriguez RSullivan BBarrera Machuca MBatmaz ATornatzky COrtega F(2024)An Artists' Perspectives on Natural Interactions for Virtual Reality 3D SketchingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642758(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642758
Show More Cited By

Recommendations

Multimodal human discourse: gesture and speech

Gesture and speech combine to form a rich basis for human conversational interaction. To exploit these modalities in HCI, we need to understand the interplay between them and the way in which they support communication. We propose a framework for the ...
Multimodal multiparty social interaction with the furhat head
ICMI '12: Proceedings of the 14th ACM international conference on Multimodal interaction

We will show in this demonstrator an advanced multimodal and multiparty spoken conversational system using Furhat, a robot head based on projected facial animation. Furhat is a human-like interface that utilizes facial animation for physical robot heads ...
A comprehensive survey of authentication methods in Internet-of-Things and its conjunctions
Abstract
Internet of Thing (IoT) is one of the most influential technologies in the present time. People, processes, and things are connected with the Internet through IoT. With the increasing demands of user applications, the number of ...
Graphical abstract

Display Omitted
Highlights
- We show the evolution of the authentication solutions for IoTs.
- We consider the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

SUI '19: Symposium on Spatial User Interaction

October 2019

164 pages

ISBN:9781450369756

DOI:10.1145/3357251

Editors:
Christoph W. Borst
University of Louisiana at Lafayette, USA
,
Arun K. Kulshreshth
University of Louisiana at Lafayette, USA
,
Gerd Bruder
University of Central Florida, USA
,
Stefania Serafin
Aalborg University, Denmark
,
Christian Sandor
City University of Hong Kong
,
Kyle Johnsen
University of Georgia, USA
,
Jinwei Ye
Louisiana State University, USA
,
Daniel Roth
Universität Würzburg, Germany
,
Sungchul Jung
HIT Lab NZ

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SUI '19

SUI '19: Symposium on Spatial User Interaction

October 19 - 20, 2019

LA, New Orleans, USA

Acceptance Rates

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

29
Total Citations
View Citations
2,371
Total Downloads

Downloads (Last 12 months)498
Downloads (Last 6 weeks)58

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kim MGlenn AVeluri BLee YGebre EBagaria APatel SGollakota S(2024)IRIS: Wireless ring for vision-based smart home interactionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676327(1-16)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3654777.3676327
Wu LLafreniere BGrossman TWhite TSantosa S(2024)Body Language for VUIs: Exploring Gestures to Enhance Interactions with Voice User InterfacesProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3660691(133-150)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3660691
Rodriguez RSullivan BBarrera Machuca MBatmaz ATornatzky COrtega F(2024)An Artists' Perspectives on Natural Interactions for Virtual Reality 3D SketchingProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642758(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642758
Yang JShi YZhang YLi KRosli DJain AZhang SLi TLanday JLam M(2024)ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language ModelsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642517(1-23)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642517
I Jeena Jacob (2023)Harnessing Power of Multimodal Interaction, their Challenges and Future Prospect – A ReviewRecent Research Reviews Journal10.36548/rrrj.2023.2.0172:2(457-479)Online publication date: Dec-2023
https://doi.org/10.36548/rrrj.2023.2.017
Gallo SPaterno FMalizia A(2023)Conversational Interfaces in IoT Ecosystems: Where We Are, What Is Still MissingProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627775(279-293)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627775
Bernardos ABergesio LBesada JCasar J(2023)Evaluation of a Multimodal Interaction System for Big DisplaysProceedings of the 15th Biannual Conference of the Italian SIGCHI Chapter10.1145/3605390.3605411(1-9)Online publication date: 20-Sep-2023
https://dl.acm.org/doi/10.1145/3605390.3605411
Liu JLi WGu TGao RChen BZhang FWu DZhang D(2023)Towards a Dynamic Fresnel Zone Model to WiFi-based Human Activity RecognitionProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962707:2(1-24)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596270
Liu XShi YYu CGao CYang TLiang CShi Y(2023)Understanding In-Situ Programming for Smart Home AutomationProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962547:2(1-31)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596254
Islam MBillah S(2023)SpaceX MagProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962537:2(1-36)Online publication date: 12-Jun-2023
https://dl.acm.org/doi/10.1145/3596253
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents