Harnessing Gestures to Externalize Discourse Ideas for Common Ground: Design Implications from a Frame of Reference Analysis
Pages 1 - 8
Abstract
When we share ideas through conversation, we convey a specific mental image to another person. However, mental images don't always match, and the effort it takes to stop and achieve common ground would interrupt conversational flow. We are currently investigating the design of a system that translates a speaker's gestures and speech into a visualization of their idea. The challenge of designing such a system is complex, as conversation is composed of many intricate factors. We focus the results in this paper on one factor in particular: speaker of reference. We ran a study with 26 participants, with a prototype meant to handle gestured descriptions of object size and noted the effects of speaker frame of reference. From our analysis of reference frames used during size descriptions, we draw implications for how our proposed system may detect and translate frame of reference to produce visualizations of the user's mental image.
References
[1]
Tony Bergstrom and Karrie Karahalios. 2007. Conversation Clock: Visualizing audio patterns in co-located groups. In 2007 40th Annual Hawaii International Conference on System Sciences (HICSS'07). IEEE, 78--78.
[2]
Sarah Anne Brown, Sharon Lynn Chu, Francis Quek, Pomaikai Canaday, Qing Li, Trystan Loustau, Sindy Wu, and Lina Zhang. 2019. Towards a Gesture-Based Story Authoring System: Design Implications from Feature Analysis of Iconic Gestures During Storytelling. In International Conference on Interactive Digital Storytelling. Springer, 364--373.
[3]
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. In arXiv preprint arXiv:1812.08008.
[4]
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2017. Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields. In CVPR.
[5]
Eve Danziger. 2010. Deixis, gesture, and cognition in spatial frame of reference typology. Studies in Language. International Journal sponsored by the Foundation "Foundations of Language" 34, 1 (2010), 167--185.
[6]
Judith Donath, Karrie Karahalios, and Fernanda Viegas. 1999. Visualizing conversation. Journal of computer-mediated communication 4, 4 (1999), JCMC442.
[7]
Christian Holz and Andrew Wilson. 2011. Data miming: inferring spatial object descriptions from human gesture. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 811--820.
[8]
Felix Kistler and Elisabeth André. 2013. User-defined body gestures for an interactive storytelling scenario. In IFIP Conference on Human-Computer Interaction. Springer, 264--281.
[9]
Stephen C Levinson. 1996. Frames of reference and Molyneux's question: Crosslinguistic evidence. Language and space 109 (1996), 169.
[10]
Fridanna Maricchiolo, Augusto Gnisci, and Marino Bonaiuto. 2012. Coding hand gestures: A reliable taxonomy and a multi-media support. In Cognitive behavioural systems. Springer, 405--416.
[11]
José Emmanuel Ramirez Marquez. 2016. Social media conversation monitoring: Visualize information contents of twitter messages using conversational metrics. (2016).
[12]
David McNeill. 1992. Hand and mind: What gestures reveal about thought. University of Chicago press.
[13]
Sushmita Mitra and Tinku Acharya. 2007. Gesture recognition: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 37, 3 (2007), 311--324.
[14]
Mohammad Obaid, Markus Häring, Felix Kistler, René Bühling, and Elisabeth André. 2012. User-defined body gestures for navigational control of a humanoid robot. In International Conference on Social Robotics. Springer, 367--377.
[15]
Francis Quek, David McNeill, Robert Bryll, Susan Duncan, Xin-Feng Ma, Cemil Kirbas, Karl E. McCullough, and Rashid Ansari. 2002. Multimodal Human Discourse: Gesture and Speech. ACM Trans. Comput.-Hum. Interact. 9, 3 (Sept. 2002), 171--193.
[16]
Siddharth S Rautaray and Anupam Agrawal. 2015. Vision based hand gesture recognition for human computer interaction: A survey. Artificial intelligence review 43, 1 (2015), 1--54.
[17]
Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh. 2017. Hand Keypoint Detection in Single Images using Multiview Bootstrapping. In CVPR.
[18]
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. 2016. Convolutional pose machines. In CVPR.
[19]
Andrew D Wilson and Aaron F Bobick. 1999. Parametric hidden markov models for gesture recognition. IEEE transactions on pattern analysis and machine intelligence 21, 9 (1999), 884--900.
[20]
Jacob O Wobbrock, Meredith Ringel Morris, and Andrew D Wilson. 2009. User-defined gestures for surface computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 1083--1092.
Index Terms
- Harnessing Gestures to Externalize Discourse Ideas for Common Ground: Design Implications from a Frame of Reference Analysis
Recommendations
Externalizing Mental Images by Harnessing Size-Describing Gestures: Design Implications for a Visualization System
AVI '20: Proceedings of the 2020 International Conference on Advanced Visual InterfacesPeople use a significant amount of gestures when engaging in creative brainstorming. This is especially typical for creative workers who frequently convey ideas, designs, and stories to team members. These gestures produced during natural conversation ...
Recognizing gaze aversion gestures in embodied conversational discourse
ICMI '06: Proceedings of the 8th international conference on Multimodal interfacesEye gaze offers several key cues regarding conversational discourse during face-to-face interaction between people. While a large body of research results exist to document the use of gaze in human-to-human interaction, and in animating realistic ...
Local Alignment of Frame of Reference Assignment in English and Swedish Dialogue
Spatial Cognition XIIAbstractIn this paper we examine how people assign, interpret, negotiate and repair the frame of reference (FoR) in online text-based dialogues discussing spatial scenes in English and Swedish. We describe our corpus and data collection which involves a ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2020
4474 pages
ISBN:9781450368193
DOI:10.1145/3334480
Copyright © 2020 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 25 April 2020
Check for updates
Author Tags
Qualifiers
- Abstract
Funding Sources
Conference
CHI '20
Sponsor:
Acceptance Rates
Overall Acceptance Rate 6,164 of 23,696 submissions, 26%
Upcoming Conference
CHI 2025
- Sponsor:
- sigchi
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 111Total Downloads
- Downloads (Last 12 months)6
- Downloads (Last 6 weeks)0
Reflects downloads up to 21 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format