SealMates: Improving Communication in Video Conferencing using a Collective Behavior-Driven Avatar
Authors: 
Mark Armstrong, 
Chi-Lan Yang, 
Kinga Skiers, Mengzhen Lim, Tamil Selvan Gunasekaran, Ziyue Wang, Takuji Narumi, Kouta Minamizawa, Yun Suen PaiAuthors Info & Claims
Article No.: 118, Pages 1 - 23
Abstract
The limited nonverbal cues and spatially distributed nature of remote communication make it challenging for unacquainted members to be expressive during social interactions over video conferencing. Though it enables seeing others' facial expressions, the visual feedback can instead lead to unexpected self-focus, resulting in users missing cues for others to engage in the conversation equally. To support expressive communication and equal participation among unacquainted counterparts, we propose SealMates, a behavior-driven avatar in which the avatar infers the engagement level of the group based on collective gaze and speech patterns and then moves across interlocutors' windows in the video conferencing. By conducting a controlled experiment with 15 groups of triads, we found the avatar's movement encouraged people to experience more self-disclosure and made them perceive everyone was equally engaged in the conversation than when there was no behavior-driven avatar. We discuss how a behavior-driven avatar influences distributed members' perceptions and the implications of avatar-mediated communication for future platforms.
Supplemental Material
MP4 File
Supplemental video
- Download
- 131.80 MB
References
[1]
Ahsan Abdullah, Jan Kolkmeier, Vivian Lo, and Michael Neff. 2021. Videoconference and Embodied VR: Communication Patterns across Task and Medium. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1--29.
[2]
Arthur Aron, Edward Melinat, Elaine N. Aron, Roberta Vallone, and Renée J. Bator. 1997. The Experimental Generation of Interpersonal Closeness: A Procedure and Some Preliminary Findings. Personality and Social Psychology Bulletin, Vol. 23 (1997), 363 -- 377.
[3]
Karol'ina Balogová. 2021. Looking At Yourself on Zoom. Ph.,D. Dissertation. Master's thesis. University College London, London, UK. https://uclic. ucl ?.
[4]
Guillermo Bernal and Pattie Maes. 2017. Emotional beasts: visually expressing emotions through avatars in VR. In Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems. 2395--2402.
[5]
Jia Borror, Sarah Ransdell, Jenna Binaco, and April Feeser. 2021. The Impact of Transitioning to Online Learning and Virtual Conferences on Students and Educators During the Coronavirus Pandemic. Distance Learning, Vol. 18, 1 (2021), 13--24.
[6]
Jose Eurico de Vasconcelos Filho, Kori M Inkpen, and Mary Czerwinski. 2009. Image, appearance and vanity in the use of media spaces and video conference systems. In Proceedings of the ACM 2009 international conference on Supporting group work. 253--262.
[7]
Ziedune Degutyte and Arlene Astell. 2021. The Role of Eye Gaze in Regulating Turn Taking in Conversations: A Systematized Review of Methods and Findings. Frontiers in Psychology, Vol. 12 (2021), 616471. https://doi.org/10.3389/fpsyg.2021.616471
[8]
Wen Duan, Naomi Yamashita, Yoshinari Shirai, and Susan R Fussell. 2021. Bridging Fluency Disparity between Native and Nonnative Speakers in Multilingual Multiparty Collaboration Using a Clarification Agent. Proceedings of the ACM on Human-Computer Interaction, Vol. 5, CSCW2 (2021), 1--31.
[9]
Cathy Mengying Fang, GR Marvez, Neska ElHaouij, and Rosalind Picard. 2022. Cardiac Arrest: Evaluating the Role of Biosignals in Gameplay Strategies and Players' Physiological Synchrony in Social Deception Games. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1--7.
[10]
Heather A Faucett, Matthew L Lee, and Scott Carter. 2017. I should listen more: real-time sensing and feedback of non-verbal communication in video telehealth. Proceedings of the ACM on Human-Computer Interaction, Vol. 1, CSCW (2017), 1--19.
[11]
Ylva Ferstl, Elena Kokkinara, and Rachel McDonnell. 2016. Do I trust you, abstract creature? A study on personality perception of abstract virtual faces. In Proceedings of the ACM Symposium on Applied Perception. 39--43.
[12]
Joey George, Akmal Mirsadikov, Misty Nabors, and Kent Marett. 2022. What do Users Actually Look at During ?Zoom'Meetings? Discovery Research on Attention, Gender and Distraction Effects. In Proceedings of the 55th Hawaii International Conference on System Sciences.
[13]
Upamanyu Ghose, Arvind S., W. Boyce, Hong Xu, and Eng Chng. 2020. PyTrack: An End-to-End Analysis Toolkit for Eye Tracking. Behavior Research Methods, Vol. 52 (March 2020). https://doi.org/10.3758/s13428-020-01392--6
[14]
Erving Goffman. 2021. The presentation of self in everyday life. Anchor.
[15]
William W Hahn. 1973. Attention and heart rate: a critical appraisal of the hypothesis of Lacey and Lacey. (1973).
[16]
Modar Hassan, Maxwell Kennard, Seiji Yoshitake, Karlos Ishac, Shion Takahashi, SunKyoung Kim, Takashi Matsui, and Masakazu Hirokawa. 2023. Augmenting the Sense of Social Presence in Online Video Games Through the Sharing of Biosignals. Available at SSRN 4409781 (2023).
[17]
Jonathan Hennessy et al. 2022. Reflections on an unexpected presentation of turn-taking difficulties in an English discussion class. Journal of Multilingual Pedagogy and Practice, Vol. 2 (2022), 94--100.
[18]
Kristiina Jokinen, Hirohisa Furukawa, Masafumi Nishida, and Seiichi Yamamoto. 2013. Gaze and turn-taking behavior in casual conversational interactions. ACM Transactions on Interactive Intelligent Systems (TiiS), Vol. 3, 2 (2013), 1--30.
[19]
Kristiina Jokinen, Masafumi Nishida, and Seiichi Yamamoto. 2010. On eye-gaze and turn-taking. In Proceedings of the 2010 workshop on Eye gaze in intelligent human machine interaction (EGIHMI '10). Association for Computing Machinery, New York, NY, USA, 118--123. https://doi.org/10.1145/2002333.2002352
[20]
Soomin Kim, Jinsu Eun, Changhoon Oh, Bongwon Suh, and Joonhwan Lee. 2020. Bot in the Bunch: Facilitating Group Chat Discussion by Improving Efficiency and Participation with a Chatbot. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (2020).
[21]
Mark R Leary. 2016. Introduction to behavioral research methods. Pearson education company.
[22]
Sueyoon Lee, Abdallah El Ali, Maarten Wijntjes, and Pablo Cesar. 2022. Understanding and Designing Avatar Biosignal Visualizations for Social Virtual Reality Entertainment. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1--15.
[23]
Fannie Liu, Mario Esparza, Maria Pavlovskaia, Geoff Kaufman, Laura Dabbish, and Andrés Monroy-Hernández. 2019. Animo: Sharing biosignals on a smartwatch for lightweight social connection. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Vol. 3, 1 (2019), 1--19.
[24]
Fannie Liu, Chunjong Park, Yu Jiang Tham, Tsung-Yu Tsai, Laura Dabbish, Geoff Kaufman, and Andrés Monroy-Hernández. 2021. Significant otter: Understanding the role of biosignals in communication. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--15.
[25]
Ning-Han Liu, Cheng-Yu Chiang, and Hsuan-Chin Chu. 2013. Recognizing the degree of human attention using EEG signals from mobile sensors. sensors, Vol. 13, 8 (2013), 10273--10286.
[26]
Lori Lorigo, Maya Haridasan, Hrönn Brynjarsdóttir, Ling Xia, Thorsten Joachims, Geri Gay, Laura Granka, Fabio Pellacini, and Bing Pan. 2008. Eye tracking and online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science and Technology, Vol. 59, 7 (2008), 1041--1052.
[27]
Matthew K Miller, Regan L Mandryk, Max V Birk, Ansgar E Depping, and Tushita Patel. 2017. Through the looking glass: The effects of feedback on self-awareness and conversational behaviour during video chat. In Proceedings of the 2017 CHI conference on human factors in computing systems. 5271--5283.
[28]
Saki Mizuno, Nobukatsu Hojo, Satoshi Kobashikawa, and Ryo Masumura. 2023. Next-Speaker Prediction Based on Non-Verbal Information in Multi-Party Video Conversation. In ICASSP 2023--2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 1--5.
[29]
Dooley Murphy. 2017. Building a hybrid virtual agent for testing user empathy and arousal in response to avatar (micro-) expressions. In Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology. 1--2.
[30]
Annabel Ngien and Bernie Hogan. 2022. The relationship between Zoom use with the camera on and Zoom fatigue: considering self-monitoring and social interaction anxiety. Information, Communication & Society (2022), 1--19.
[31]
Kristine L Nowak and Jesse Fox. 2018. Avatars and computer-mediated communication: A review of the definitions, uses, and effects of digital representations. Review of Communication Research, Vol. 6 (2018), 30--53.
[32]
Brid O'Conaill, Steve Whittaker, and Sylvia Wilbur. 1993. Conversations over video conferences: An evaluation of the spoken aspects of video-mediated communication. Human-computer interaction, Vol. 8, 4 (1993), 389--428.
[33]
Soo Youn Oh, Jeremy Bailenson, Nicole Kr"amer, and Benjamin Li. 2016. Let the avatar brighten your smile: Effects of enhancing facial expressions in virtual environments. PloS one, Vol. 11, 9 (2016), e0161794.
[34]
Gabrielle Oliveira and Jon M Wargo. 2021. Collison, Connection, and Conflict? Writing (Righting) Community Amid COVID-19. Journal of Adolescent & Adult Literacy, Vol. 64, 6 (2021), 708.
[35]
Inyoung Park, Young June Sah, Suhyun Lee, and Daeho Lee. 2023. Avatar-Mediated Communication in Video Conferencing: Effect of Self-Affirmation on Debating Participation Focusing on Moderation Effect of Avatar. International Journal of Human-Computer Interaction, Vol. 39, 3 (2023), 464--475.
[36]
Per Persson. 2003. Exms: an animated and avatar-based messaging system for expressive peer communication. In GROUP '03.
[37]
MoShi Qin. 2023. GAZE IN REMOTE COLLABORATION: WHERE DO PEOPLE LOOK DURING VIDEO CONFERENCE IN THE CONTEXT OF STUDENT COLLABORATION. (2023).
[38]
Vikas Rajendra and Omid Dehzangi. 2017. Detection of distraction under naturalistic driving using Galvanic Skin Responses. In 2017 IEEE 14th International Conference on Wearable and Implantable Body Sensor Networks (BSN). IEEE, 157--160.
[39]
Mariachiara Rapuano, Antonella Ferrara, Filomena Leonela Sbordone, Francesco Ruotolo, Gennaro Ruggiero, and Tina Iachini. 2020. The appearance of the avatar can enhance the sense of co-presence during virtual interactions with users. In PSYCHOBIT.
[40]
Priyanka Sahu, Mohit Dua, and Ankit Kumar. 2018. Challenges and issues in adopting speech recognition. Speech and language processing for human-machine communications (2018), 209--215.
[41]
Samiha Samrose, Ru Zhao, Jeffery White, Vivian Li, Luis Nova, Yichen Lu, Mohammad Rafayet Ali, and Mohammed Ehsan Hoque. 2018. Coco: Collaboration coach for understanding team dynamics during video conferencing. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies, Vol. 1, 4 (2018), 1--24.
[42]
Masataka Seo, Takahiro Yamamoto, Yen-Wei Chen, and Toshihiro Kitajima. 2019. Automatic Gaze Correction based on Deep Learning and Image Warping. In 2019 IEEE International Conference on Consumer Electronics (ICCE). 1--4. https://doi.org/10.1109/ICCE.2019.8661836
[43]
Ameneh Shamekhi, Qingzi Vera Liao, Dakuo Wang, Rachel K. E. Bellamy, and Thomas Erickson. 2018a. Face Value? Exploring the Effects of Embodiment for a Group Facilitation Agent. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (2018).
[44]
Ameneh Shamekhi, Qingzi Vera Liao, Dakuo Wang, Rachel K. E. Bellamy, and Thomas Erickson. 2018b. Face Value? Exploring the Effects of Embodiment for a Group Facilitation Agent. Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (2018).
[45]
Dilruba Showkat and Cindy Grimm. 2018. Identifying gender differences in information processing style, self-efficacy, and tinkering for robot tele-operation. In 2018 15th international conference on ubiquitous robots (UR). IEEE, 443--448.
[46]
Susan Sprecher. 2021. Closeness and other affiliative outcomes generated from the Fast Friends procedure: A comparison with a small-talk task and unstructured self-disclosure and the moderating role of mode of communication. Journal of Social and Personal Relationships, Vol. 38 (2021), 1452 -- 1471.
[47]
P. O. N. Staff. 2022. Body Language in Negotiation Can Build Rapport-Without Saying a Word. https://www.pon.harvard.edu/daily/negotiation-skills-daily/build-rapport-without-saying-a-word-nb/ Section: Negotiation Skills.
[48]
Simone Stumpf, Anicia Peters, Shaowen Bardzell, Margaret Burnett, Daniela Busse, Jessica Cauchard, Elizabeth Churchill, et al. 2020. Gender-inclusive HCI research and design: A conceptual review. Foundations and Trends® in Human-Computer Interaction, Vol. 13, 1 (2020), 1--69.
[49]
Keita Suzuki, Masanori Yokoyama, Shigeo Yoshida, Takayoshi Mochizuki, Tomohiro Yamada, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2017. FaceShare: Mirroring with Pseudo-Smile Enriches Video Chat Communications. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 5313--5317. https://doi.org/10.1145/3025453.3025574
[50]
Jordan Tewell, Jon Bird, and George R Buchanan. 2017. The heat is on: A temperature display for conveying affective feedback. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 1756--1767.
[51]
Niels van Berkel, Jorge Goncalves, Simo Hosio, Zhanna Sarsenbayeva, Eduardo Velloso, and Vassilis Kostakos. 2020. Overcoming compliance bias in self-report studies: A cross-study analysis. International Journal of Human-Computer Studies, Vol. 134 (2020), 1--12.
[52]
Hua Wang, Helmut Prendinger, and Takeo Igarashi. 2004. Communicating emotions in online chat using physiological sensors and animated text. In CHI'04 extended abstracts on Human factors in computing systems. 1171--1174.
[53]
Alex Wuqi Zhang, Ting-Han Lin, Xuan Zhao, and Sarah Sebo. 2023. Ice-Breaking Technology: Robots and Computers Can Foster Meaningful Connections between Strangers through In-Person Conversations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1--14.
[54]
He Zhenyi, Keru Wang, Brandon Yushan Feng, Du Ruofei, Ken Perlin, Zhenyi He, Keru Wang, Brandon Yushan Feng, Ruofei Du, and Ken Perlin. 2021. GazeChat: Enhancing Virtual Conferences with Gaze-aware 3D Photos. The 34th Annual ACM Symposium on User Interface Software and Technology (2021). io
Index Terms
- SealMates: Improving Communication in Video Conferencing using a Collective Behavior-Driven Avatar
Recommendations
Development of an Embodied Avatar System using Avatar-Shadow's Color Expressions with an Interaction-activated Communication Model
HAI '16: Proceedings of the Fourth International Conference on Human Agent InteractionIn reality, shadows are usually natural and unintentional. In virtual reality, however, they play an important role in three-dimensional effects and the perceived reality of the virtual space. An avatar's shadow can have interactive effects with the ...
Your Avatar Seems Hesitant to Share About Yourself: How People Perceive Others' Avatars in the Transparent System
CHI '24: Proceedings of the 2024 CHI Conference on Human Factors in Computing SystemsIn avatar-mediated communications, users often cannot identify how others’ avatars are created, which is one of the important information they need to evaluate others. Thus, we tested a social virtual world that is transparent about others’ avatar-...
Video Avatar: Embedded Video for Collaborative Virtual Environment
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2This paper proposes the video avatar, the avatar with the embedded video of the human body in the collaborative virtual environment. The video avatar enables the visual communication between the avatars. The video avatar has three features for enhancing ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 26 April 2024
Published in PACMHCI Volume 8, Issue CSCW1
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
- JST?????????????????????????????????????????????????
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 160Total Downloads
- Downloads (Last 12 months)160
- Downloads (Last 6 weeks)33
Reflects downloads up to 11 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in