More Web Proxy on the site http://driver.im/

research-article

Field Trial Analysis of Socially Aware Robot Assistant

Authors:

Florian Pecune,

Yoichi Matsuyama,

Justine CassellAuthors Info & Claims

AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

Pages 1241 - 1249

Published: 09 July 2018 Publication History

Abstract

The Socially-Aware Robot Assistant (SARA) is an embodied conversational agent that works toward using detection of visual, vocal and verbal cues as an input to estimate the strength of its relationship (namely the level of rapport) with a user. SARA then answers to the user through similar visual, vocal and verbal behaviors with the goal of building and maintaining rapport with that user as we hypothesize that this will improve task performance and user satisfaction over time. In this paper, we report results of a field trial with a semi-automatic SARA system that took place in a large high-profile conference. Participants interacted with SARA during the whole conference, receiving recommendations about sessions to attend and/or people to meet. We analyzed these interactions to shed light on the dynamics of the rapport level between SARA and the conference attendees, and investigate how SARA's task performance would influence the evolution of rapport over time. Although we did not find evidence supporting our claim that the recommendations' outcomes would influence rapport dynamics, our findings emphasize the importance of interactional features plays in both rapport and SARA's task performance. We thus propose design guidelines for the next generation of socially aware personal assistants.

References

[1]

Nalini Ambady and Robert Rosenthal . 1992. Thin slices of expressive behavior as predictors of interpersonal consequences: A meta-analysis. (1992).

[2]

Amos Azaria and Jason Hong . 2016. Recommender Systems with Personality. In RecSys. 207--210.

Digital Library

[3]

Tadas Baltruvsaitis, Peter Robinson, and Louis-Philippe Morency . 2016. OpenFace: an open source facial behavior analysis toolkit Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV).

[4]

Timo Baumann, Casey Kennington, Julian Hough, and David Schlangen . 2017. Recognising conversational speech: What an incremental asr should do for a dialogue system and how to get there. Dialogues with Social Robots. Springer, 421--432.

[5]

Pauline M Berry, Melinda Gervasio, Bart Peintner, and Neil Yorke-Smith . 2011. PTIME: Personalized assistance for calendaring. ACM Transactions on Intelligent Systems and Technology (TIST), Vol. 2, 4 (2011), 40.

Digital Library

[6]

Timothy W Bickmore, Laura M Pfeifer Vardoulakis, and Daniel Schulman . 2013. Tinker: a relational agent museum guide. Autonomous agents and multi-agent systems Vol. 27, 2 (2013), 254--276.

[7]

Woody Bledsoe . 1986. I had a dream: AAAI presidential address. AI Magazine, Vol. 7, 1 (1986), 57.

Digital Library

[8]

Penelope Brown and Stephen C Levinson . 1987. Politeness: Some universals in language usage. Vol. Vol. 4. Cambridge university press.

[9]

Angelo Cafaro, Nadine Glas, and Catherine Pelachaud . 2016. The effects of interrupting behavior on interpersonal attitude and engagement in dyadic interactions. In Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 911--920.

Digital Library

[10]

Justine Cassell . 2000. Embodied conversational agents. The MIT Press.

[11]

Justine Cassell and Timothy Bickmore . 2003. Negotiated collusion: Modeling social language and its relationship effects in intelligent agents. User Modeling and User-Adapted Interaction Vol. 13, 1--2 (2003), 89--132.

Digital Library

[12]

Justine Cassell, Alastair J Gill, and Paul A Tepper . 2007. Coordination in conversation and rapport. In Proceedings of the workshop on Embodied Language Processing. Association for Computational Linguistics, 41--50.

Digital Library

[13]

Justine Cassell, Hannes Högni Vilhjálmsson, and Timothy Bickmore . 2004. BEAT: the behavior expression animation toolkit. Life-Like Characters. Springer, 163--185.

[14]

David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et almbox. . 2014. SimSensei Kiosk: A virtual human interviewer for healthcare decision support Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems, 1061--1068.

Digital Library

[15]

Florian Eyben, Felix Weninger, Florian Gross, and Björn Schuller . 2013. Recent developments in opensmile, the munich open-source multimedia feature extractor Proceedings of the 21st ACM international conference on Multimedia. ACM, 835--838.

Digital Library

[16]

Michael Freed, Jaime G Carbonell, Geoffrey J Gordon, Jordan Hayes, Brad A Myers, Daniel P Siewiorek, Stephen F Smith, Aaron Steinfeld, and Anthony Tomasic . 2008. RADAR: A Personal Assistant that Learns to Reduce Email Overload. AAAI. 1287--1293.

Digital Library

[17]

Erving Goffman . 2006. The presentation of self. Life as theater: A dramaturgical sourcebook (2006).

[18]

Mathieu Guillame-Bert and James L. Crowley . 2012. Learning Temporal Association Rules on Symbolic Time Sequences. (2012), 159--174.

[19]

Lixing Huang, Louis-Philippe Morency, and Jonathan Gratch . 2011. Virtual Rapport 2.0 International Workshop on Intelligent Virtual Agents. Springer, 68--79.

Digital Library

[20]

Stefan Kopp, Brigitte Krenn, Stacy Marsella, Andrew N Marshall, Catherine Pelachaud, Hannes Pirker, Kristinn R Thórisson, and Hannes Vilhjálmsson . 2006. Towards a common framework for multimodal generation: The behavior markup language International workshop on intelligent virtual agents. Springer, 205--217.

Digital Library

[21]

Christine Lisetti, Reza Amini, Ugan Yasavur, and Naphtali Rishe . 2013. I can help you change! an empathic virtual agent delivers behavior change health interventions. ACM Transactions on Management Information Systems (TMIS), Vol. 4, 4 (2013), 19.

Digital Library

[22]

Michael Madaio, Justine Cassell, and Amy Ogan . 2017. The impact of peer tutors' use of indirect feedback and instructions. Philadelphia, PA: International Society of the Learning Sciences.

[23]

Pattie Maes . 1989. How to do the right thing. Connection Science, Vol. 1, 3 (1989), 291--323.

[24]

Yoichi Matsuyama, Arjun Bhardwaj, Ran Zhao, Oscar J Romero, Sushma Anand Akoju, and Justine Cassell . 2016. Socially-aware animated intelligent personal assistant agent 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 224.

[25]

Karen Myers, Pauline Berry, Jim Blythe, Ken Conley, Melinda Gervasio, Deborah L McGuinness, David Morley, Avi Pfeffer, Martha Pollack, and Milind Tambe . 2007. An intelligent personal assistant for task and time management. AI Magazine, Vol. 28, 2 (2007), 47.

Digital Library

[26]

Ani Nenkova, Agustin Gravano, and Julia Hirschberg . 2008. High frequency word entrainment in spoken dialogue Proceedings of the 46th annual meeting of the association for computational linguistics on human language technologies: Short papers. Association for Computational Linguistics, 169--172.

Digital Library

[27]

Kate G Niederhoffer and James W Pennebaker . 2002. Linguistic style matching in social interaction. Journal of Language and Social Psychology Vol. 21, 4 (2002), 337--360.

[28]

Florian Nothdurft and Wolfgang Minker . 2016. Justification and transparency explanations in dialogue systems to maintain human-computer trust. Situated Dialog in Speech-Based Human-Computer Interaction. Springer, 41--50.

[29]

Hannes Ritschel, Tobias Baur, and Elisabeth André . 2017. Adapting a Robot's Linguistic Style Based on Socially-Aware Reinforcement Learning. (2017).

[30]

Oscar J. Romero . 2011. An evolutionary behavioral model for decision making. Adaptive Behavior, Vol. 19, 6 (2011), 451--475.

Digital Library

[31]

Oscar J Romero, Ran Zhao, and Justine Cassell . 2017. Cognitive-Inspired Conversational-Strategy Reasoner for Socially-Aware Agents. IJCAI (2017), 3807--3813.

Digital Library

[32]

Tanmay Sinha . {n. d.}. Cognitive Correlates of Rapport Dynamics in Longitudinal Peer Tutoring. (. {n. d.}).

[33]

Tanmay Sinha and Justine Cassell . 2015. We click, we align, we learn: Impact of influence and convergence processes on student learning and rapport building. In Proceedings of the 1st Workshop on Modeling INTERPERsonal SynchrONy And infLuence. ACM, 13--20.

Digital Library

[34]

Gabriel Skantze and Anna Hjalmarsson . 2013. Towards incremental speech generation in conversational systems. Computer Speech & Language Vol. 27, 1 (2013), 243--262.

Digital Library

[35]

Helen Spencer-Oatey . 2005. (Im) politeness, face and perceptions of rapport: unpackaging their bases and interrelationships. (2005).

[36]

Marcus Thiebaux, Stacy Marsella, Andrew N Marshall, and Marcelo Kallmann . 2008. Smartbody: Behavior realization for embodied conversational agents Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1. International Foundation for Autonomous Agents and Multiagent Systems, 151--158.

Digital Library

[37]

Linda Tickle-Degnen and Robert Rosenthal . 1990. The nature of rapport and its nonverbal correlates. Psychological inquiry, Vol. 1, 4 (1990), 285--293.

[38]

Vivian Tsai, Timo Baumann, Florian Pecune, and Justine Casell . 2018. Faster Responses are Better Responses: Introducing Incrementality into Sociable Virtual Personal Assistants. In Proceedings of the 2018 International Workshop on Spoken Dialog System Technology.

[39]

Gokhan Tur, Andreas Stolcke, Lynn Voss, Stanley Peters, Dilek Hakkani-Tur, John Dowding, Benoit Favre, Raquel Fernández, Matthew Frampton, Mike Frandsen, et almbox. . 2010. The CALO meeting assistant system. IEEE Transactions on Audio, Speech, and Language Processing, Vol. 18, 6 (2010), 1601--1611.

[40]

Ning Wang, David V Pynadath, and Susan G Hill . 2016. The impact of pomdp-generated explanations on trust and performance in human-robot teams Proceedings of the 2016 International Conference on Autonomous Agents & Multiagent Systems. International Foundation for Autonomous Agents and Multiagent Systems, 997--1005.

Digital Library

[41]

Nigel G Ward, Anais G Rivera, Karen Ward, and David G Novick . 2005. Root causes of lost time and user stress in a simple dialog system. (2005).

[42]

Yorick Wilks . 2010. Close engagements with artificial companions: key social, psychological, ethical and design issues. Vol. Vol. 8. John Benjamins Publishing.

[43]

Zhou Yu, Alexander I. Rudnicky, and Alan W. Black . 2017. Learning Conversational Systems that Interleave Task and Non-Task Content Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017. 4214--4220.

Digital Library

[44]

Ran Zhao, Alexandros Papangelis, and Justine Cassell . 2014. Towards a dyadic computational model of rapport management for human-virtual agent interaction International Conference on Intelligent Virtual Agents. Springer, 514--527.

[45]

Ran Zhao, Tanmay Sinha, Alan W Black, and Justine Cassell . 2016 a. Automatic recognition of conversational strategies in the service of a socially-aware dialog system. In 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. 381.

[46]

Ran Zhao, Tanmay Sinha, Alan W Black, and Justine Cassell . 2016 b. Socially-aware virtual agents: Automatically assessing dyadic rapport from temporal patterns of behavior. In International Conference on Intelligent Virtual Agents. Springer, 218--233.

[47]

Zian Zhao, Michael Madaio, Florian Pecune, Yoichi Matsuyama, and Justine Casell . 2018. Socially-Conditioned Task Reasoning for a Virtual Tutoring Agent Proceedings of the 2018 international conference on Autonomous agents and multi-agent systems. International Foundation for Autonomous Agents and Multiagent Systems.

Digital Library

Cited By

Jaber RZhong SKuoppamäki SHosseini AGessinger IBrumby DCowan BMcmillan D(2024)Cooking With Agents: Designing Context-aware Voice InteractionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642183(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642183
Razavi SSchubert Lvan Orden KAli MKane BHoque E(2022)Discourse Behavior of Older Adults Interacting with a Dialogue Agent Competent in Multiple TopicsACM Transactions on Interactive Intelligent Systems10.1145/348451012:2(1-21)Online publication date: 23-Jul-2022
https://dl.acm.org/doi/10.1145/3484510
Pecune FCallebert LMarsella SObaid MMubin ONagai YOsawa HAbdelrahman YFjeld M(2020)A Socially-Aware Conversational Recommender System for Personalized Recipe RecommendationsProceedings of the 8th International Conference on Human-Agent Interaction10.1145/3406499.3415079(78-86)Online publication date: 10-Nov-2020
https://dl.acm.org/doi/10.1145/3406499.3415079
Show More Cited By

Index Terms

Field Trial Analysis of Socially Aware Robot Assistant
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. Field studies
    2. Interaction paradigms
      1. Natural language interfaces

Recommendations

A User Simulator Architecture for Socially-Aware Conversational Agents
IVA '18: Proceedings of the 18th International Conference on Intelligent Virtual Agents

Over the last two decades, Reinforcement Learning (RL) has emerged the method of choice for data-driven dialog management. However, one of the limitations of RL methods for the optimization of dialog managers in the context of virtual conversational ...
A Socially-Aware Conversational Recommender System for Personalized Recipe Recommendations
HAI '20: Proceedings of the 8th International Conference on Human-Agent Interaction

One potential solution to help people change their eating behavior is to develop conversational systems able to recommend healthy recipes. Beyond the intrinsic quality of the recommendations themselves, various factors might also influence users? ...
Acoustic-Prosodic Entrainment and Rapport in Collaborative Learning Dialogues
MLA '14: Proceedings of the 2014 ACM workshop on Multimodal Learning Analytics Workshop and Grand Challenge

In spoken dialogue analysis, the speech signal is a rich source of information. We explore in this paper how low level features of the speech signal, such as pitch, loudness, and speaking rate, can inform a model of student interaction in collaborative ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

AAMAS '18: Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems

July 2018

2312 pages

General Chairs:
Elisabeth Andre
Augsburg University, Germany
,
Sven Koenig
University of Southern California, USA
,
Program Chairs:
Mehdi Dastani
Utrecht University, Netherlands
,
Gita Sukthankar
University of Central Florida, USA

Sponsors

In-Cooperation

ACM: Association for Computing Machinery

Publisher

International Foundation for Autonomous Agents and Multiagent Systems

Richland, SC

Publication History

Published: 09 July 2018

Check for updates

Author Tags

Qualifiers

Research-article

Conference

AAMAS '18

Sponsor:

SIGAI

AAMAS '18: Autonomous Agents and MultiAgent Systems

July 10 - 15, 2018

Stockholm, Sweden

Acceptance Rates

AAMAS '18 Paper Acceptance Rate 149 of 607 submissions, 25%;

Overall Acceptance Rate 1,155 of 5,036 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
182
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)0

Reflects downloads up to 19 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Jaber RZhong SKuoppamäki SHosseini AGessinger IBrumby DCowan BMcmillan D(2024)Cooking With Agents: Designing Context-aware Voice InteractionProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642183(1-13)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642183
Razavi SSchubert Lvan Orden KAli MKane BHoque E(2022)Discourse Behavior of Older Adults Interacting with a Dialogue Agent Competent in Multiple TopicsACM Transactions on Interactive Intelligent Systems10.1145/348451012:2(1-21)Online publication date: 23-Jul-2022
https://dl.acm.org/doi/10.1145/3484510
Pecune FCallebert LMarsella SObaid MMubin ONagai YOsawa HAbdelrahman YFjeld M(2020)A Socially-Aware Conversational Recommender System for Personalized Recipe RecommendationsProceedings of the 8th International Conference on Human-Agent Interaction10.1145/3406499.3415079(78-86)Online publication date: 10-Nov-2020
https://dl.acm.org/doi/10.1145/3406499.3415079
Gunson NSieińska WWalsh CDondrup CLemon O(2020)It's Good to Chat?Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents10.1145/3383652.3423889(1-8)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3383652.3423889
Pecune FMarsella SJain A(2020)A framework to co-optimize task and social dialogue policies using Reinforcement LearningProceedings of the 20th ACM International Conference on Intelligent Virtual Agents10.1145/3383652.3423877(1-8)Online publication date: 20-Oct-2020
https://dl.acm.org/doi/10.1145/3383652.3423877
Pecune FMurali STsai VMatsuyama YCassell JOka NKoda TObaid MNakanishi HMubin OTanaka K(2019)A Model of Social Explanations for a Conversational Movie Recommendation SystemProceedings of the 7th International Conference on Human-Agent Interaction10.1145/3349537.3351899(135-143)Online publication date: 25-Sep-2019
https://dl.acm.org/doi/10.1145/3349537.3351899
Hoegen RAneja DMcDuff DCzerwinski MPelachaud CMartin JBuschmeier HLucas GKopp S(2019)An End-to-End Conversational Style Matching AgentProceedings of the 19th ACM International Conference on Intelligent Virtual Agents10.1145/3308532.3329473(111-118)Online publication date: 1-Jul-2019
https://dl.acm.org/doi/10.1145/3308532.3329473
Jain APecune FMatsuyama YCassell JBogdanovych ARichards DSimoff SPelachaud CHeylen DTrescak T(2018)A User Simulator Architecture for Socially-Aware Conversational AgentsProceedings of the 18th International Conference on Intelligent Virtual Agents10.1145/3267851.3267916(133-140)Online publication date: 5-Nov-2018
https://dl.acm.org/doi/10.1145/3267851.3267916

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents