More Web Proxy on the site http://driver.im/

short-paper

Analysis of Mood Changes and Facial Expressions during Cognitive Behavior Therapy through a Virtual Agent

Authors:

Kazuhiro Shidara,

Hiroyoshi Adachi,

Daisuke Kanayama,

Yukako Sakagami,

Satoshi NakamuraAuthors Info & Claims

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

Pages 477 - 481

https://doi.org/10.1145/3395035.3425223

Published: 27 December 2020 Publication History

Abstract

In cognitive behavior therapy (CBT) with a virtual agent, facial expression processing is expected to be useful for dialogue response selection empathic dialogue. Unfortunately, its use in current works remains limited. One reason for this situation is the lack of research on the relationship between mood changes facial expressions through CBT-oriented interaction. This study confirms the improvement of negative moods through interaction with a virtual agent and identifying facial expressions that correlate with mood changes. Based on the cognitive restructuring of CBT, we created a fixed dialogue scenario and implemented it in a virtual agent. We recorded facial expressions during dialogues with 23 undergraduate and graduate students, calculated 17 types of action units (AUs), which are the units of facial movements, and performed a correlation analysis using the change rate of mood scores and the amount of the changes in the AUs. The mean mood improvement rate was 35%, and the mood improvements showed correlations with AU5 (r = -0.51), AU17 (r = 0.45), AU25 (r = -0.43), and AU45 (r = 0.45). These results imply that mood changes are reflected in facial expressions. The AUs identified in this study have the potential to be used for agent-interaction modeling.

References

[1]

Tadas Baltru?aitis, Peter Robinson, and Louis-Philippe Morency. 2016. Openface: an open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--10.

[2]

AT Beck, AJ Rush, BF Shaw, and G Emery. 1979. Cognitive therapy of depression Guilford Press. New York (1979).

[3]

Judith S Beck and AT Beck. 2011. Cognitive behavior therapy. New York: Basics and beyond. Guilford Publication (2011).

[4]

David DeVault, Ron Artstein, Grace Benn, Teresa Dey, Ed Fast, Alesia Gainer, Kallirroi Georgila, Jon Gratch, Arno Hartholt, Margaux Lhommet, et al. 2014. SimSensei Kiosk: A virtual human interviewer for healthcare decision support. In Proceedings of the 2014 international conference on Autonomous agents and multi-agent systems. 1061--1068.

Digital Library

[5]

Paul Ekman andWallace V Friesen. 1978. Facial action coding systems. Consulting Psychologists Press.

[6]

Kathleen Kara Fitzpatrick, Alison Darcy, and Molly Vierhile. 2017. Delivering cognitive behavior therapy to young adults with symptoms of depression and anxiety using a fully automated conversational agent (Woebot): a randomized controlled trial. JMIR mental health 4, 2 (2017), e19.

[7]

Russell Fulmer, Angela Joerin, Breanna Gentile, Lysanne Lakerink, and Michiel Rauws. 2018. Using psychological artificial intelligence (Tess) to relieve symptoms of depression and anxiety: randomized controlled trial. JMIR mental health 5, 4 (2018), e64.

[8]

Dennis Greenberger and Christine A Padesky. 1995. Mind over Mood: a cognitive therapy treatment manual for clients.

[9]

Becky Inkster, Shubhankar Sarda, and Vinod Subramanian. 2018. An empathydriven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study. JMIR mHealth and uHealth 6, 11 (2018), e12106.

[10]

Ronald C Kessler, Gavin Andrews, Lisa J Colpe, Eva Hiripi, Daniel K Mroczek, S-LT Normand, Ellen E Walters, and Alan M Zaslavsky. 2002. Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological medicine 32, 6 (2002), 959--976.

[11]

Everlyne Kimani, Timothy Bickmore, Ha Trinh, and Paola Pedrelli. 2019. You'll be Great: Virtual Agent-based Cognitive Restructuring to Reduce Public Speaking Anxiety. In 2019 8th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 641--647.

[12]

Sander L Koole and Wolfgang Tschacher. 2016. Synchrony in psychotherapy: A review and an integrative framework for the therapeutic alliance. Frontiers in psychology 7 (2016), 862.

[13]

Akinobu Lee, Keiichiro Oura, and Keiichi Tokuda. 2013. MMDAgent'A fully open-source toolkit for voice interaction systems. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 8382--8385.

[14]

Joao Luis Zeni Montenegro, Cristiano André da Costa, and Rodrigo da Rosa Righi. 2019. Survey of conversational agents in health. Expert Systems with Applications 129 (2019), 56--67.

Digital Library

[15]

Jacqueline B Persons and David D Burns. 1985. Mechanisms of action of cognitive therapy: The relative contributions of technical and interpersonal interventions. Cognitive therapy and Research 9, 5 (1985), 539--551.

[16]

Lazlo Ring, Timothy Bickmore, and Paola Pedrelli. 2016. An affectively aware virtual therapist for depression counseling. In ACM SIGCHI Conference on Human Factors in Computing Systems (CHI) workshop on Computing and Mental Health. 01951--12.

[17]

Shinichiro Suganuma, Daisuke Sakamoto, and Haruhiko Shimoyama. 2018. An embodied conversational agent for unguided internet-based cognitive behavior therapy in preventative mental health: feasibility and acceptability pilot trial. JMIR mental health 5, 3 (2018), e10454.

[18]

Hiroki Tanaka, Hideki Negoro, Hidemi Iwasaka, and Satoshi Nakamura. 2017. Embodied conversational agents for multimodal automated social skills training in people with autism spectrum disorders. PloS one 12, 8 (2017).

[19]

Hiroki Tanaka, Sakti Sakriani, Graham Neubig, Tomoki Toda, Hideki Negoro, Hidemi Iwasaka, and Satoshi Nakamura. 2016. Teaching social communication skills through human-agent interaction. ACM Transactions on Interactive Intelligent Systems (TiiS) 6, 2 (2016), 1--26.

Digital Library

Cited By

Islam RBae S(2024)FacePsy: An Open-Source Affective Mobile Sensing System - Analyzing Facial Behavior and Head Gesture for Depression Detection in Naturalistic SettingsProceedings of the ACM on Human-Computer Interaction10.1145/36765058:MHCI(1-32)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676505
Kitson ASlovak PAntle A(2024)Supporting Cognitive Reappraisal With Digital Technology: A Content Analysis and Scoping Review of Challenges, Interventions, and Future DirectionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642488(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642488
Kruse LHertel JMostajeran FSchmidt SSteinicke F(2023)Would You Go to a Virtual Doctor? A Systematic Literature Review on User Preferences for Embodied Virtual Agents in Healthcare2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00082(672-682)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00082
Show More Cited By

Index Terms

Analysis of Mood Changes and Facial Expressions during Cognitive Behavior Therapy through a Virtual Agent

Index terms have been assigned to the content through auto-classification.

Recommendations

Multi-modal emotion analysis from facial expressions and electroencephalogram

Emotion recognition from spontaneous facial expression with a new percentage feature.Extraction and selection of spectral power and spectral power difference features for EEG.A multi-modal emotion recognition for valence and arousal classes.A comparison ...
Cross-cultural design of facial expressions for humanoids: is there cultural difference between Japan and Denmark?
MMAsia '20: Proceedings of the 2nd ACM International Conference on Multimedia in Asia

In this research, the authors succeeded in creating facial expressions made with the minimum necessary elements for recognizing a face. The elements are two eyes and a mouth made using precise circles, which are transformed to make facial expressions ...
Interpreting Human and Avatar Facial Expressions
INTERACT '09: Proceedings of the 12th IFIP TC 13 International Conference on Human-Computer Interaction: Part I

This paper investigates the impact of contradictory emotional content on people's ability to identify the emotion expressed on avatar faces as compared to human faces. Participants saw emotional faces (human or avatar) coupled with emotional texts. The ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

October 2020

548 pages

ISBN:9781450380027

DOI:10.1145/3395035

General Chairs:
Khiet Truong
University of Twente, the Netherlands
,
Dirk Heylen
University of Twente, the Netherlands
,
Mary Czerwinski
Microsoft Research, USA
,
Program Chairs:
Nadia Berthouze
University College London, United Kingdom
,
Mohamed Chetouani
Sorbonne University, France
,
Mikio Nakano
C4A Research Institute, Japan

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

JST CREST

Conference

ICMI '20

Sponsor:

SIGCHI

ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 25 - 29, 2020

Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
231
Total Downloads

Downloads (Last 12 months)58
Downloads (Last 6 weeks)14

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Islam RBae S(2024)FacePsy: An Open-Source Affective Mobile Sensing System - Analyzing Facial Behavior and Head Gesture for Depression Detection in Naturalistic SettingsProceedings of the ACM on Human-Computer Interaction10.1145/36765058:MHCI(1-32)Online publication date: 24-Sep-2024
https://dl.acm.org/doi/10.1145/3676505
Kitson ASlovak PAntle A(2024)Supporting Cognitive Reappraisal With Digital Technology: A Content Analysis and Scoping Review of Challenges, Interventions, and Future DirectionsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642488(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642488
Kruse LHertel JMostajeran FSchmidt SSteinicke F(2023)Would You Go to a Virtual Doctor? A Systematic Literature Review on User Preferences for Embodied Virtual Agents in Healthcare2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00082(672-682)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00082
Lepe-Salazar FSalgado-Torres S(2023)Multiple Composite Scenarios: A Game-Based Methodology for the Prevention of Mental DisordersEntertainment Computing10.1016/j.entcom.2022.10051944(100519)Online publication date: Jan-2023
https://doi.org/10.1016/j.entcom.2022.100519
Shidara KTanaka HAdachi HKanayama DSakagami YKudo TNakamura S(2022)Automatic Thoughts and Facial Expressions in Cognitive Restructuring With Virtual AgentsFrontiers in Computer Science10.3389/fcomp.2022.7624244Online publication date: 2-Feb-2022
https://doi.org/10.3389/fcomp.2022.762424
Fu YMichelson RLin YNguyen LTayebi THiniker A(2022)Social Emotional Learning with Conversational AgentsProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35346226:2(1-23)Online publication date: 7-Jul-2022
https://dl.acm.org/doi/10.1145/3534622
Jeong SAymerich-Franch LArias KAlghowinem SLapedriza APicard RPark HBreazeal C(2022)Deploying a robotic positive psychology coach to improve college students’ psychological well-beingUser Modeling and User-Adapted Interaction10.1007/s11257-022-09337-833:2(571-615)Online publication date: 11-Jul-2022
https://doi.org/10.1007/s11257-022-09337-8
Tanaka HNakamura S(2021)Virtual Agent Design for Social Skills Training Considering Autistic Traits2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC)10.1109/EMBC46164.2021.9630741(4953-4956)Online publication date: 1-Nov-2021
https://doi.org/10.1109/EMBC46164.2021.9630741
Shidara KTanaka HAdachi HKanayama DSakagami YKudo TNakamura S(2021)Relationship between Mood Improvement and Questioning to Evaluate Automatic Thoughts in Cognitive Restructuring with a Virtual Agent2021 9th International Conference on Affective Computing and Intelligent Interaction Workshops and Demos (ACIIW)10.1109/ACIIW52867.2021.9666312(1-5)Online publication date: 28-Sep-2021
https://doi.org/10.1109/ACIIW52867.2021.9666312

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents