Improving Emotional Expression Recognition of Robots Using Regions of Interest from Human Data
Authors: 
Anne C. Bloem, 
Emilia Barakova, Inge M. Hootsmans, Lena M. Opheij, Romain H. A. Toebosch, Matthias Kerzel, Pablo BarrosAuthors Info & Claims
Pages 142 - 144
Abstract
This paper is the first step of an attempt to equip social robots with emotion recognition capabilities comparable to those of humans. Most of the recent deep learning solutions for facial expression recognition under-perform when deployed in Human-Robot-Interaction scenarios, although they are capable of breaking records on the most varied benchmarks on facial expression recognition. The main reason for that we believe is that they are using techniques that are developed for recognition of static pictures, while in real-life scenarios, we infer emotions from intervals of expression. Utilising on the feature of CNN to form regions of interests that are similar to human gaze patterns, we use recordings from human-gaze patterns to train such a network to infer facial emotions from 3 seconds video footage of humans expressing 6 basic emotions.
References
[1]
Hervé Abdi. 2007. RV coefficient and congruence coefficient. Encyclopedia of measurement and statistics, Vol. 849 (2007), 853.
[2]
Michael A Arbib and Jean-Marc Fellous. 2004. Emotions: from brain to robot. Trends in cognitive sciences, Vol. 8, 12 (2004), 554--561.
[3]
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In International conference on machine learning . 214--223.
[4]
Emilia I Barakova, Roman Gorbunov, and Matthias Rauterberg. 2015. Automatic interpretation of affective facial expressions in the context of interpersonal interaction. IEEE transactions on human-machine systems, Vol. 45, 4 (2015), 409--418.
[5]
Pablo Barros, German Parisi, and Stefan Wermter. 2019. A Personalized Affective Memory Model for Improving Emotion Recognition. In International Conference on Machine Learning. 485--494.
[6]
Pablo Barros and Stefan Wermter. 2016. Developing crossmodal expression recognition based on a deep neural model. Adaptive behavior, Vol. 24, 5 (2016), 373--396.
[7]
Cynthia Breazeal, Aaron Edsinger, Paul Fitzpatrick, and Brian Scassellati. 2001. Active vision for sociable robots. IEEE Transactions on systems, man, and cybernetics-part A: Systems and Humans, Vol. 31, 5 (2001), 443--453.
[8]
Cynthia Breazeal and Brian Scassellati. 1999. A context-dependent attention system for a social robot. rn, Vol. 255 (1999), 3.
[9]
Chaona Chen, Oliver GB Garrod, Jiayu Zhan, Jonas Beskow, Philippe G Schyns, and Rachael E Jack. 2018a. Reverse engineering psychologically valid facial expressions of emotion into social robots. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE, 448--452.
[10]
Chaona Chen, Laura B Hensel, Yaocong Duan, Robin AA Ince, Oliver GB Garrod, Jonas Beskow, Rachael E Jack, and Philippe G Schyns. 2019. Equipping social robots with culturally-sensitive facial expressions of emotion using data-driven methods. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE, 1--8.
[11]
Luefeng Chen, Mengtian Zhou, Wanjuan Su, Min Wu, Jinhua She, and Kaoru Hirota. 2018b. Softmax regression based deep sparse autoencoder network for facial emotion recognition in human-robot interaction. Information Sciences, Vol. 428 (2018), 49--61.
[12]
Abhinav Dhall, Roland Goecke, Simon Lucey, Tom Gedeon, et almbox. 2012. Collecting large, richly annotated facial-expression databases from movies. IEEE multimedia, Vol. 19, 3 (2012), 34--41.
[13]
Pedro D Marrero Fernandez, Fidel A Guerrero Pe na, Tsang Ing Ren, and Alexandre Cunha. 2019. FERAtt: Facial Expression Recognition with Attention Net. arXiv preprint arXiv:1902.03284 (2019).
[14]
Amogh Gudi, H Emrah Tasli, Tim M Den Uyl, and Andreas Maroulis. 2015. Deep learning based facs action unit occurrence and intensity estimation. In 2015 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), Vol. 6. IEEE, 1--5.
[15]
Devamanyu Hazarika, Sruthi Gorantla, Soujanya Poria, and Roger Zimmermann. 2018. Self-attentive feature-level fusion for multimodal emotion detection. In 2018 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 196--201.
[16]
Roy S Hessels, Chantal Kemner, Carlijn van den Boomen, and Ignace TC Hooge. 2016. The area-of-interest problem in eyetracking research: A noise-robust solution for face and sparse stimuli. Behavior research methods, Vol. 48, 4 (2016), 1694--1712.
[17]
Kun-Yi Huang, Chung-Hsien Wu, Qian-Bei Hong, Ming-Hsiang Su, and Yi-Hsuan Chen. 2019. Speech Emotion Recognition Using Deep Neural Network Considering Verbal and Nonverbal Speech Sounds. In ICASSP 2019--2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 5866--5870.
[18]
Mariska E Kret, Karin Roelofs, Jeroen J Stekelenburg, and Beatrice de Gelder. 2013. Emotional signals from faces, bodies and scenes influence observers' face expressions, fixations and pupil-size. Frontiers in human neuroscience, Vol. 7 (2013).
[19]
Giorgio Metta, Giulio Sandini, David Vernon, Lorenzo Natale, and Francesco Nori. 2008. The iCub humanoid robot: an open platform for research in embodied cognition. In Proceedings of the 8th workshop on performance metrics for intelligent systems. ACM, 50--56.
[20]
Ali Mollahosseini, Behzad Hasani, and Mohammad H Mahoor. 2017. Affectnet: A database for facial expression, valence, and arousal computing in the wild. arXiv preprint arXiv:1708.03985 (2017).
[21]
Ariel Ruiz-Garcia, Nicola Webb, Vasile Palade, Mark Eastwood, and Mark Elshaw. 2018. Deep Learning for Real Time Facial Expression Recognition in Social Robots. In International Conference on Neural Information Processing. Springer, 392--402.
[22]
Evangelos Sariyanidi, Hatice Gunes, and Andrea Cavallaro. 2014. Automatic analysis of facial affect: A survey of registration, representation, and recognition. IEEE transactions on pattern analysis and machine intelligence, Vol. 37, 6 (2014), 1113--1133.
[23]
Alessandra Sciutti, Francesco Rea, and Giulio Sandini. 2014. When you are young,(robot's) looks matter. Developmental changes in the desired properties of a robot friend. In The 23rd IEEE international symposium on robot and human interactive communication. IEEE, 567--573.
[24]
Mohammad Soleymani, Maja Pantic, and Thierry Pun. 2012. Multimodal emotion recognition in response to videos. IEEE transactions on affective computing, Vol. 3, 2 (2012), 211--223.
[25]
Fei Wang, Hu Chen, Li Kong, and Weihua Sheng. 2018. Real-time Facial Expression Recognition on Robot for Healthcare. In 2018 IEEE International Conference on Intelligence and Safety for Robotics (ISR). IEEE, 402--406.
[26]
AmirAli Bagher Zadeh, Paul Pu Liang, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Multimodal language analysis in the wild: Cmu-mosei dataset and interpretable dynamic fusion graph. In The 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vol. 1. 2236--2246.
[27]
Stefanos Zafeiriou, Dimitrios Kollias, Mihalis A Nicolaou, Athanasios Papaioannou, Guoying Zhao, and Irene Kotsia. 2017. Aff-wild: Valence and arousal 'in-the-wild'challenge. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2017 IEEE Conference on. IEEE, 1980--1987.
[28]
Matthew D Zeiler and Rob Fergus. 2014. Visualizing and understanding convolutional networks. In European conference on computer vision. Springer, 818--833.
Index Terms
- Improving Emotional Expression Recognition of Robots Using Regions of Interest from Human Data
Recommendations
Human-Computer Interaction Using Emotion Recognition from Facial Expression
EMS '11: Proceedings of the 2011 UKSim 5th European Symposium on Computer Modeling and SimulationThis paper describes emotion recognition system based on facial expression. A fully automatic facial expression recognition system is based on three steps: face detection, facial characteristic extraction and facial expression classification. We have ...
Human centric facial expression recognition
HCI '18: Proceedings of the 32nd International BCS Human Computer Interaction ConferenceFacial expression recognition (FER) is an area of active research, both in computer science and in behavioural science. Across these domains there is evidence to suggest that humans and machines find it easier to recognise certain emotions, for example ...
A fast identity-independent expression recognition system for robust cartoonification using smart devices
ICVGIP '16: Proceedings of the Tenth Indian Conference on Computer Vision, Graphics and Image ProcessingFacial expressions convey rich information about emotions, intentions and other internal states of a person. Automatic facial expression and cartoonification systems are aiming towards the application of computer vision systems in human computer ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2020
702 pages
ISBN:9781450370578
DOI:10.1145/3371382
- General Chairs:
- Tony Belpaeme,
- James Young,
- Program Chairs:
- Hatice Gunes,
- Laurel Riek
Copyright © 2020 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 01 April 2020
Check for updates
Author Tags
Qualifiers
- Abstract
Conference
HRI '20: ACM/IEEE International Conference on Human-Robot Interaction
March 23 - 26, 2020
Cambridge, United Kingdom
Acceptance Rates
Overall Acceptance Rate 192 of 519 submissions, 37%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 219Total Downloads
- Downloads (Last 12 months)5
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Jan 2025
Other Metrics
Citations
Cited By
View allView Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in