More Web Proxy on the site http://driver.im/

research-article

Automatic Emotion Recognition from Facial Expressions when Wearing a Mask

Authors:

Giovanna Castellano,

Berardina De Carolis,

Nicola MacchiaruloAuthors Info & Claims

CHItaly '21: Proceedings of the 14th Biannual Conference of the Italian SIGCHI Chapter

Article No.: 22, Pages 1 - 5

https://doi.org/10.1145/3464385.3464730

Published: 13 July 2021 Publication History

Abstract

People communicate emotions through several nonverbal channels and facial expressions play an important part in this communicative process. Automatic Facial Expression Recognition (FER) is a very hot topic that has attracted a lot of interest in the last years. Most FER systems try to recognize emotions from the entire face of a person. Unfortunately, due to pandemic situation, people wear a mask most of the time, thus their faces are not fully visible. In our study, we investigate the effectiveness of a FER system in recognizing emotions only from the eyes region, which is the sole visible region when wearing a mask by comparing the results of the same approach when applied to the entire face. The proposed pipeline involves several steps: detecting a face in an image, detecting a mask on a face, extracting the eyes region, and recognize the emotion expressed on the basis of such region. As it was expected, emotions that are related mainly to the mouth region (e.g. disgust) are not recognized at all and positive emotions are the ones that are better determined by considering only the region of the eyes.

References

[1]

M. Taufik Akbar, M. Nasrul Ilmi, Imanuel V. Rumayar, Jurike Moniaga, Tin-Kai Chen, and Andry Chowanda. 2019. Enhancing Game Experience with Facial Expression Recognition as Dynamic Balancing. Procedia Computer Science 157 (2019), 388 – 395. https://doi.org/10.1016/j.procs.2019.08.230 The 4th International Conference on Computer Science and Computational Intelligence (ICCSCI 2019) : Enabling Collaboration to Escalate Impact of Research Results for Society.

Digital Library

[2]

M. A. Assari and M. Rahmati. 2011. Driver drowsiness detection using face expression recognition. 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA) (2011), 337–341.

[3]

John N Bassili. 1979. Emotion recognition: the role of facial movement and the relative importance of upper and lower areas of the face.Journal of personality and social psychology 37, 11(1979), 2049.

[4]

Claus-Christian Carbon. 2020. Wearing face masks strongly confuses counterparts in reading emotions. Frontiers in Psychology 11 (2020), 2526.

[5]

Paul Ekman. 1999. Basic emotions. Handbook of cognition and emotion 98, 45-60 (1999), 16.

[6]

Ian J. Goodfellow, Dumitru Erhan, Pierre Luc Carrier, Aaron Courville, Mehdi Mirza, Ben Hamner, Will Cukierski, Yichuan Tang, David Thaler, Dong-Hyun Lee, Yingbo Zhou, Chetan Ramaiah, Fangxiang Feng, Ruifan Li, Xiaojie Wang, Dimitris Athanasakis, John Shawe-Taylor, Maxim Milakov, John Park, Radu Ionescu, Marius Popescu, Cristian Grozea, James Bergstra, Jingjing Xie, Lukasz Romaszko, Bing Xu, Zhang Chuang, and Yoshua Bengio. 2013. Challenges in Representation Learning: A report on three machine learning contests. arxiv:1307.0414 [stat.ML]

[7]

Felix Grundmann, Kai Epstude, and Susanne Scheibe. 2021. Face masks reduce emotion-recognition accuracy and perceived closeness. PloS one 16, 4 (2021), e0249792.

[8]

Davis E. King. 2009. Dlib-ml: A Machine Learning Toolkit. Journal of Machine Learning Research 10 (2009), 1755–1758.

Digital Library

[9]

Diederik P. Kingma and Jimmy Ba. 2017. Adam: A Method for Stochastic Optimization. arxiv:1412.6980 [cs.LG]

[10]

Mariska Kret and Beatrice De Gelder. 2012. Islamic headdress influences how emotion is recognized from the eyes. Frontiers in psychology 3 (2012), 110.

[11]

Michael Lankes, Stefan Riegler, Astrid Weiss, Thomas Mirlacher, Michael Pirker, and Manfred Tscheligi. 2008. Facial Expressions as Game Input with Different Emotional Feedback Conditions. In Proceedings of the 2008 International Conference on Advances in Computer Entertainment Technology (Yokohama, Japan) (ACE ’08). Association for Computing Machinery, New York, NY, USA, 253–256. https://doi.org/10.1145/1501750.1501809

Digital Library

[12]

Debi Roberson, Mariko Kikutani, Paula Döge, Lydia Whitaker, and Asifa Majid. 2012. Shades of emotion: What the addition of sunglasses or masks to faces reveals about the development of facial expression processing. Cognition 125, 2 (2012), 195–206.

[13]

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252.

Digital Library

[14]

Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4510–4520.

[15]

Karen Simonyan and Andrew Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. arxiv:1409.1556 [cs.CV]

[16]

Marko Tkalčič, Nima Maleki, Matevž Pesek, Mehdi Elahi, Francesco Ricci, and Matija Marolt. 2019. Prediction of Music Pairwise Preferences from Facial Expressions. In Proceedings of the 24th International Conference on Intelligent User Interfaces (Marina del Ray, California) (IUI ’19). Association for Computing Machinery, New York, NY, USA, 150–159. https://doi.org/10.1145/3301275.3302266

Digital Library

[17]

Bo Yang, Jianming Wu, and Gen Hattori. 2020. Facial Expression Recognition with the advent of face masks. In 19th International Conference on Mobile and Ubiquitous Multimedia. 335–337.

Digital Library

[18]

Benisemeni Esther Zakka and Hima Vadapalli. 2021. Detecting Learning Affect in E-Learning Platform Using Facial Emotion Expression. In Proceedings of the 11th International Conference on Soft Computing and Pattern Recognition (SoCPaR 2019), Ajith Abraham, M. A. Jabbar, Sanju Tiwari, and Isabel M. S. Jesus (Eds.). Springer International Publishing, Cham, 217–225.

Cited By

Yamaguchi KOkuyama KAtsuumi KMatsui KTaniguchi KHirai HNishikawa A(2024)Exploring the Feasibility of Personal Authentication through Facial Expression Recognition Mask Utilizing Conductive Threads導電糸を用いた表情識別マスクによる個人認証の実現可能性の検討IEEJ Transactions on Electronics, Information and Systems10.1541/ieejeiss.144.469144:5(469-470)Online publication date: 1-May-2024
https://doi.org/10.1541/ieejeiss.144.469
Shehu HBrowne WEisenbarth H(2024)Attention-Based Methods for Emotion Categorization From Partially Covered FacesIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2023.33328918:1(1057-1070)Online publication date: Feb-2024
https://doi.org/10.1109/TETCI.2023.3332891
Casas-Ortiz AEcheverria JJimenez-Tellez NSantos O(2024)Exploring the Impact of Partial Occlusion on Emotion Classification From Facial Expressions: A Comparative Study of XR Headsets and Face MasksIEEE Access10.1109/ACCESS.2024.338043912(44613-44627)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3380439
Show More Cited By

Recommendations

Automatic facial emotion recognition at the COVID-19 pandemic time
Abstract
People use various nonverbal communicative channels to convey emotions, among which facial expressions are considered the most important ones. Thus, automatic Facial Expression Recognition (FER) is a fundamental task to increase the perceptive ...
A Model Based Approach for Expressions Invariant Face Recognition
ICB '09: Proceedings of the Third International Conference on Advances in Biometrics

This paper describes an idea of recognizing the human face in the presence of strong facial expressions using model based approach. The features extracted for the face image sequences can be efficiently used for face recognition. The approach follows in ...
Robust facial expressions recognition using 3D average face and ameliorated adaboost
MM '13: Proceedings of the 21st ACM international conference on Multimedia

One of the most crucial techniques associated with Computer Vision is technology that deals with facial recognition, especially, the automatic estimation of facial expressions. However, in real-time facial expression recognition, when a face turns ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

CHItaly '21: Proceedings of the 14th Biannual Conference of the Italian SIGCHI Chapter

July 2021

237 pages

ISBN:9781450389778

DOI:10.1145/3464385

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 July 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

CHItaly '21

CHItaly '21: 14th Biannual Conference of the Italian SIGCHI Chapter

July 11 - 13, 2021

Bolzano, Italy

Acceptance Rates

Overall Acceptance Rate 109 of 242 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
295
Total Downloads

Downloads (Last 12 months)34
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yamaguchi KOkuyama KAtsuumi KMatsui KTaniguchi KHirai HNishikawa A(2024)Exploring the Feasibility of Personal Authentication through Facial Expression Recognition Mask Utilizing Conductive Threads導電糸を用いた表情識別マスクによる個人認証の実現可能性の検討IEEJ Transactions on Electronics, Information and Systems10.1541/ieejeiss.144.469144:5(469-470)Online publication date: 1-May-2024
https://doi.org/10.1541/ieejeiss.144.469
Shehu HBrowne WEisenbarth H(2024)Attention-Based Methods for Emotion Categorization From Partially Covered FacesIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2023.33328918:1(1057-1070)Online publication date: Feb-2024
https://doi.org/10.1109/TETCI.2023.3332891
Casas-Ortiz AEcheverria JJimenez-Tellez NSantos O(2024)Exploring the Impact of Partial Occlusion on Emotion Classification From Facial Expressions: A Comparative Study of XR Headsets and Face MasksIEEE Access10.1109/ACCESS.2024.338043912(44613-44627)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3380439
Farhadipour ATaghipour P(2023)Facial Emotion Recognition Under Mask Coverage Using a Data Augmentation Technique2023 13th International Conference on Computer and Knowledge Engineering (ICCKE)10.1109/ICCKE60553.2023.10326241(001-006)Online publication date: 1-Nov-2023
https://doi.org/10.1109/ICCKE60553.2023.10326241
Jiang CHasan MGedeon THossain M(2023)MaskTheFER: Mask-Aware Facial Expression Recognition using Convolutional Neural Network2023 International Conference on Digital Image Computing: Techniques and Applications (DICTA)10.1109/DICTA60407.2023.00069(456-463)Online publication date: 28-Nov-2023
https://doi.org/10.1109/DICTA60407.2023.00069
Okuyama KYamaguchi KAtsuumi KMatsui KHirai HNishikawa ATaniguchi K(2022)A Facial Expression Discrimination System Using the Expansion and Contraction of Face Masks and the Conductive Threadマスクの伸縮と導電糸を用いた表情識別システムの開発Journal of the Robotics Society of Japan10.7210/jrsj.40.84140:9(841-844)Online publication date: 2022
https://doi.org/10.7210/jrsj.40.841
Yang BWu JIkeda KHattori GSugano MIwasawa YMatsuo Y(2022)Face-mask-aware Facial Expression Recognition based on Face Parsing and Vision TransformerPattern Recognition Letters10.1016/j.patrec.2022.11.004164:C(173-182)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1016/j.patrec.2022.11.004
Castellano GDe Carolis BMacchiarulo N(2022)Automatic facial emotion recognition at the COVID-19 pandemic timeMultimedia Tools and Applications10.1007/s11042-022-14050-082:9(12751-12769)Online publication date: 22-Oct-2022
https://dl.acm.org/doi/10.1007/s11042-022-14050-0

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents