More Web Proxy on the site http://driver.im/

research-article

Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective

Authors:

Nikolaos Kegkeroglou,

Panagiotis P. Filntisis,

Petros MaragosAuthors Info & Claims

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

Pages 69 - 76

https://doi.org/10.1145/3594806.3594829

Published: 10 August 2023 Publication History

Abstract

The COVID-19 pandemic has undoubtedly changed the standards and affected all aspects of our lives, especially social communication. It has forced people to extensively wear medical face masks, in order to prevent transmission. This face occlusion can strongly irritate emotional reading from the face and urges us to incorporate the whole body as an emotional cue. In this paper, we conduct insightful studies about the effect of face occlusion on emotion recognition performance, and showcase the superiority of full body input over the plain masked face. We utilize a deep learning model based on the Temporal Segment Network framework, and aspire to fully overcome the face mask consequences. Although facial and bodily features can be learned from a single input, this may lead to irrelevant information confusion. By processing those features separately and fusing their prediction scores, we are more effectively taking advantage of both modalities. This framework also naturally supports temporal modeling, by mingling information among neighboring frames. In combination, these techniques form an effective system capable of tackling emotion recognition difficulties, caused by safety protocols applied in crucial areas.

References

[1]

Brandon Amos, Bartosz Ludwiczuk, and Mahadev Satyanarayanan. 2016. OpenFace: A general-purpose face recognition library with mobile applications. Technical Report. CMU-CS-16-118, CMU School of Computer Science.

[2]

Hillel Aviezer, Yaacov Trope, and Alexander Todorov. 2012. Body Cues, Not Facial Expressions, Discriminate Between Intense Positive and Negative Emotions. Science 338, 6111 (Nov. 2012), 1225–1229. https://doi.org/10.1126/science.1224313

[3]

Valentin Bazarevsky, Ivan Grishchenko, Karthik Raveendran, Tyler Zhu, Fan Zhang, and Matthias Grundmann. 2020. BlazePose: On-device Real-time Body Pose tracking. https://doi.org/10.48550/ARXIV.2006.10204

[4]

Tony Belpaeme, Paul Baxter, Joachim de Greeff, James Kennedy, Robin Read, Rosemarijn Looije, Mark Neerincx, Ilaria Baroni, and Mattia Coti Zelati. 2013. Child-Robot Interaction: Perspectives and Challenges. In Social Robotics. Springer International Publishing, 452–459. https://doi.org/10.1007/978-3-319-02675-6_45

Digital Library

[5]

Cynthia Breazeal. 2003. Emotion and sociable humanoid robots. International Journal of Human-Computer Studies 59, 1-2 (July 2003), 119–155. https://doi.org/10.1016/s1071-5819(03)00018-1

Digital Library

[6]

Claus-Christian Carbon. 2020. Wearing Face Masks Strongly Confuses Counterparts in Reading Emotions. Frontiers in Psychology 11 (Sept. 2020). https://doi.org/10.3389/fpsyg.2020.566886

[7]

Claus-Christian Carbon and Martin Serrano. 2021. The Impact of Face Masks on the Emotional Reading Abilities of Children—A Lesson From a Joint School–University Project. i-Perception 12, 4 (July 2021), 204166952110382. https://doi.org/10.1177/20416695211038265

[8]

Beatrice de Gelder. 2009. Why bodies? Twelve reasons for including bodily expressions in affective neuroscience. Philosophical Transactions of the Royal Society B: Biological Sciences 364, 1535 (Dec. 2009), 3475–3484. https://doi.org/10.1098/rstb.2009.0190

[9]

Niki Efthymiou, Panagiotis P. Filntisis, Gerasimos Potamianos, and Petros Maragos. 2021. A Robotic Edutainment Framework for Designing Child-Robot Interaction Scenarios. In The 14th PErvasive Technologies Related to Assistive Environments Conference (Corfu, Greece) (PETRA 2021). Association for Computing Machinery, New York, NY, USA, 160–166. https://doi.org/10.1145/3453892.3458048

Digital Library

[10]

Panagiotis P. Filntisis, Niki Efthymiou, Petros Koutras, Gerasimos Potamianos, and Petros Maragos. 2019. Fusing Body Posture With Facial Expressions for Joint Recognition of Affect in Child–Robot Interaction. IEEE Robotics and Automation Letters 4, 4 (Oct. 2019), 4011–4018. https://doi.org/10.1109/lra.2019.2930434

[11]

Panagiotis P. Filntisis, Niki Efthymiou, Gerasimos Potamianos, and Petros Maragos. 2020. Emotion Understanding in Videos Through Body, Context, and Visual-Semantic Embedding Loss. In Computer Vision – ECCV 2020 Workshops, Adrien Bartoli and Andrea Fusiello (Eds.). Springer International Publishing, Cham, 747–755.

[12]

Panagiotis P. Filntisis, Niki Efthymiou, Gerasimos Potamianos, and Petros Maragos. 2021. An Audiovisual Child Emotion Recognition System for Child-Robot Interaction Applications. In 2021 29th European Signal Processing Conference (EUSIPCO). 791–795. https://doi.org/10.23919/EUSIPCO54536.2021.9616106

[13]

Goren Gordon, Samuel Spaulding, Jacqueline Kory Westlund, Jin Joo Lee, Luke Plummer, Marayna Martinez, Madhurima Das, and Cynthia Breazeal. 2016. Affective Personalization of a Social Robot Tutor for Children’s Second Language Skills. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (Phoenix, Arizona) (AAAI’16). AAAI Press, 3951–3957.

Digital Library

[14]

Felix Grundmann, Kai Epstude, and Susanne Scheibe. 2021. Face masks reduce emotion-recognition accuracy and perceived closeness. PLOS ONE 16, 4 (April 2021), e0249792. https://doi.org/10.1371/journal.pone.0249792

[15]

Hatice Gunes and Massimo Piccardi. 2007. Bi-modal emotion recognition from expressive face and body gestures. Journal of Network and Computer Applications 30, 4 (Nov. 2007), 1334–1345. https://doi.org/10.1016/j.jnca.2006.09.007

Digital Library

[16]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE. https://doi.org/10.1109/cvpr.2016.90

[17]

Geoffrey E. Hinton, Simon Osindero, and Yee-Whye Teh. 2006. A Fast Learning Algorithm for Deep Belief Nets. Neural Comput. 18, 7 (jul 2006), 1527–1554. https://doi.org/10.1162/neco.2006.18.7.1527

Digital Library

[18]

Yibo Huang, Hongqian Wen, Linbo Qing, Rulong Jin, and Leiming Xiao. 2021. Emotion Recognition Based on Body and Context Fusion in the Wild. In 2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). IEEE. https://doi.org/10.1109/iccvw54120.2021.00403

[19]

Yury Kartynnik, Artsiom Ablavatski, Ivan Grishchenko, and Matthias Grundmann. 2019. Real-time Facial Surface Geometry from Monocular Video on Mobile GPUs. https://doi.org/10.48550/ARXIV.1907.06724

[20]

Iolanda Leite, Ginevra Castellano, André Pereira, Carlos Martinho, and Ana Paiva. 2012. Modelling Empathic Behaviour in a Robotic Game Companion for Children: An Ethnographic Study in Real-World Settings. In Proceedings of the Seventh Annual ACM/IEEE International Conference on Human-Robot Interaction (Boston, Massachusetts, USA) (HRI ’12). Association for Computing Machinery, New York, NY, USA, 367–374. https://doi.org/10.1145/2157689.2157811

Digital Library

[21]

Shan Li and Weihong Deng. 2022. Deep Facial Expression Recognition: A Survey. IEEE Transactions on Affective Computing 13, 3 (July 2022), 1195–1215. https://doi.org/10.1109/taffc.2020.2981446

[22]

Ji Lin, Chuang Gan, and Song Han. 2018. TSM: Temporal Shift Module for Efficient Video Understanding. https://doi.org/10.48550/ARXIV.1811.08383

[23]

Yu Luo, Jianbo Ye, Reginald B. Adams, Jia Li, Michelle G. Newman, and James Z. Wang. 2018. ARBEE: Towards Automated Recognition of Bodily Expression of Emotion In the Wild. (2018). https://doi.org/10.48550/ARXIV.1808.09568

[24]

Marco Marini, Alessandro Ansani, Fabio Paglieri, Fausto Caruana, and Marco Viola. 2021. The impact of facemasks on emotion recognition, trust attribution and re-identification. Scientific Reports 11, 1 (March 2021). https://doi.org/10.1038/s41598-021-84806-5

[25]

Elisabeta Marinoiu, Mihai Zanfir, Vlad Olaru, and Cristian Sminchisescu. 2018. 3D Human Sensing, Action and Emotion Recognition in Robot Assisted Therapy of Children with Autism. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE. https://doi.org/10.1109/cvpr.2018.00230

[26]

Dung Nguyen, Kien Nguyen, Sridha Sridharan, Afsane Ghasemi, David Dean, and Clinton Fookes. 2017. Deep Spatio-Temporal Features for Multimodal Emotion Recognition. In 2017 IEEE Winter Conference on Applications of Computer Vision (WACV). 1215–1223. https://doi.org/10.1109/WACV.2017.140

[27]

Behnaz Nojavanasghari, Tadas Baltrušaitis, Charles E. Hughes, and Louis-Philippe Morency. 2016. EmoReact: A Multimodal Approach and Dataset for Recognizing Emotional Responses in Children. In Proceedings of the 18th ACM International Conference on Multimodal Interaction (Tokyo, Japan) (ICMI ’16). Association for Computing Machinery, New York, NY, USA, 137–144. https://doi.org/10.1145/2993148.2993168

Digital Library

[28]

Eilidh Noyes, Josh P. Davis, Nikolay Petrov, Katie L. H. Gray, and Kay L. Ritchie. 2021. The effect of face masks and sunglasses on identity and expression recognition with super-recognizers and typical observers. Royal Society Open Science 8, 3 (March 2021). https://doi.org/10.1098/rsos.201169

[29]

Reinhard Pekrun, Thomas Goetz, Wolfram Titz, and Raymond P. Perry. 2002. Academic Emotions in Students' Self-Regulated Learning and Achievement: A Program of Qualitative and Quantitative Research. Educational Psychologist 37, 2 (Jan. 2002), 91–105. https://doi.org/10.1207/s15326985ep3702_4

[30]

Ioannis Pikoulis, Panagiotis P. Filntisis, and Petros Maragos. 2021. Leveraging Semantic Scene Characteristics and Multi-Stream Convolutional Architectures in a Contextual Approach for Video-Based Visual Emotion Recognition in the Wild. (2021). https://doi.org/10.48550/ARXIV.2105.07484

[31]

J.C. Read and P. Markopoulos. 2013. Child-computer interaction. International Journal of Child-Computer Interaction 1, 1 (Jan. 2013), 2–6. https://doi.org/10.1016/j.ijcci.2012.09.001

[32]

Byron Reeves and Clifford Nass. 1996. The Media Equation: How People Treat Computers, Television, and New Media Like Real People and Pla. Bibliovault OAI Repository, the University of Chicago Press (01 1996).

[33]

Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2016. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. (2016). https://doi.org/10.48550/ARXIV.1610.02391

[34]

Karen Simonyan and Andrew Zisserman. 2014. Two-Stream Convolutional Networks for Action Recognition in Videos. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1 (Montreal, Canada) (NIPS’14). MIT Press, Cambridge, MA, USA, 568–576.

[35]

Maria Tsantani, Vita Podgajecka, Katie L. H. Gray, and Richard Cook. 2022. How does the presence of a surgical face mask impair the perceived intensity of facial emotions?PLOS ONE 17, 1 (Jan. 2022), e0262344. https://doi.org/10.1371/journal.pone.0262344

[36]

Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. 2016. Temporal Segment Networks: Towards Good Practices for Deep Action Recognition. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 20–36.

[37]

Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018. Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. https://doi.org/10.48550/ARXIV.1801.07455

[38]

Zhengyuan Yang, Amanda Kay, Yuncheng Li, Wendi Cross, and Jiebo Luo. 2020. Pose-based Body Language Recognition for Emotion and Psychiatric Symptom Interpretation. CoRR abs/2011.00043 (2020). arXiv:2011.00043https://arxiv.org/abs/2011.00043

[39]

Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann. 2020. MediaPipe Hands: On-device Real-time Hand Tracking. https://doi.org/10.48550/ARXIV.2006.10214

Cited By

Nangam PPuangchuen PThongkerd KSuwannsiri OKerdvibulvech C(2023)A Study of Integration of Digital Fiber in Medical Masks for Health Monitoring of WearersTransactions of the Indian National Academy of Engineering10.1007/s41403-023-00433-89:1(129-139)Online publication date: 26-Oct-2023
https://doi.org/10.1007/s41403-023-00433-8

Index Terms

Medical Face Masks and Emotion Recognition from the Body: Insights from a Deep Learning Perspective
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems

Recommendations

Face-mask-aware Facial Expression Recognition based on Face Parsing and Vision Transformer
Highlights
- Face parsing is the first time used to locate the scope of the facial area for face-mask-aware facial expression recognition.
Abstract
As wearing face masks is becoming an embedded practice due to the COVID-19 pandemic, facial expression recognition (FER) that takes face masks into account is now a problem that needs to be solved. In this paper, we propose a face ...
Facial Expression Recognition with the advent of face masks
MUM '20: Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia

With the worldwide spread of COVID-19, wearing face masks while interaction in public is becoming a common behavior to protect against infection. Thus, how to improve effectiveness of existing facial expression recognition (FER) technology on masked ...
An efficient deep learning technique for facial emotion recognition
Abstract
Emotion recognition from facial images is considered as a challenging task due to the varying nature of facial expressions. The prior studies on emotion classification from facial images using deep learning models have focused on emotion ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Other conferences

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

July 2023

797 pages

ISBN:9798400700699

DOI:10.1145/3594806

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 August 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Horizon Europe

Conference

PETRA '23

PETRA '23: Proceedings of the 16th International Conference on PErvasive Technologies Related to Assistive Environments

July 5 - 7, 2023

Corfu, Greece

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
51
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)9

Reflects downloads up to 10 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Nangam PPuangchuen PThongkerd KSuwannsiri OKerdvibulvech C(2023)A Study of Integration of Digital Fiber in Medical Masks for Health Monitoring of WearersTransactions of the Indian National Academy of Engineering10.1007/s41403-023-00433-89:1(129-139)Online publication date: 26-Oct-2023
https://doi.org/10.1007/s41403-023-00433-8

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents