More Web Proxy on the site http://driver.im/

short-paper

Investigating LSTM for Micro-Expression Recognition

Authors:

Roland GoeckeAuthors Info & Claims

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

Pages 7 - 11

https://doi.org/10.1145/3395035.3425248

Published: 27 December 2020 Publication History

Abstract

This study investigates the utility of Long Short-Term Memory (LSTM) networks for modelling spatial-temporal patterns for micro-expression recognition (MER). Micro-expressions are involuntary, short facial expressions, often of low intensity. RNNs have attracted a lot of attention in recent years for modelling temporal sequences. The RNN-LSTM combination to be highly effective results in many application areas. The proposed method combines the recent VGGFace2 model, basically a ResNet-50 CNN trained on the VGGFace2 dataset, with uni-directional and bi-directional LSTM to explore different ways modelling spatial-temporal facial patterns for MER. The Grad-CAM heat map visualisation is used in the training stages to determine the most appropriate layer of the VGGFace2 model for retraining. Experiments are conducted with pure VGGFace2, VGGFace2 + uni-directional LSTM, and VGGFace2 + Bi-directional LSTM on the SMIC database using 5-fold cross-validation.

References

[1]

Michael W Browne. 2000. Cross-Validation Methods. Journal of Mathematical Psychology 44, 1 (mar 2000), 108--132. https://doi.org/10.1006/jmps.1999.1279

Digital Library

[2]

Q. Cao, L. Shen, W. Xie, O.M. Parkhi, and A. Zisserman. 2018. VGGFace2: A Dataset for Recognising Faces across Pose and Age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). IEEE. https: //doi.org/10.1109/fg.2018.00020

Digital Library

[3]

Jeff Donahue, Lisa Anne Hendricks, Marcus Rohrbach, Subhashini Venugopalan, Sergio Guadarrama, Kate Saenko, and Trevor Darrell. [n.d.]. Long-term Recurrent Convolutional Networks for Visual Recognition and Description. ([n. d.]). arXiv:http://arxiv.org/abs/1411.4389v4 [cs.CV]

Digital Library

[4]

P. Ekman and W.V. Friesen. 1969. Nonverbal Leakage and Clues to Deception. Psychiatry 32, 1 (1969), 88--106. https://doi.org/10.1080/00332747.1969.11023575 27785970.

[5]

Malgorzata. Frank, Mark.Herbasz. 2009. I see how you feel: training laypeople and professionals to recognize fleeting emotions. In the annual meeting of the International Communication Association. Annual Meeting of the International Communication Association, New York City, NY.

[6]

K. He, X. Zhang, S. Ren, and J. Sun. 2016. Deep Residual Learning for Image Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).

[7]

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation 9, 8 (nov 1997), 1735--1780. https://doi.org/10.1162/neco. 1997.9.8.1735

Digital Library

[8]

Xiaohua Huang, Su-JingWang, Guoying Zhao, and Matti Piteikainen. 2015. Facial Micro-Expression Recognition Using Spatiotemporal Local Binary Pattern with Integral Projection. In 2015 IEEE International Conference on Computer Vision Workshop (ICCVW). IEEE. https://doi.org/10.1109/iccvw.2015.10

Digital Library

[9]

Xiaohua Huang, Guoying Zhao, Xiaopeng Hong, Wenming Zheng, and Matti Pietikäinen. 2016. Spontaneous facial micro-expression analysis using Spatiotemporal Completed Local Quantized Patterns. Neurocomputing 175 (jan 2016), 564--578. https://doi.org/10.1016/j.neucom.2015.10.096

Digital Library

[10]

Zhiheng Huang, Wei Xu, and Kai Yu. [n.d.]. Bidirectional LSTM-CRF Models for Sequence Tagging. ([n. d.]). arXiv:http://arxiv.org/abs/1508.01991v1 [cs.CL]

[11]

Huai-Qian Khor, John See, Raphael C. W. Phan, and Weiyao Lin. [n.d.]. Enriched Long-term Recurrent Convolutional Network for Facial Micro-Expression Recognition. ([n. d.]). arXiv:http://arxiv.org/abs/1805.08417v1 [cs.CV]

[12]

Dae Hoe Kim, Wissam J. Baddar, and Yong Man Ro. 2016. Micro-Expression Recognition with Expression-State Constrained Spatio-Temporal Feature Representations. In Proceedings of the 2016 ACM on Multimedia Conference - MM '16. ACM Press. https://doi.org/10.1145/2964284.2967247

Digital Library

[13]

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q.Weinberger (Eds.). Curran Associates, Inc., 1097--1105. http://papers.nips.cc/paper/ 4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf

Digital Library

[14]

Jing Li, YandanWang, John See, andWenbin Liu. 2018. Micro-expression recognition based on 3D flow convolutional neural network. Pattern Analysis and Applications 22, 4 (nov 2018), 1331--1339. https://doi.org/10.1007/s10044-018-0757--5

[15]

Q. Li, S. Zhan, L. Xu, and C. Wu. 2018. Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow. Multimedia Tools and Applications 78, 20 (Dec. 2018), 29307--29322. https://doi.org/10.1007/s11042-018- 6857--9

[16]

Si Miao, Haoyu Xu, Zhenqi Han, and Yongxin Zhu. 2019. Recognizing Facial Expressions Using a Shallow Convolutional Neural Network. IEEE Access 7 (2019), 78000--78011. https://doi.org/10.1109/access.2019.2921220

[17]

Anh Cat Le Ngo, Yee-Hui Oh, Raphael C.-W. Phan, and John See. 2016. Eulerian emotion magnification for subtle expression recognition. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE. https://doi.org/10.1109/icassp.2016.7471875

Digital Library

[18]

Wei Peng, Xiaopeng Hong, Yingyue Xu, and Guoying Zhao. 2019. A Boost in Revealing Subtle Facial Expressions: A Consolidated Eulerian Framework. In 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019). IEEE. https://doi.org/10.1109/fg.2019.8756541

Digital Library

[19]

T. Pfister, X. Li, G. Zhao, and M. Pietikainen. 2011. Recognising Spontaneous Facial Micro-expressions. IEEE International Conference on Computer Vision (6 2011), 1449--1456. https://doi.org/10.1109/ICCV.2011.6126401

Digital Library

[20]

Sai Prasanna Teja Reddy, Surya Teja Karri, Shiv Ram Dubey, and Snehasis Mukherjee. 2019. Spontaneous Facial Micro-Expression Recognition using 3D Spatiotemporal Convolutional Neural Networks. arXiv:1904.01390 [cs.CV]

[21]

John A. Ruiz-Hernandez and Matti Pietikainen. 2013. Encoding Local Binary Patterns using the re-parametrization of the second order Gaussian jet. In 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG). IEEE. https://doi.org/10.1109/fg.2013.6553709

[22]

R.R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. 2017. Grad-CAM: Visual Explanations From Deep Networks via Gradient-Based Localization. In IEEE Int. Conf. on Computer Vision (ICCV).

[23]

Leslie N. Smith. [n.d.]. A disciplined approach to neural network hyperparameters: Part 1 -- learning rate, batch size, momentum, and weight decay. ([n. d.]). arXiv:http://arxiv.org/abs/1803.09820v2 [cs.LG]

[24]

M.A. Takalkar and M. Xu. 2017. Image Based Facial Micro-Expression Recognition Using Deep Learning on Small Datasets. In 2017 International Conference on Digital Image Computing: Techniques and Applications (DICTA). IEEE. https: //doi.org/10.1109/dicta.2017.8227443

[25]

Amin Ullah, Jamil Ahmad, Khan Muhammad, Muhammad Sajjad, and SungWook Baik. 2018. Action Recognition in Video Sequences using Deep Bi-Directional LSTM With CNN Features. IEEE Access 6 (2018), 1155--1166. https://doi.org/10. 1109/access.2017.2778011

[26]

Michiel Verburg and Vlado Menkovski. [n.d.]. Micro-expression detection in long videos using optical flow and recurrent neural networks. ([n. d.]). arXiv:http://arxiv.org/abs/1903.10765v1 [cs.CV]

[27]

Yandan Wang, John See, Raphael C.-W. Phan, and Yee-Hui Oh. 2015. Efficient Spatio-Temporal Local Binary Patterns for Spontaneous Facial Micro-Expression Recognition. PLOS ONE 10, 5 (may 2015), e0124674. https://doi.org/10.1371/ journal.pone.0124674

[28]

Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, and William Freeman. 2012. Eulerian video magnification for revealing subtle changes in the world. ACM Transactions on Graphics 31, 4 (jul 2012), 1--8. https://doi.org/ 10.1145/2185520.2185561

Digital Library

[29]

Z. Xia, X. Hong, X. Gao, X. Feng, and G. Zhao. 2019. Spatiotemporal Recurrent Convolutional Networks for Recognizing Spontaneous Micro-expressions. IEEE Transactions on Multimedia (2019), 1--1. https://doi.org/10.1109/tmm.2019. 2931351

[30]

Wen-Jing Yan, Qi Wu, Jing Liang, Yu-Hsin Chen, and Xiaolan Fu. 2013. How Fast are the Leaked Facial Expressions: The Duration of Micro-Expressions. Journal of Nonverbal Behavior 37, 4 (July 2013), 217--230. https://doi.org/10.1007/s10919- 013-0159--8

[31]

J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3320--3328.

Digital Library

[32]

Yuan Zong, Wenming Zheng, Xiaopeng Hong, Chuangao Tang, Zhen Cui, and Guoying Zhao. 2019. Cross-Database Micro-Expression Recognition. In Proceedings of the 2019 on International Conference on Multimedia Retrieval - ICMR '19. ACM Press. https://doi.org/10.1145/3323873.3326590

Digital Library

Cited By

Gomez LMorales AFierrez JOrozco-Arroyave J(2023)Exploring facial expressions and action unit domains for Parkinson detectionPLOS ONE10.1371/journal.pone.028124818:2(e0281248)Online publication date: 2-Feb-2023
https://doi.org/10.1371/journal.pone.0281248
Mohana MSubashini PKrishnaveni M(2023)Emotion Recognition from Facial Expression Using Hybrid CNN–LSTM NetworkInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142356008637:08Online publication date: 7-Jul-2023
https://doi.org/10.1142/S0218001423560086
Zhao GLi XLi YPietikäinen M(2023)Facial Micro-Expressions: An OverviewProceedings of the IEEE10.1109/JPROC.2023.3275192111:10(1215-1235)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3275192
Show More Cited By

Index Terms

Investigating LSTM for Micro-Expression Recognition
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Machine learning
    1. Learning paradigms
    2. Machine learning approaches
      1. Neural networks
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction techniques
      1. Gestural input

Index terms have been assigned to the content through auto-classification.

Recommendations

Detection of Micro-expression Recognition Based on Spatio-Temporal Modelling and Spatial Attention
ICMI '20: Proceedings of the 2020 International Conference on Multimodal Interaction

My PhD project aims to make contributions in the affective computing application to assist in the depression diagnosis by micro-expression recognition. My motivation is the similarities of the low-intensity facial expressions in micro-expressions and ...
Facial micro-expression recognition based on the fusion of deep learning and enhanced optical flow
Abstract
Micro-expression is a kind of split-second subtle expression which could not be controlled by the autonomic nervous system. Micro-expression indicates that a person is hiding his truly emotion consciously. Because the micro-expression is closely ...
OFF-ApexNet on micro-expression recognition system
Abstract
When a person attempts to conceal an emotion, the genuine emotion is manifest as a micro-expression. Exploration of automatic facial micro-expression recognition systems is relatively new in the computer vision domain. This is due to ...
Highlights
- Only two frames from each video were utilized to represent significant motion features.

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '20 Companion: Companion Publication of the 2020 International Conference on Multimodal Interaction

October 2020

548 pages

ISBN:9781450380027

DOI:10.1145/3395035

General Chairs:
Khiet Truong
University of Twente, the Netherlands
,
Dirk Heylen
University of Twente, the Netherlands
,
Mary Czerwinski
Microsoft Research, USA
,
Program Chairs:
Nadia Berthouze
University College London, United Kingdom
,
Mohamed Chetouani
Sorbonne University, France
,
Mikio Nakano
C4A Research Institute, Japan

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 December 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ICMI '20

Sponsor:

SIGCHI

ICMI '20: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

October 25 - 29, 2020

Virtual Event, Netherlands

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

9
Total Citations
View Citations
271
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)4

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gomez LMorales AFierrez JOrozco-Arroyave J(2023)Exploring facial expressions and action unit domains for Parkinson detectionPLOS ONE10.1371/journal.pone.028124818:2(e0281248)Online publication date: 2-Feb-2023
https://doi.org/10.1371/journal.pone.0281248
Mohana MSubashini PKrishnaveni M(2023)Emotion Recognition from Facial Expression Using Hybrid CNN–LSTM NetworkInternational Journal of Pattern Recognition and Artificial Intelligence10.1142/S021800142356008637:08Online publication date: 7-Jul-2023
https://doi.org/10.1142/S0218001423560086
Zhao GLi XLi YPietikäinen M(2023)Facial Micro-Expressions: An OverviewProceedings of the IEEE10.1109/JPROC.2023.3275192111:10(1215-1235)Online publication date: Oct-2023
https://doi.org/10.1109/JPROC.2023.3275192
Shang ZLiu JLi X(2023)Micro-Expression Recognition Based on Spatio–Temporal Capsule NetworkIEEE Access10.1109/ACCESS.2023.324287111(13704-13713)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3242871
Bisogni CCimmino LDe Marsico MHao FNarducci F(2023)Emotion recognition at a distance: The robustness of machine learning based on hand-crafted facial features vs deep learning modelsImage and Vision Computing10.1016/j.imavis.2023.104724136(104724)Online publication date: Aug-2023
https://doi.org/10.1016/j.imavis.2023.104724
Shang ZWang PLi X(2023)Micro-expression recognition based on differential feature fusionMultimedia Tools and Applications10.1007/s11042-023-15626-083:4(11111-11126)Online publication date: 27-Jun-2023
https://doi.org/10.1007/s11042-023-15626-0
Li YWei JLiu YKauttonen JZhao G(2022)Deep Learning for Micro-Expression Recognition: A SurveyIEEE Transactions on Affective Computing10.1109/TAFFC.2022.320517013:4(2028-2046)Online publication date: 1-Oct-2022
https://doi.org/10.1109/TAFFC.2022.3205170
Mohammad FAhmed MKim Y(2021)Efficient Energy Management Based on Convolutional Long Short-Term Memory Network for Smart Power Distribution SystemEnergies10.3390/en1419616114:19(6161)Online publication date: 27-Sep-2021
https://doi.org/10.3390/en14196161
Gómez-Gómez LMorales AOrozco-Arroyave JFierrez J(undefined)Exploring Facial Expressions and Action Unit Domains For Parkinson DetectionSSRN Electronic Journal10.2139/ssrn.4069648
https://doi.org/10.2139/ssrn.4069648

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten