More Web Proxy on the site http://driver.im/

short-paper

Video-based emotion recognition using CNN-RNN and C3D hybrid networks

Authors:

Yuanliu LiuAuthors Info & Claims

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

Pages 445 - 450

https://doi.org/10.1145/2993148.2997632

Published: 31 October 2016 Publication History

Abstract

In this paper, we present a video-based emotion recognition system submitted to the EmotiW 2016 Challenge. The core module of this system is a hybrid network that combines recurrent neural network (RNN) and 3D convolutional networks (C3D) in a late-fusion fashion. RNN and C3D encode appearance and motion information in different ways. Specifically, RNN takes appearance features extracted by convolutional neural network (CNN) over individual video frames as input and encodes motion later, while C3D models appearance and motion of video simultaneously. Combined with an audio module, our system achieved a recognition accuracy of 59.02% without using any additional emotion-labeled video clips in training set, compared to 53.8% of the winner of EmotiW 2015. Extensive experiments show that combining RNN and C3D together can improve video-based emotion recognition noticeably.

References

[1]

Dhall, A., Goecke, R., Lucey, S. and Gedeon, T. 2012. Collecting large, richly annotated facial-expression databases from movies. IEEE Multimedia.

Digital Library

[2]

Dhall, A., Goecke, R. and Gedeon, T. 2015. Automatic Group Happiness Intensity Analysis. IEEE Transaction on Affective Computing.

Digital Library

[3]

Yao, A., Shao, J., Ma, N. and Chen,Y. 2015. Capturing AUAware Facial Features and Their Latent Relations for Emotion Recognition in the Wild. ACM ICMI.

Digital Library

[4]

Liu, M., Wang, R., Li, S., Shan, S., Huang Z. and Chen, X.2014. Combining Multiple Kernel Methods on Riemannian Manifold for Emotion Recognition in the Wild. ACM ICMI.

Digital Library

[5]

Ebrahimi Kahou, S., Michalski, V., Konda, K., Memisevic, R., and Pal, C. 2015. Recurrent neural networks for emotion recognition in video. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 467- 474. ACM.

Digital Library

[6]

Tran, D., Bourdev, L., Fergus, R., Torresani, L., & Paluri, M. 2015.

[7]

Learning spatiotemporal features with 3d convolutional networks. In 2015 IEEE International Conference on Computer Vision (ICCV) .4489-4497. IEEE.

Digital Library

[8]

Eyben, F., Wöllmer, M., & Schuller, B. (2010, October). Opensmile: the munich versatile and fast open-source audio feature extractor. InProceedings of the 18th ACM international conference on Multimedia. 1459-1462. ACM.

Digital Library

[9]

Donahue, J., Anne Hendricks, L., Guadarrama, S., Rohrbach, M., Venugopalan, S., Saenko, K., & Darrell, T. 2015. Longterm recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.2625-2634.

[10]

Kahou, S. E., Pal, C., Bouthillier, X., Froumenty, P., Gülçehre, Ç, Memisevic, R. and Mirza, M. 2013. Combining modality specific deep neural networks for emotion recognition in video. In Proceedings of the 15th ACM on International conference on multimodal interaction. 543-550. ACM.

Digital Library

[11]

Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In ACM MM.

Digital Library

[12]

Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D. and Rabinovich, A. 2015. Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.1-9.

[13]

He, K., Zhang, X., Ren, S. and Sun, J. 2015. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385.

[14]

Simonyan, K., & Zisserman, A. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.

[15]

Parkhi, O. M., Vedaldi, A., & Zisserman, A. 2015. Deep face recognition. In British Machine Vision Conference (Vol. 1, No. 3, p. 6).

[16]

Deng, J., Dong, W., Socher, R., Li, L. J., Li, K. and Li, F.F., L. 2009. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition. CVPR. 248- 255. IEEE.

[17]

Carrier, P. L., Courville, A., Goodfellow, I. J., Mirza, M. and Bengio, Y. 2013 .FER-2013 face database. Technical report, 1365, Université de Montréal.

[18]

Graves, A., Liwicki, M., Fernández, S., Bertolami, R., Bunke, H. and Schmidhuber, J. 2009. A novel connectionist system for unconstrained handwriting recognition. IEEE transactions on pattern analysis and machine intelligence, 31(5), 855-868.

Digital Library

[19]

Sak, H., Senior, A. W. and Beaufays, F. 2014. Long shortterm memory recurrent neural network architectures for large scale acoustic modeling. In INTERSPEECH. 338-342.

[20]

Kim, B. K., Dong, S. Y., Roh, J., Kim, G. and Lee, S. Y. 2016. Fusing Aligned and Non-Aligned Face Information for Automatic Affect Recognition in the Wild: A Deep Learning Approach. In Computer Vision and Pattern Recognition. CVPR.

[21]

Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R. and Li, F.F. 2014. Large-scale Video Classification with Convolutional Neural Networks.

[22]

Ng, J., Hausknecht, M., Vijayanarasimhan S., Monga R., Vinyals O., Toderici G.2015. Beyond Short Snippets: Deep Networks for Video Classification. In Computer Vision and Pattern Recognition. CVPR. 4694-4702. IEEE.

[23]

Sharma S., Kiros R., Salakhutdinov R.2016 Action Recognition using Visual Attention. Workshop track - ICLR.

[24]

Kaya, H., Gürpinar, F., Afshar, S. and Salah, A. A. 2015. Contrasting and Combining Least Squares Based Learners for Emotion Recognition in the Wild. In Proceedings of the 2015 ACM on International Conference on Multimodal Interaction. 459-466. ACM.

Digital Library

[25]

Venugopalan, S., Rohrbach, M., Donahue, J., Mooney, R., Darrell, T.m. and Saenko, K. 2015. Sequence to sequencevideo to text. In Proceedings of the IEEE International Conference on Computer Vision. 4534-4542.

Digital Library

[26]

Pan, P., Xu, Z., Yang, Y., Wu, F. and Zhuang, Y. 2015.

[27]

Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning. arXiv preprint arXiv:1511.03476.

[28]

Graves, A., Mohamed, A. R. and Hinton, G. 2013. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing. 6645-6649. IEEE.

[29]

Dhall, A., Goecke, R., Joshi, J., Hoey, J. and Gedeon, T. 2016. EmotiW 2016: Video and Group-level Emotion Recognition Challenges, ACM ICMI 2016.

Digital Library

[30]

Jianguo L., Tao W., Yimin Z. 2011. ICCV: Face Detection using SURF Cascade. In Computer Vision Workshops.

[31]

Fernández, S., Graves, A., Schmidhuber, J. 2007. An application of recurrent neural networks to discriminative keyword spotting. In International Conference on Artificial Neural Networks. 220-229. Springer Berlin Heidelberg

Digital Library

Cited By

Kopalidis TSolachidis VVretos NDaras P(2024)Advances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and DatasetsInformation10.3390/info1503013515:3(135)Online publication date: 28-Feb-2024
https://doi.org/10.3390/info15030135
Mao MLee AHong M(2024)Deep Learning Innovations in Video Classification: A Survey on Techniques and Dataset EvaluationsElectronics10.3390/electronics1314273213:14(2732)Online publication date: 11-Jul-2024
https://doi.org/10.3390/electronics13142732
Zhang YZhang FZhou YXu X(2024)ACA-Net: adaptive context-aware network for basketball action recognitionFrontiers in Neurorobotics10.3389/fnbot.2024.147132718Online publication date: 25-Sep-2024
https://doi.org/10.3389/fnbot.2024.1471327
Show More Cited By

Index Terms

Video-based emotion recognition using CNN-RNN and C3D hybrid networks
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Activity recognition and understanding

Recommendations

Recurrent Neural Networks for Emotion Recognition in Video
ICMI '15: Proceedings of the 2015 ACM on International Conference on Multimodal Interaction

Deep learning based approaches to facial analysis and video analysis have recently demonstrated high performance on a variety of key tasks such as face recognition, emotion recognition and activity recognition. In the case of video, information often ...
Multi-Feature Based Emotion Recognition for Video Clips
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

In this paper, we present our latest progress in Emotion Recognition techniques, which combines acoustic features and facial features in both non-temporal and temporal mode. This paper presents the details of our techniques used in the Audio-Video ...
Video-based Emotion Recognition Using Deeply-Supervised Neural Networks
ICMI '18: Proceedings of the 20th ACM International Conference on Multimodal Interaction

Emotion recognition (ER) based on natural facial images/videos has been studied for some years and considered a comparatively hot topic in the field of affective computing. However, it remains a challenge to perform ER in the wild, given the noises ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

ICMI '16: Proceedings of the 18th ACM International Conference on Multimodal Interaction

October 2016

605 pages

ISBN:9781450345569

DOI:10.1145/2993148

General Chairs:
Yukiko I. Nakano
Seikei University, Japan
,
Elisabeth André
Augsburg University, Germany
,
Toyoaki Nishida
Kyoto University, Japan
,
Program Chairs:
Louis-Philippe Morency
Carnegie Mellon University, USA
,
Carlos Busso
University of Texas at Dallas, USA
,
Catherine Pelachaud
ISIR, France / University of Paris6, France

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Conference

ICMI '16

Sponsor:

SIGCHI

ICMI '16: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 12 - 16, 2016

Tokyo, Japan

Acceptance Rates

Overall Acceptance Rate 453 of 1,080 submissions, 42%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

391
Total Citations
View Citations
5,144
Total Downloads

Downloads (Last 12 months)179
Downloads (Last 6 weeks)25

Reflects downloads up to 13 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Kopalidis TSolachidis VVretos NDaras P(2024)Advances in Facial Expression Recognition: A Survey of Methods, Benchmarks, Models, and DatasetsInformation10.3390/info1503013515:3(135)Online publication date: 28-Feb-2024
https://doi.org/10.3390/info15030135
Mao MLee AHong M(2024)Deep Learning Innovations in Video Classification: A Survey on Techniques and Dataset EvaluationsElectronics10.3390/electronics1314273213:14(2732)Online publication date: 11-Jul-2024
https://doi.org/10.3390/electronics13142732
Zhang YZhang FZhou YXu X(2024)ACA-Net: adaptive context-aware network for basketball action recognitionFrontiers in Neurorobotics10.3389/fnbot.2024.147132718Online publication date: 25-Sep-2024
https://doi.org/10.3389/fnbot.2024.1471327
Kulaglic AÖrpek ZKayı BOzmen S(2024)Analysis of Emotions and Movements of Asian and European Facial ExpressionsAdvances in Science, Technology and Engineering Systems Journal10.25046/aj0901059:1(42-48)Online publication date: Jan-2024
https://doi.org/10.25046/aj090105
Bazangani OEliasi PPicek SBatina L(2024)Can Machine Learn Pipeline Leakage?2024 Design, Automation & Test in Europe Conference & Exhibition (DATE)10.23919/DATE58400.2024.10546629(1-6)Online publication date: 25-Mar-2024
https://doi.org/10.23919/DATE58400.2024.10546629
Li LZheng YLiu SXu XLi TCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Domain Knowledge Enhanced Vision-Language Pretrained Model for Dynamic Facial Expression RecognitionProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681708(5673-5682)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681708
Chen HHuang HDong JZheng MShao DCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)FineCLIPER: Multi-modal Fine-grained CLIP for Dynamic Facial Expression Recognition with AdaptERsProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680827(2301-2310)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680827
Mehta SSharma ADogra AHariharan S(2024)EmotiveNet: Five-Dimensional Emotion Detection with CNN-SVM Fusion2024 2nd World Conference on Communication & Computing (WCONF)10.1109/WCONF61366.2024.10692180(1-6)Online publication date: 12-Jul-2024
https://doi.org/10.1109/WCONF61366.2024.10692180
Jelčicová ZKasapaki EAndersson OSparsø J(2024) PeakEngine : A Deterministic On-the-Fly Pruning Neural Network Accelerator for Hearing Instruments IEEE Transactions on Very Large Scale Integration (VLSI) Systems10.1109/TVLSI.2023.330091032:1(150-163)Online publication date: Jan-2024
https://doi.org/10.1109/TVLSI.2023.3300910
Liu HLou TZhang YWu YXiao YJensen CZhang D(2024)EEG-Based Multimodal Emotion Recognition: A Machine Learning PerspectiveIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.336913073(1-29)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3369130
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents