[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

A novel modular deep fully convolutional network for efficient low resolution facial expression recognition

  • Original Research
  • Published:
Journal of Ambient Intelligence and Humanized Computing Aims and scope Submit manuscript

Abstract

Facial expression is one of the most important and natural way to express human feelings. Although deep convolutional neural networks have improved the performance of facial expression recognition (FER) systems, recognizing facial expressions from low resolution images is still a challenging task for real-time applications. A new modular deep fully convolutional neural network is designed to tackle this problem. The proposed network is composed of four modules namely, feature extraction (FE), residual spatial-channel attention (RSCA), atrous spatial pyramid pooling (ASPP), and classification module. The prominent facial regions relevant to facial expressions are extracted using FE module and then strengthened using RSCA and ASPP modules. Finally, a classification module using convolutional layers with adjusted stride parameter values is employed instead of fully connected layers to significantly reduce the number of learnable parameters. Experimental results using CK+, RAF-DB, and SFEW 2.0 datasets show that our proposed method achieves improved accuracies of 99.9%, 84.96%, and 53.0% at image resolutions of \(32\times 32\), \(48\times 48\), and \(26\times 26\), respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

The datasets used in the current study are available in the below links. RAF-DB: http://www.whdeng.cn/raf/model1.html. SFEW 2.0: https://cs.anu.edu.au/few/emotiw2015.html. CK+: https://www.pitt.edu/~emotion/ck-spread.htm. FER-2013: https://www.kaggle.com/c/challenges-in-representation-learning-facial-expression-recognition-challenge.

References

  • Ben X, Ren Y, Zhang J, Wang S-J, Kpalma K, Meng W, Liu Y-J (2021) Video-based facial micro-expression analysis: a survey of datasets, features and algorithms. IEEE Trans Pattern Anal Mach Intell. https://doi.org/10.1109/TPAMI.2021.3067464

    Article  Google Scholar 

  • Cai J, Meng Z, Khan AS, Li Z, O’Reilly J, Tong Y (2018) Island loss for learning discriminative features in facial expression recognition. In: 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018). IEEE, pp 302–309

  • Chen Y, Phonevilay V, Tao J, Chen X, Xia R, Zhang Q, Yang K, Xiong J, Xie J (2021) The face image super-resolution algorithm based on combined representation learning. Multim Tools Appl 80(20):30839–30861

    Article  Google Scholar 

  • Cheng B, Wang Z, Zhang Z, Li Z, Liu D, Yang J, Huang S, Huang TS (2017) Robust emotion recognition from low quality and low bit rate video: a deep learning approach. In: 2017 seventh international conference on affective computing and intelligent interaction (ACII). IEEE, pp 65–70

  • Chirra VRR, Uyyala SR, Kolli VKK (2021) Virtual facial expression recognition using deep cnn with ensemble learning. J Ambient Intell Humaniz Comput 12:10581–10599. https://doi.org/10.1007/s12652-020-02866-3

    Article  Google Scholar 

  • Chu W-S, De la Torre F, Cohn JF (2016) Selective transfer machine for personalized facial expression analysis. IEEE Trans Pattern Anal Mach Intell 39(3):529–545

    Article  Google Scholar 

  • Clevert D-A, Unterthiner T, Hochreiter S (2016) Fast and accurate deep network learning by exponential linear units (elus)

  • Dhall A, Ramana Murthy OV, Goecke R, Joshi J, Gedeon T (2015) Video and image based emotion recognition challenges in the wild: Emotiw 2015. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 423–426

  • Ding H, Zhou SK, Chellappa R (2017) Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, pp 118–126

  • Gogić I, Manhart M, Pandžić IS, Ahlberg J (2020) Fast facial expression recognition using local binary features and shallow neural networks. Vis Comput 36(1):97–112

    Article  Google Scholar 

  • Happy SL, Routray A (2014) Automatic facial expression recognition using features of salient facial patches. IEEE Trans Affect Comput 6(1):1–12

    Article  Google Scholar 

  • Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  • Jain DK, Shamsolmoali P, Sehdev P (2019) Extended deep neural network for facial emotion recognition. Pattern Recognit Lett 120:69–74. https://doi.org/10.1016/j.patrec.2019.01.008

    Article  Google Scholar 

  • Jyoti Kumari R, Rajesh R, Pooja KM (2015) Facial expression recognition: a survey. Procedia Comput Sci 58(1):486–491

  • Khan RA, Meyer A, Konik H, Bouakaz S (2013) Framework for reliable, real-time facial expression recognition for low resolution images. Pattern Recognit Lett 34(10):1159–1168

    Article  Google Scholar 

  • Li S, Deng W (2018) Reliable crowd sourcing and deep locality-preserving learning for unconstrained facial expression recognition. IEEE Trans Image Process 28(1):356–370

    Article  MATH  Google Scholar 

  • Li S, Deng W (2020) Deep facial expression recognition: a survey. IEEE Trans Affect Comput. ISSN: 1949-3045. https://doi.org/10.1109/TAFFC.2020.2981446

  • Li X, He M, Li H, Shen H (2021) A combined loss-based multiscale fully convolutional network for high-resolution remote sensing image change detection. IEEE Geosci Remote Sens Lett 19:1–5

    Google Scholar 

  • Liu Z, Li L, Wu Y, Zhang C (2020) Facial expression restoration based on improved graph convolutional networks. In: International conference on multimedia modeling. Springer, pp 527–539

  • Lopes AT, de Aguiar E, De Souza AF, Oliveira-Santos T (2017) Facial expression recognition with convolutional neural networks: coping with few data and the training sample order. Pattern Recognit 61:610–628

    Article  Google Scholar 

  • Lozano-Monasor E, López MT, Vigo-Bustos F, Fernández-Caballero A (2017) Facial expression recognition in ageing adults: from lab to ambient assisted living. J Ambient Intell Humaniz Comput 8(4):567–578

    Article  Google Scholar 

  • Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended Cohn-Kanade dataset (ck+): a complete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  • Luo J, Liu J, Lin J, Wang Z (2020) A lightweight face detector by integrating the convolutional neural network with the image pyramid. Pattern Recognit Lett 133:180–187

    Article  Google Scholar 

  • Ma Y, Wang X, Wei L (2021) Multi-level spatial and semantic enhancement network for expression recognition. Appl Intell 1–14

  • Uma Maheswari V, Varaprasad G, Viswanadha Raju S (2021) Local directional maximum edge patterns for facial expression recognition. J Ambient Intell Humaniz Comput 12(5):4775–4783

    Article  Google Scholar 

  • Mandal M, Verma M, Mathur S, Vipparthi SK, Murala S, Kumar DK (2019) Regional adaptive affinitive patterns (radap) with logical operators for facial expression recognition. IET Image Process 13(5):850–861

    Article  Google Scholar 

  • Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE winter conference on applications of computer vision (WACV). IEEE, pp 1–10

  • Nan F, Jin W, Tian F, Zhang J, Chao K-M, Hong Z, Zheng Q (2021) Feature super-resolution based facial expression recognition for multi-scale low-resolution images. Knowl Based Syst 107678

  • Połap D (2019) Analysis of skin marks through the use of intelligent things. IEEE Access 7:149355–149363

    Article  Google Scholar 

  • Rao T, Li J, Wang X, Sun Y, Chen H (2021) Facial expression recognition with multiscale graph convolutional networks. IEEE Multim 28(2):11–19

    Article  Google Scholar 

  • Saleem Sharmeen M, Abdullah A, Abdulazeez AM (2021) Facial expression recognition based on deep learning convolution neural network: a review. J Soft Comput Data Min 2(1):53–65

    Google Scholar 

  • Sariyanidi E, Gunes H, Cavallaro A (2014) Automatic analysis of facial affect: a survey of registration, representation, and recognition. IEEE Trans Pattern Anal Mach Intell 37(6):1113–1133

    Article  Google Scholar 

  • Shao J, Cheng Q (2021) E-fcnn for tiny facial expression recognition. Appl Intell 51(1):549–559

    Article  Google Scholar 

  • Shao J, Qian Y (2019) Three convolutional neural network models for facial expression recognition in the wild. Neurocomputing 355:82–92

    Article  Google Scholar 

  • Sikkandar H, Thiyagarajan R (2021) Deep learning based facial expression recognition using improved cat swarm optimization. J Ambient Intell Humaniz Comput 12(2):3037–3053

    Article  Google Scholar 

  • Umer S, Rout RK, Pero C, Nappi M (2021) Facial expression recognition with trade-offs between data augmentation and deep learning features. J Ambient Intell Humaniz Comput. https://doi.org/10.1007/s12652-020-02845-8

    Article  Google Scholar 

  • Whitehill J, Serpell Z, Lin Y-C, Foster A, Movellan JR (2014) The faces of engagement: automatic recognition of student engagement from facial expressions. IEEE Trans Affect Comput 5(1):86–98. https://doi.org/10.1109/TAFFC.2014.2316163

    Article  Google Scholar 

  • Yan Y, Zhang Z, Chen S, Wang H (2020) Low-resolution facial expression recognition: a filter learning perspective. Signal Process 169:107370

    Article  Google Scholar 

  • Zhang K, Huang Y, Yong D, Wang L (2017) Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans Image Process 26(9):4193–4203

    Article  MathSciNet  MATH  Google Scholar 

  • Zhang K, Zhang Z, Li Z, Qiao Yu (2016) Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Process Lett 23(10):1499–1503

    Article  Google Scholar 

  • Zhao R, Gan Q, Wang S, Ji Q (2016) Facial expression intensity estimation using ordinal information. In: Proceedings of the IEEE conference on computer vision and pattern recognition. IEEE, pp 3466–3474

Download references

Acknowledgements

Walaa Aly would like to thank the Deanship of Scientific Research at Majmaah University for supporting this work under Project No. R-2023-103.

Author information

Authors and Affiliations

  1. Department of Information Technology, College of Computer and Information Sciences, Majmaah University, Al-Majmaah, 11952, Saudi Arabia

    Walaa Aly & Saleh Aly

  2. Department of Electrical Engineering, Faculty of Engineering, Aswan University, Aswan, 81542, Egypt

    Walaa Aly & Saleh Aly

  3. Department of Natural and Applied Sciences, Community College, Majmaah University, Al-Majmaah, 11952, Saudi Arabia

    Ahmed I. Shahin

Authors
  1. Walaa Aly
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ahmed I. Shahin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Saleh Aly
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Saleh Aly.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Aly, W., Shahin, A.I. & Aly, S. A novel modular deep fully convolutional network for efficient low resolution facial expression recognition. J Ambient Intell Human Comput 14, 7747–7759 (2023). https://doi.org/10.1007/s12652-023-04586-w

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s12652-023-04586-w

Keywords

Search

Navigation