[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Global and local fusion ensemble network for facial expression recognition

  • 1178: Pattern Recognition for Adaptive User Interfaces
  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Accurate and fast facial feature learning is vital for Facial Expression Recognition (FER). Recent researches have proved that ensemble methods can perform efficiently and effectively on the FER, whereas these methods still confront the issues: incomplete information extraction of facial images and weak robustness on large-scale datasets. In this paper, we propose an efficient global and local perception ensemble network with attention units to tackle the above issues. The overall ensemble module has two components: an efficient ensemble and a locality extraction module for perceiving global information and local details simultaneously. The locality extraction module is proposed to attend to local details from facial regions of interest (ROIs). Furthermore, global and local information is fused by our attention units at the decision-level, which enhances the robustness of the network. The conducted experiments validate the effectiveness and efficiency of our method on diverse benchmark datasets. The results demonstrate that our network not only achieves real-time performance but also outperforms state-of-the-art methods on the in-the-wild facial expression datasets.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

References

  1. Arora M, Kumar M (2021) Autofer: Pca and pso based automatic facial emotion recognition. Multimed Tools Appl 80(2):3039–3049

    Article  Google Scholar 

  2. Barsoum E, Zhang C, Ferrer CC, Zhang Z (2016) Training deep networks for facial expression recognition with crowd-sourced label distribution. In: Proceedings of the 18th ACM international conference on multimodal interaction. ICMI ’16. Association for Computing Machinery, New York, pp 279–283, DOI https://doi.org/10.1145/2993148.2993165, (to appear in print)

  3. Chen L, Yang X, Jeon G, Anisetti M, Liu K (2020) A trusted medical image super-resolution method based on feedback adaptive weighted dense network. Artif Intell Med 106:101857

    Article  Google Scholar 

  4. Chen L, Tang R, Anisetti M, Yang X (2021) A lightweight iterative error reconstruction network for infrared image super-resolution in smart grid. Sustain Cities Soc 66:102520

    Article  Google Scholar 

  5. Cohn J, Zlochower A (1995) A computerized analysis of facial expression: feasibility of automated discrimination. American Psychological Society 2(6)

  6. Dalal N, Triggs B (2005) Histograms of oriented gradients for human detection. In: 2005 IEEE Computer society conference on computer vision and pattern recognition (CVPR’05), vol 1, pp 886–893

  7. Dietterich TG (2000) Ensemble methods in machine learning. In: International workshop on multiple classifier systems. Springer, pp 1–15

  8. Ding H, Zhou SK, Chellappa R (2017) Facenet2expnet: regularizing a deep face recognition net for expression recognition. In: 2017 12th IEEE international conference on automatic face gesture recognition (FG 2017), pp 118–126

  9. Fasel B (2002) Head-pose invariant facial expression recognition using convolutional neural networks. In: Proceedings. Fourth IEEE international conference on multimodal interfaces. IEEE, pp 529–534

  10. Goodfellow IJ, Erhan D, Carrier PL, Courville A, Mirza M, Hamner B, Cukierski W, Tang Y, Thaler D, Lee DH et al (2013) Challenges in representation learning: a report on three machine learning contests. In: International conference on neural information processing. Springer, pp 117–124

  11. Hewitt C, Gunes H (2018) Cnn-based facial affect analysis on mobile devices. arXiv:http://arxiv.org/abs/180708775

  12. Hu J, Shen L, Sun G (2018) Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7132–7141

  13. Huang C (2017) Combining convolutional neural networks for emotion recognition. In: IEEE MIT undergraduate research technology conference (URTC), pp 1–4

  14. Huang X, Zhao G, Zheng W, Pietikinen M (2012) Towards a dynamic expression recognition system under facial occlusion. Pattern Recognit Lett 33(16):2181–2191

    Article  Google Scholar 

  15. Jain N, Kumar S, Kumar A, Shamsolmoali P, Zareapoor M (2018) Hybrid deep neural networks for face emotion recognition. Pattern Recogn Lett 115:101–106

    Article  Google Scholar 

  16. Jyoti S, Sharma G, Dhall A (2019) Expression empowered residen network for facial action unit detection. In: 2019 14th IEEE international conference on automatic face & gesture recognition (FG 2019). IEEE, pp 1–8

  17. Kandeel AA, Abbas HM, Hassanein HS (2021) Explainable model selection of a convolutional neural network for driver’s facial emotion identification. In: International conference on pattern recognition. Springer, pp 699–713

  18. Kim BK, Lee H, Roh J, Lee SY (2015) Hierarchical committee of deep cnns with exponentially-weighted decision fusion for static facial expression recognition. In: Proceedings of the 2015 ACM on international conference on multimodal interaction, pp 427–434

  19. Kim BK, Roh J, Dong SY, Lee SY (2016) Hierarchical committee of deep convolutional neural networks for robust facial expression recognition. J Multimodal User Interfaces 10(2):173–189

    Article  Google Scholar 

  20. Li S, Deng W (2020) Deep facial expression recognition: a survey. IEEE Trans Affect Comput

  21. Li S, Deng W, Du J (2017) Reliable crowdsourcing and deep locality-preserving learning for expression recognition in the wild. In: 2017 IEEE Conference on computer vision and pattern recognition (CVPR). IEEE, pp 2584–2593

  22. Li M, Xu H, Huang X, Song Z, Liu X, Li X (2018) Facial expression recognition with identity and emotion joint learning. IEEE Trans Affect Comput

  23. Li Y, Zeng J, Shan S, Chen X (2019) Occlusion aware facial expression recognition using cnn with attention mechanism. IEEE Trans Image Process 28(5):2439–2450

    Article  Google Scholar 

  24. Liu M, Wang R, Li S, Shan S, Huang Z, Chen X (2014) Combining multiple kernel methods on riemannian manifold for emotion recognition in the wild. In: Proceedings of the 16th international conference on multimodal interaction, pp 494–501

  25. Lowe DG (1999) Object recognition from local scale-invariant features. In: Proceedings of the seventh IEEE international conference on computer vision, vol 2. IEEE, pp 1150–1157

  26. Lucey P, Cohn JF, Kanade T, Saragih J, Ambadar Z, Matthews I (2010) The extended cohn-kanade dataset (ck+): acomplete dataset for action unit and emotion-specified expression. In: 2010 IEEE computer society conference on computer vision and pattern recognition-workshops. IEEE, pp 94–101

  27. Lyons M, Akamatsu S, Kamachi M, Gyoba J (1998) Coding facial expressions with gabor wavelets. In: Proceedings third IEEE international conference on automatic face and gesture recognition. IEEE, pp 200–205

  28. Mahmood MR (2021) Two feature selection methods comparison chi-square and relief-f for facial expression recognition. In: J Phys: Conf Ser. IOP Publishing, vol 1804, p 012056

  29. Meena HK, Joshi SD, Sharma KK (2019) Facial expression recognition using graph signal processing on hog. IETE J Res 1–7

  30. Miao S, Xu H, Han Z, Zhu Y (2019) Recognizing facial expressions using a shallow convolutional neural network. IEEE Access 7:78000–78011

    Article  Google Scholar 

  31. Mollahosseini A, Chan D, Mahoor MH (2016) Going deeper in facial expression recognition using deep neural networks. In: 2016 IEEE Winter conference on applications of computer vision (WACV), pp 1–10

  32. Mollahosseini A, Hasani B, Mahoor MH (2017) Affectnet: a database for facial expression, valence, and arousal computing in the wild. IEEE Trans Affect Comput 10(1):18–31

    Article  Google Scholar 

  33. Mousavi R, Eftekhari M (2015) A new ensemble learning methodology based on hybridization of classifier ensemble selection approaches. Appl Soft Comput 37:652–666

    Article  Google Scholar 

  34. Nair V, Hinton GE (2010) Rectified linear units improve restricted boltzmann machines. In: ICML

  35. Pan B, Wang S, Xia B (2019) Occluded facial expression recognition enhanced through privileged information. In: Proceedings of the 27th ACM international conference on multimedia, pp 566–573

  36. Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D (2017) Grad-cam: visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision, pp 618–626

  37. Shan C, Gong S, Mcowan PW (2009) Facial expression recognition based on local binary patterns: a comprehensive study. Image Vis Comput 27 (6):803–816

    Article  Google Scholar 

  38. Simcock G, McLoughlin LT, De Regt T, Broadhouse KM, Beaudequin D, Lagopoulos J, Hermens DF (2020) Associations between facial emotion recognition and mental health in early adolescence. Int J Environ Res Public Health 17(1):330

    Article  Google Scholar 

  39. Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv:http://arxiv.org/abs/14091556

  40. Siqueira H, Barros P, Magg S, Wermter S (2018a) An ensemble with shared representations based on convolutional networks for continually learning facial expressions. In: 2018 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, pp 1563–1568

  41. Siqueira H, Barros P, Magg S, Wermter S (2018b) An ensemble with shared representations based on convolutional networks for continually learning facial expressions. In: 2018 IEEE/RSJ International conference on intelligent robots and systems (IROS). IEEE, pp 1563–1568

  42. Siqueira H, Magg S, Wermter S (2020) Efficient facial feature learning with wide ensemble-based convolutional neural networks. In: Proceedings of the AAAI conference on artificial intelligence, vol 34, pp 5800–5809

  43. Tang B, He H (2015) Enn: extended nearest neighbor method for pattern recognition [research frontier]. IEEE Comput Intell Mag 10(3):52–60

    Article  Google Scholar 

  44. Tonguç G, Ozkara BO (2020) Automatic recognition of student emotions from facial expressions during a lecture. Comput Educ 148:103797

    Article  Google Scholar 

  45. Viola P, Jones M (2001) Rapid object detection using a boosted cascade of simple features. In: Proceedings of the 2001 IEEE computer society conference on computer vision and pattern recognition. CVPR 2001, vol 1. IEEE, pp I–I

  46. Wang K, Peng X, Yang J, Lu S, Qiao Y (2020) Suppressing uncertainties for large-scale facial expression recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp 6897–6906

  47. Wen G, Hou Z, Li H, Li D, Jiang L, Xun E (2017) Ensemble of deep neural networks with probability-based fusion for facial expression recognition. Cognit Comput 9(5):597–610

    Article  Google Scholar 

  48. Yaddaden Y, Adda M, Bouzouane A (2020) A study of dimensionality reduction for facial expression recognition. In: International conference on computing systems and applications. Springer, pp 14–24

  49. Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS (2018) Generative image inpainting with contextual attention. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5505–5514

  50. Zeng N, Zhang H, Song B, Liu W, Yurong D (2018) Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273:643–649

    Article  Google Scholar 

  51. Zhang T (2017) Facial expression recognition based on deep learning: a survey. In: International conference on intelligent and interactive systems and applications, Springer, pp 345–352

  52. Zhang J, Xiao N (2020) Capsule network-based facial expression recognition method for a humanoid robot. In: Recent trends in intelligent computing, communication and devices. Springer, pp 113–121

  53. Zhang T, Zheng W, Cui Z, Zong Y, Li Y (2018) Spatial–temporal recurrent neural network for emotion recognition. IEEE Trans Cybern 49 (3):839–847

    Article  Google Scholar 

  54. Zhao L, Li X, Zhuang Y, Wang J (2017) Deeply-learned part-aligned representations for person re-identification. In: Proceedings of the IEEE international conference on computer vision, pp 3219–3228

  55. Zhong L, Liu Q, Yang P, Liu B, Huang J, Metaxas DN (2012) Learning active facial patches for expression analysis. In: 2012 IEEE conference on computer vision and pattern recognition. IEEE, pp 2562–2569

Download references

Acknowledgements

The research in our paper is sponsored by Science Foundation of Sichuan Science and Technology Department 2021YFH0119 and the funding from Sichuan University under grant 2020SCUNG205

Author information

Authors and Affiliations

  1. College of Electronics and Information Engineering, Sichuan University, Chengdu, Sichuan, 610064, China

    Zheng He, Bin Meng, Lining Wang & Xiaomin Yang

  2. School of Electronic Engineering, Xidian University, Xi’an, 710071, China

    Gwanggil Jeon

  3. Department of Embedded Systems Engineering, Incheon National University, Incheon, 22012, Korea

    Gwanggil Jeon

  4. TAL Education Group, Beijing, China

    Zitao Liu

Authors
  1. Zheng He
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bin Meng
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Lining Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Gwanggil Jeon
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Zitao Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Xiaomin Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding authors

Correspondence to Gwanggil Jeon or Xiaomin Yang.

Ethics declarations

Conflict of interest

The authors have declared that no conflict of interests or competing interests exist.

Additional information

Publisher’s note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

He, Z., Meng, B., Wang, L. et al. Global and local fusion ensemble network for facial expression recognition. Multimed Tools Appl 82, 5473–5494 (2023). https://doi.org/10.1007/s11042-022-12321-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-022-12321-4

Keywords

Search

Navigation