[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The limitation of the small-scale expression samples generally causes the performance degradation for facial expression recognition-based methods. Also, the correlation between different expression is always ignored when performing feature extraction process. Given above, we propose a novel approach that develops multi-class differentiation feature representation guided joint dictionary learning for FER. The proposed approach mainly includes two steps: firstly, we construct multi-class differentiation feature dictionaries corresponding to different expressions of training samples, aiming to enlarge inter-expression distance to mitigate the problem of nonlinear distribution in training samples. Secondly, we joint learn the multiple feature dictionaries by optimizing the resolutions of each feature dictionary, aiming to establish the strong relationship and enhance the representation ability among multiple feature dictionaries. To sum up, the proposed approach has more discriminative ability from the representation perspective. Comprehensive experiments carried out using three public datasets, including JAFFE, CK+ , and KDEF datasets, demonstrate that the proposed approach has strong performance for small-scale samples compared to several state-of-the-art methods. Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig.8
Fig.9

Similar content being viewed by others

Availability of data and materials

Data can be submitted on reasonable request.

References

  1. Ma, T.S., Tian, W.H., Xie, Y.L.: Multi-level knowledge distillation for low-resolution object detection and facial expression recognition. Knowl.-Based Syst. 240, 108136 (2022)

    Article  Google Scholar 

  2. Poux, D., Allaert, B., Ihaddadene, N., Bilasco, I.M., Djeraba, C., Bennamoun, M.: Dynamic facial expression recognition under partial occlusion with optical flow reconstruction. IEEE Trans. Image Process. 31, 446–457 (2022)

    Article  Google Scholar 

  3. Zou, W., Zhang, D., Lee, D.J.: A new multi-feature fusion based convolutional neural network for facial expression recognition. Appl. Intell. 52, 2918–2929 (2022)

    Article  Google Scholar 

  4. Truong, H.P., Nguyen, T.P., Kim, Y.G.: Weighted statistical binary patterns for facial feature representation. Appl. Intell. 52, 1893–1912 (2022)

    Article  Google Scholar 

  5. Chen, D., Song, P., Zheng, W.: Learning transferable sparse representations for cross-corpus facial expression recognition. IEEE Trans. Affect. Comput. 14(2), 1322–1333 (2021)

    Article  Google Scholar 

  6. Hu, H.F., Zhang, P., Ma, Z.M.: Direct kernel neighborhood discriminant analysis for face recognition. Pattern Recogn. Lett. 30, 902–907 (2009)

    Article  Google Scholar 

  7. Liu, Z.H., Lai, Z.H., Ou, W.H., et al.: Discriminative sparse least square regression for semi-supervised learning. Inf. Sci. 636, 118903 (2023)

    Article  Google Scholar 

  8. Zeng, N.Y., Zhang, H., Song, B.Y., Liu, W.B., Li, Y.R., Dobaie, A.M.: Facial expression recognition via learning deep sparse autoencoders. Neurocomputing 273, 643–649 (2018)

    Article  Google Scholar 

  9. Chu, W.S., De la Torre, F., Cohn, J.F.: selective transfer machine for personalized facial expression analysis. IEEE Trans. Pattern Anal. Mach. Intell. 39, 529–545 (2017)

    Article  Google Scholar 

  10. Bashar, F., Khan, A., Ahmed, F., Kabir, M.H., Robust facial expression recognition based on median ternary pattern (MTP), In: International conference on electrical information and communication technology, 1–5. (2014)

  11. Turan, C., Lam, K.M.: Histogram-based local descriptors for facial expression recognition (FER): a comprehensive study. J. Vis. Commun. Image Represent. 55, 331–341 (2018)

    Article  Google Scholar 

  12. Yu, W.M., Xu, H.: Co-attentive multi-task convolutional neural network for facial expression recognition. Pattern Recognit 123, 108401 (2022)

    Article  Google Scholar 

  13. Zhu, Q., Mao, Q.R., Jia, H.J., Elias, O., Noi, N., Tu, J.J.: Convolutional relation network for facial expression recognition in the wild with few-shot learning. Expert Syst. Appl. 189, 116046 (2022)

    Article  Google Scholar 

  14. Han, J.Y., Du, L., Ye, X.Q., Zhang, L., Feng, J.F.: The devil is in the face: exploiting harmonious representations for facial expression recognition. Neurocomputing 486, 104–113 (2022)

    Article  Google Scholar 

  15. F. Xue, Q. Wang, G. Guo, TransFER: learning relation-aware facial expression representations with transformers, In: International conference on computer vision 3581–3590. (2021)

  16. Zhao, Z., Liu, Q., Wang, S.: Learning deep global multi-scale and local attention features for facial expression recognition in the wild. IEEE Trans. Image Process. 30, 6544–6556 (2021)

    Article  Google Scholar 

  17. Sun, Z., Chiong, R., Hu, Z.P., Dhakal, S.: A dynamic constraint representation approach based on cross-domain dictionary learning for expression recognition. J. V. Commun. Image Represent. 85, 103458 (2022)

    Article  Google Scholar 

  18. Tanfous, A.B., Drira, H., Amor, B.B.: Sparse coding of shape trajectories for facial expression and action recognition. IEEE Trans. Pattern Anal. Mach. Intell. 42, 2594–2607 (2020)

    Article  Google Scholar 

  19. Sun, Z., Chiong, R., Hu, Z.P.: Self-adaptive feature learning based on a priori knowledge for facial expression recognition. Knowl.-Based Syst. 204, 106124 (2020)

    Article  Google Scholar 

  20. Yan, K.Y., Zheng, W.M., Cui, Z., Zong, Y., Zhang, T., Tang, C.G.: Unsupervised facial expression recognition using domain adaptation based dictionary learning approach. Neurocomputing 319, 84–91 (2018)

    Article  Google Scholar 

  21. Yang, M., Zhang, L., Feng, X. Fisher discrimination dictionary learning for sparse representation, In: IEEE international conference on computer vision, 543–550. (2011)

  22. Aharon, M., Elad, M., Bruckstein, A.: K-SVD: an algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54, 4311–4322 (2006)

    Article  Google Scholar 

  23. Moeini, A., Faez, K., Moeini, H., Safai, A.M.: Facial expression recognition using dual dictionary learning. J. Vis. Commun. Image Represent. 45, 20–33 (2017)

    Article  Google Scholar 

  24. Luo, X., Xu, Y., Yang, J.: Multi-resolution dictionary learning for face recognition. Pattern Recogn. 93, 283–292 (2019)

    Article  Google Scholar 

  25. Song, P., Deng, X., Mota, J.F.C., Deligiannis, N., Dragotti, P.L., Rodrigues, M.R.D.: Multimodal image super-resolution via joint sparse representations induced by coupled dictionaries. IEEE Trans. Comput. Imagin. 6, 57–72 (2020)

    Article  MathSciNet  Google Scholar 

  26. Juefei-Xu, F., Pal, D.K., Savvides, M. NIR-VIS heterogeneous face recognition via cross-spectral joint dictionary learning and reconstruction, In: IEEE conference on computer vision and pattern recognition workshops, 141–150. (2015)

  27. Peng, Y.L., Li, L.P., Liu, S.G., Lei, T.: Space-frequency domain based joint dictionary learning and collaborative representation for face recognition. Signal Process. 147, 101–109 (2018)

    Article  Google Scholar 

  28. Zhang, G.Q., Porikli, F., Sun, H.J., Sun, Q.S., Xia, G.Y., Zheng, Y.H.: Cost-sensitive joint feature and dictionary learning for face recognition. Neurocomputing 391, 177–188 (2020)

    Article  Google Scholar 

  29. Huang, Q.H., Huang, C.Q., Wang, X.Z., Jiang, F.: Facial expression recognition with grid-wise attention and visual transformer. Inf. Sci. 580, 35–54 (2021)

    Article  MathSciNet  Google Scholar 

  30. Sun, N., Li, Q., Huan, R.Z., Liu, J.X., Han, G.: Deep spatial-temporal feature fusion for facial expression recognition in static images. Pattern Recogn. Lett. 119, 49–61 (2019)

    Article  Google Scholar 

  31. Chan, T.H., Jia, K., Gao, S., Lu, J., Zeng, Z., Ma, Y.: PCANet: A simple deep learning baseline for image classification? IEEE Trans. Image Process. 24, 5017–5032 (2015)

    Article  MathSciNet  Google Scholar 

  32. Sun, Z., Chiong, R., Hu, Z.P.: An extended dictionary representation approach with deep subspace learning for facial expression recognition. Neurocomputing 316, 1–9 (2018)

    Article  Google Scholar 

  33. Sun, Z., Hu, Z.P., Chiong, R., Wang, M., He, W.: Combining the kernel collaboration representation and deep subspace learning for facial expression recognition. J. Circuits Syst. Comput. 27(08), 1850121 (2018)

    Article  Google Scholar 

  34. Wright, J., Yang, A.Y., Ganesh, A., Sastry, S.S., Ma, Y.: Robust face recognition via sparse representation. IEEE Trans. Pattern Anal. Mach. Intell. 31, 210–227 (2009)

    Article  Google Scholar 

  35. Lyons, M., Budynek, J., Akamatsu, S.: Automatic classification of single facial images. IEEE Trans. Pattern Anal. Mach. Intell. 21, 1357–1362 (1999)

    Article  Google Scholar 

  36. Lucey, P., Jeffrey, F. C., Kanade, T., Saragih, J., Ambadar, Z., Matthews, I. In: The extended Cohn-Kanade dataset (CK+): a complete dataset for action unit and emotion specified expression, computer vision and pattern recognition 94–101. (2010)

  37. Lyons, M., Akamatsu, S., Kamachi, M., Gyoba, J. Coding facial expressions with Gabor wavelets, IEEE International conference on automatic face and gesture recognition 200–205. (1998)

  38. Nurzynska, K.: Emotion recognition: the influence of texture’s descriptors on classification accuracy, communications in computer and information. Science 716, 427–438 (2017)

    Google Scholar 

  39. Ouellet, S. Real-time emotion recognition for gaming using deep convolutional network features. ArXiv preprint (2014).

  40. Poursaberi, A., Noubari, H.A., Gavrilova, M., Yanushkevich, S.N.: Gauss-Laguerre wavelet textural feature fusion with geometrical information for facial expression identification. EURASIP J. Image Video Process. 2012, 17 (2012)

    Article  Google Scholar 

  41. Kas, M., El Merabet, Y., Ruichek, Y., Messoussi, R.: New framework for person-independent facial expression recognition combining textural and shape analysis through new feature extraction approach. Inf. Sci. 549, 200–220 (2021)

    Article  MathSciNet  Google Scholar 

  42. Wu, B.F., Lin, C.H.: Adaptive feature mapping for customizing deep learning based facial expression recognition model. IEEE Access 6, 12451–12461 (2018)

    Article  Google Scholar 

  43. Du, L.S., Hu, H.F.: Weighted patch-based manifold regularization dictionary pair learning model for facial expression recognition using iterative optimization classification strategy. Comput. Vis. Image Underst. 186, 13–24 (2019)

    Article  Google Scholar 

  44. Zhang, K., Huang, Y., Du, Y., Wang, L.: Facial expression recognition based on deep evolutional spatial-temporal networks. IEEE Trans. Image Process. 26, 4193–4203 (2017)

    Article  MathSciNet  Google Scholar 

  45. Sun, Z., Hu, Z.P., Wang, M., Zhao, S.H.: Discriminative feature learning-based pixel difference representation for facial expression recognition. IET Comput. Vision 11, 675–682 (2017)

    Article  Google Scholar 

  46. Guo, S., Feng, L., Feng, Z.B., Li, Y.H., Wang, Y., Liu, S.L., Qiao, H.: Multi-view laplacian least squares for human emotion recognition. Neurocomputing 370, 78–87 (2019)

    Article  Google Scholar 

  47. Cai, S., Zuo, W., Zhang, L. Support vector guided dictionary learning, In: Proceedings of the european conference on computer vision, Springer International Publishing, (2014).

  48. Gu, S., Zhang, L., Zuo, W., Feng, X. Projective dictionary pair learning for pattern classification, In: Proceedings of the 28th annual conference on neural information processing systems (NIPS), 793–801. (2014)

Download references

Funding

This work is funded by the National Natural Science Foundation of China under Grants 62001413, Science and Technology Project of Hebei Education Department under Grants BJK2023117 and Key Project of basic innovation and scientific research cultivation of Yanshan University under Grants 2023LGZD006.

Author information

Authors and Affiliations

  1. Department of Information Science and Engineering, Yanshan University, Hebei Street, Qinhuangdao, Hebei, China

    Zhe Sun, Jiatong Bai & Hehao Zhang

Authors
  1. Zhe Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jiatong Bai
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Hehao Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Zhe Sun (Corresponding author) involved in conceptualization, methodology, supervision, writing—review & editing, and funding acquisition. Jiatong Bai involved in formal analysis, investigation, validation, and writing—original draft. Hehao Zhang involved in investigation, validation, and writing—editing. All authors reviewed and approved the manuscript.

Corresponding author

Correspondence to Zhe Sun.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Ethical approval

The authors followed all the Ethics during the research and submission time of the article.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Sun, Z., Bai, J. & Zhang, H. Multi-class differentiation feature representation guided joint dictionary learning for facial expression recognition. SIViP 18 (Suppl 1), 747–756 (2024). https://doi.org/10.1007/s11760-024-03189-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-024-03189-y

Keywords

Search

Navigation