[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content

Advertisement

Springer Nature Link
Log in

Similarity-based face image retrieval using sparsely embedded deep features and binary code learning

  • Regular Paper
  • Published:
International Journal of Multimedia Information Retrieval Aims and scope Submit manuscript

Abstract

Human face retrieval has long been established as one of the most interesting research topics in computer vision. With the recent development of deep learning, many researchers have addressed this problem by building deep hashing models to learn binary code from face images, while performing face retrieval as a classification task. Nevertheless, the performance is still unsatisfactory since these models are incapable of handling inter-class variation between multiple persons, as we need to make a class label for each identity. In this backdrop, we propose in this paper an effective deep learning-based framework for face image retrieval. The key to our framework is mainly based on the matching of face pairs, where a two-stream network, named \(\chi Net+\chi Match\), is designed to learn similarities in terms of person identity. Such similarities are investigated by embedding both deep local representation via face components, and deep global face representation via the whole face image. Since the similarities captured over face components are supposed to diversify due to variation in pose, expression and occlusion, we also introduce a Sparse Score Fusion layer that learns automatically the weight of each component according to its contribution to face matching. To allow fast retrieval, we farther propose a method that generates binary codes corresponding to the groups of similar faces through the hierarchical k-means, where the path down binary tree is exploited as a binary code for indexing. The final retrieval is then conducted within a privileged subset of images in the database. Our experiments on different challenging datasets show that our approach obtains outstanding results while outperforming most existing methods.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12

Similar content being viewed by others

Data availibility

All datasets used in this work are publicly available and have been properly referenced in the text.

Notes

  1. https://github.com/mzhang367/opqn.

References

  1. Mishra A (2019) Dhfml: deep heterogeneous feature metric learning for matching photograph and cartoon pairs. Int J Multimed Info Retr 8:135–142

    Article  Google Scholar 

  2. Kafai M, An L, Bhanu B (2014) Reference face graph for face recognition. IEEE Trans Info Foren Secur 9(12):2132–2143

    Article  Google Scholar 

  3. Khan AM, Jalal AS (2020) A framework for suspect face retrieval using linguistic descriptions. Expert Syst Appl 141:112925

    Article  Google Scholar 

  4. Yang S, Kafai M, An L, Bhanu B (2014) Zapping index: using smile to measure advertisement zapping likelihood. IEEE Trans Affect Comput 5(4):432–444

    Article  Google Scholar 

  5. Ameur B, Belahcene M, Masmoudi S, Ben Hamida A (2019) Hybrid descriptors and weighted PCA-EFMNet for face verification in the wild. Int J Multimed Info Retr 8:143–154

    Article  Google Scholar 

  6. Chen BC, Chen YY, Kuo YH, Hsu WH (2013) Scalable face image retrieval using attribute-enhanced sparse codewords. IEEE Trans Multimedia 15(5):1163–1173

    Article  Google Scholar 

  7. Sun Y, Chen Y, Wang X, Tang X (2014) Deep learning face representation by joint identification-verification. Advances in neural information processing systems

  8. Wang D, Otto C, Jain AK (2015) Face search at scale: 80 million gallery, arXiv:1507.07242

  9. Xia R, Pan Y, Lai H, Liu C, Yan S (2014) Supervised hashing for image retrieval via image representation learning. In: Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence

  10. Dong Z, Jia S, Wu T, Pei M (2016) Face video retrieval via deep learning of binary hash representations. In: Thirtieth AAAI Conference on Artificial Intelligence

  11. Lin J, Li Z, Tang J (2017) Discriminative deep hashing for scalable face image retrieval. In: Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence

  12. Tang J, Li Z, Zhu X (2018) Supervised deep hashing for scalable face image retrieval. Pattern Recogn 75:25–32

    Article  Google Scholar 

  13. Tang J, Lin J, Li Z, Yang J (2018) Discriminative deep quantization hashing for face image retrieval. IEEE Trans Neural Netw Learn Syst 99:1–9

    Google Scholar 

  14. Xiong Z, Li B, Gu X, Gu W, Wang W (2019) Discriminative deep attention-aware hashing for face image retrieval. In: Pacific Rim International Conference on Artificial Intelligence

  15. Ng HW, Winkler S (2014) A data-driven approach to cleaning large face datasets. In: Proceedings of IEEE International Conference on Image Processing (ICIP)

  16. Wolf L, Hassner T, Maoz I (2011) Face recognition in unconstrained videos with matched background similarity. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

  17. Cao Q, Shen L, Xie W, Parkhi OM, Zisserman A (2018) VGGFace2: a dataset for recognizing face across pose and age. In: 13th International Conference on Automatic Face and Gesture Recognition

  18. Wang F, Chen L, Li C, Huang S, Chen Y, Qian C, Loy CC (2018) The devil of face recognition is in the noise, In: Proceedings of the European Conference on Computer Vision (ECCV’18) pp 1–16

  19. Wang D, Hoi SC, He Y, Zhu J, Mei T, Luo J (2014) Retrieval-based face annotation by weak label regularized local coordinate coding. IEEE Trans Pattern Anal Mach Intell 36(3):550–563

    Article  Google Scholar 

  20. Klare B.F, Blanton A, Klein B (2014) Efficient face retrieval using synecdoches, In: IEEE. IEEE International Joint Conference on Biometrics pp 1–7

  21. Chen BC, Chen YY, Kuo YH, Ngo TD, Le DD, Satoh SI, Hsu WH (2017) Scalable face track retrieval in video archives using bag-of-faces sparse representation. IEEE Trans Circuits Syst Video Technol 27(7):1595–1603

    Article  Google Scholar 

  22. Liu D, Li J, Wang N, Peng C, Gao X (2018) Composite components-based face sketch recognition. Neurocomputing 302:46–54

    Article  Google Scholar 

  23. Zaeemzadeh A, Ghadar S, Faieta B, Lin Z, Rahnavard N, Shah M, Kalarot R (2021) Face image retrieval with attribute manipulation, In: Proceedings of the IEEE/CVF International Conference on Computer Vision pp 12116–12125

  24. Dai P, Wang X, Zhang W, Zhang P, You W (2018) Implicit relative attribute enabled cross-modality hashing for face image-video retrieval. Multimedia Tools Appl 77(18):23547–23577

    Article  Google Scholar 

  25. An L, Zou C, Zhang L, Denney B (2016) Scalable attribute-driven face image retrieval. Neurocomputing 172:215–224

    Article  Google Scholar 

  26. Li Y, Wang R, Liu H, Jiang H, Shan S, Chen X (2015) Two birds, one stone: jointly learning binary code for large-scale face image retrieval and attributes prediction, In: Proceedings of the IEEE International Conference on Computer Vision pp 3819–3827

  27. Li Y, Wang R, Huang Z, Shan S, Chen X (2015) Face video retrieval with image query via hashing across euclidean space and riemannian manifold. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition

  28. Li Y, Wang R, Cui Z, Shan S, Chen X (2016) Spatial pyramid covariance-based compact video code for robust face retrieval in tv-series. IEEE Trans Image Process 25(12):5905–5919

    Article  MathSciNet  Google Scholar 

  29. Kafai M, Eshghi K, Bhanu B (2014) Discrete cosine transform locality-sensitive hashes for face retrieval. IEEE Trans Multimedia 16(4):1090–1103

    Article  Google Scholar 

  30. Li Y, Wang R, Shan S, Chen X (2015) Hierarchical hybrid statistic based video binary code and its application to face retrieval in tv-series, vol. 1, IEEE. In: 11th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition pp 1–8

  31. Li Y, Wang R, Cui Z, Shan S, Chen X (2014) Compact video code and its application to robust face retrieval in TV-series

  32. Gui J, Liu T, Sun Z, Tao D, Tan T (2017) Fast supervised discrete hashing. IEEE Trans Pattern Anal Mach Intell 40(2):490–496

    Article  Google Scholar 

  33. Dubey SR (2017) Face retrieval using frequency decoded local descriptor. Multimedia Tools Appl 78(12):16411–16431

    Article  Google Scholar 

  34. Geng X (2016) Label distribution learning. IEEE Trans Knowl Data Eng 28:1734–1748

    Article  Google Scholar 

  35. Xu N, Liu YP, Geng X (2019) Label enhancement for label distribution learning. IEEE Trans Knowl Data Eng 33:1632–1643

    Article  Google Scholar 

  36. Xu N, Shu J, Zheng R, Geng X, Meng D, Zhang ML (2022) Variational label enhancement. IEEE Trans Pattern Anal Mach Intell 45:6537–6551

    Google Scholar 

  37. LeCun Y, Bengio Y, Hinton G (2015) Deep learning. Nature 521(7553):436

    Article  Google Scholar 

  38. Wang D, Jain AK (2015) Face retriever: Pre-filtering the gallery via deep neural net, IEEE. In: International Conference on Biometrics pp 473–480

  39. Li Q, Sun Z, He R, Tan T (2017) Deep supervised discrete hashing. In NIPS

  40. Zhe X, Chen S, Yan H (2020) Deep class-wise hashing: semantics-preserving hashing via class-wise loss. IEEE Trans Neural Netw Learn Syst 31(5):1681–1695

    Article  Google Scholar 

  41. Zhang R, Lin L, Zhang R, Zuo W, Zhang L (2015) Bit-scalable deep hashing with regularized similarity learning for image retrieval and person re-identification. IEEE Trans Image Process 24(12):4766–4779

    Article  MathSciNet  Google Scholar 

  42. Wang R, Wang R, Qiao S, Shan S, Chen X (2020) Deep position-aware hashing for semantic continuous image retrieval. In WACV

  43. Zhang M, Zhe X, Chen S, Yan H (2021) Deep center-based dual-constrained hashing for discriminative face image retrieval. Pattern Recogn 117:107976

    Article  Google Scholar 

  44. Zhang M, Zhe X, Yan H (2023) Orthonormal product quantization network for scalable face image retrieval. Pattern Recogn 141:109671

    Article  Google Scholar 

  45. Tarawneh AS, Hassanat A, Celik C, Chetverikov D, Rahman MS, Verma C (2019) Deep face image retrieval: a comparative study with dictionary learning, IEEE. In: 10th International Conference on Information and Communication Systems (ICICS) pp 185–192

  46. Jing C, Dong Z, Pei M, Jia Y (2019) Heterogeneous hashing network for face retrieval across image and video domains. IEEE Trans Multimedia 21(3):782–794

    Article  Google Scholar 

  47. Xu C, Liu Q, Ye M (2017) Age invariant face recognition and retrieval by coupled auto-encoder networks. Neurocomputing 222:62–71

    Article  Google Scholar 

  48. Liu D, Gao X, Wang N, Li J, Peng C (2020) Coupled attribute learning for heterogeneous face recognition. IEEE Trans Neural Netw Learn Syst 31:4699–4712

    Article  Google Scholar 

  49. Liu D, Gao X, Peng C, Wang N, Li J (2022) Heterogeneous face interpretable disentangled representation for joint face recognition and synthesis. IEEE Trans Neural Netw Learn Syst 33:5611–5625

    Article  Google Scholar 

  50. Liu D, Yang W, Peng C, Wang N, Hu R, Gao X (2023) Modality-agnostic augmented multi-collaboration representation for semi-supervised heterogenous face recognition, In: Proceedings of the 31st ACM International Conference on Multimedia pp 4647–4656

  51. Jiang H, Learned-Miller E (2017) Face detection with the faster R-CNN. In: IEEE International Conference on Automatic Face and Gesture Recognition

  52. Hannane R, Elboushaki A, Afdel K (2020) A divide-and-conquer strategy for facial landmark detection using dual-task cnn architecture. Pattern Recongnition 107:107504

    Article  Google Scholar 

  53. Han X, Leung T, Jia Y, Sukthankar R, Berg AC (2015) Matchnet: Unifying feature and metric learning for patch-based matching, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp 3279–3286

  54. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, In: Proceedings of the IEEE conference on computer vision and pattern recognition pp 770–778

  55. Huang GB, Ramesh M, Berg T, Learned-Miller E (2007) Labeled faces in the wild: a database for studying face recognition in unconstrained environments. University of Massachusetts, Amherst, Technical Report, pp 07–49

    Google Scholar 

  56. Yuan L, Wang T, Zhang X, Tay FE, Jie Z, Liu W, Feng J (2020) Central similarity quantization for efficient image and video retrieval, In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition pp 3083–3092

  57. Jang YK, Cho NI (2020) Generalized product quantization network for semi-supervised image retrieval, In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition pp 3420–3429

  58. Xiong Z, Wu D, Gu W, Zhang H, Li B, Wang W (2020) Deep discrete attention guided hashing for face image retrieval, In: Proceedings of the 2020 International Conference on Multimedia Retrieval pp 136–144

  59. Feng P, Zhang H, Sun Y, Tang Z (2023) Lightweight image hashing based on knowledge distillation and optimal transport for face retrieval, In: 29th International Conference on Multimedia Modeling pp 423–434

  60. Liu H, Wang R, Shan S, Chen X (2016) Deep supervised hashing for fast image retrieval, In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition pp 2064–2072

  61. Li W.J, Wang S, Kang W.C (2016) Feature learning based deep supervised hashing with pairwise labels, In: Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence pp 1711–1717

  62. Wang X, Shi Y, Kitani K.M (2016) Deep supervised hashing with triplet labels, In: 13th Asian Conference on Computer Vision pp 70–84

  63. Zhou L, Wang Y, Xiao B, Xu Q (2021) Dfh-gan: a deep face hashing with generative adversarial network, In: 25th International Conference on Pattern Recognition (ICPR) pp 7012–7019

Download references

Acknowledgements

This work is supported in part by the PPR2-2015 project under grant number 14UIZ2015, and in part by the Al Khawarizmi project under grant number ALKHAWARIZMI/2020/02 financed by the Moroccan government through the CNRST funding program.

Author information

Authors and Affiliations

  1. Laboratory of Computer Engineering and Systems, Faculty of Sciences and Technology, Cadi Ayyad University, 40000, Marrakech, Morocco

    Abdessamad Elboushaki

  2. Laboratory of Computer Systems Engineering, Faculty of Sciences Semlalia Marrakech, Cadi Ayyad University, 40000, Marrakech, Morocco

    Rachida Hannane

  3. Laboratory of Computer Systems and Vision, Faculty of Science, Ibn Zohr University, 80000, Agadir, Morocco

    Karim Afdel

Authors
  1. Abdessamad Elboushaki
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Rachida Hannane
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Karim Afdel
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Abdessamad Elboushaki.

Ethics declarations

Conflict of interest

The authors have no Conflict of interest to declare that are relevant to the content of this article.

Human participants

This article does not contain any studies involving human participants or animals performed by any of the authors.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Elboushaki, A., Hannane, R. & Afdel, K. Similarity-based face image retrieval using sparsely embedded deep features and binary code learning. Int J Multimed Info Retr 13, 28 (2024). https://doi.org/10.1007/s13735-024-00337-5

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13735-024-00337-5

Keywords

Search

Navigation