[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

FRCSyn-onGoing: : Benchmarking and comprehensive evaluation of real and synthetic data to improve face recognition systems

Published: 02 July 2024 Publication History

Abstract

This article presents FRCSyn-onGoing, an ongoing challenge for face recognition where researchers can easily benchmark their systems against the state of the art in an open common platform using large-scale public databases and standard experimental protocols. FRCSyn-onGoing is based on the Face Recognition Challenge in the Era of Synthetic Data (FRCSyn) organized at WACV 2024. This is the first face recognition international challenge aiming to explore the use of real and synthetic data independently, and also their fusion, in order to address existing limitations in the technology. Specifically, FRCSyn-onGoing targets concerns related to data privacy issues, demographic biases, generalization to unseen scenarios, and performance limitations in challenging scenarios, including significant age disparities between enrollment and testing, pose variations, and occlusions. To enhance face recognition performance, FRCSyn-onGoing strongly advocates for information fusion at various levels, starting from the input data, where a mix of real and synthetic domains is proposed for specific tasks of the challenge. Additionally, participating teams are allowed to fuse diverse networks within their proposed systems to improve the performance. In this article, we provide a comprehensive evaluation of the face recognition systems and results achieved so far in FRCSyn-onGoing. The results obtained in FRCSyn-onGoing, together with the proposed public ongoing benchmark, contribute significantly to the application of synthetic data to improve face recognition technology.

Highlights

Challenges in face recognition: Privacy, demographic bias, and generalization.
Synthetic data offer privacy and large-scale databases with desired characteristics.
FRCSyn-onGoing: first challenge investigates face recognition and synthetic data.
Fusion of real and synthetic data, among others, can mitigate several challenges.

References

[1]
Minaee S., Abdolrashidi A., Su H., Bennamoun M., Zhang D., Biometrics recognition using deep learning: A survey, Artif. Intell. Rev. (2023) 1–49.
[2]
J. Deng, J. Guo, N. Xue, S. Zafeiriou, ArcFace: Additive angular margin loss for deep face recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[3]
M. Kim, A.K. Jain, X. Liu, AdaFace: Quality Adaptive Margin for Face Recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[4]
Wang M., Deng W., Deep face recognition: A survey, Neurocomputing 429 (2021) 215–244.
[5]
G.B. Huang, M. Mattar, T. Berg, E. Learned-Miller, Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments, in: Proceedings of the Workshop on Faces in ‘Real-Life’ Images: Detection, Alignment, and Recognition, 2008.
[6]
Adjabi I., Ouahabi A., Benzaoui A., Taleb-Ahmed A., Past, present, and future of face recognition: A review, Electronics 9 (8) (2020) 1188.
[7]
Wanyonyi D., Celik T., Open-source face recognition frameworks: A review of the landscape, IEEE Access 10 (2022) 50601–50623.
[8]
Du H., Shi H., Zeng D., Zhang X.-P., Mei T., The elements of end-to-end deep face recognition: A survey of recent advances, ACM Comput. Surv. 54 (10s) (2022) 1–42.
[9]
Zhou K., Liu Z., Qiao Y., Xiang T., Loy C.C., Domain generalization: A survey, IEEE Trans. Pattern Anal. Mach. Intell. (2022).
[10]
Y. Shi, X. Yu, K. Sohn, M. Chandraker, A.K. Jain, Towards universal representation learning for deep face recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 6817–6826.
[11]
Zeng D., Veldhuis R., Spreeuwers L., A survey of face recognition techniques under occlusion, IET Biometr. 10 (6) (2021) 581–606.
[12]
Ali W., Tian W., Din S.U., Iradukunda D., Khan A.A., Classical and modern face recognition approaches: a complete review, Multimedia Tools Appl. 80 (2021) 4825–4880.
[13]
Y. Deng, J. Yang, D. Chen, F. Wen, X. Tong, Disentangled and controllable face image generation via 3d imitative-contrastive learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[14]
G. Bae, M. de La Gorce, T. Baltrušaitis, C. Hewitt, D. Chen, J. Valentin, R. Cipolla, J. Shen, DigiFace-1M: 1 Million Digital Face Images for Face Recognition, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2023.
[15]
C. Zhang, X. Chen, S. Chai, C.H. Wu, D. Lagun, T. Beeler, F. De la Torre, ITI-GEN: Inclusive Text-to-Image Generation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[16]
M. Kim, F. Liu, A. Jain, X. Liu, DCFace: Synthetic Face Generation with Dual Condition Diffusion Model, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[17]
P. Melzi, C. Rathgeb, R. Tolosana, R. Vera-Rodriguez, A. Morales, D. Lawatsch, F. Domin, M. Schaubert, Synthetic Data for the Mitigation of Demographic Biases in Face Recognition, in: Proceedings of the IEEE International Joint Conference on Biometrics, 2023.
[18]
Boutros F., Struc V., Fierrez J., Damer N., Synthetic data for face recognition: Current state and future prospects, Image Vis. Comput. (2023).
[19]
Tolosana R., Vera-Rodriguez R., Fierrez J., Morales A., Ortega-Garcia J., Deepfakes and beyond: A survey of face manipulation and fake detection, Inf. Fusion 64 (2020) 131–148.
[20]
Rathgeb C., Tolosana R., Vera-Rodriguez R., Busch C., Handbook of Digital Face Manipulation and Detection: from DeepFakes to Morphing Attacks, Springer Nature, 2022.
[21]
Neves J.C., Tolosana R., Vera-Rodriguez R., Lopes V., Proença H., Fierrez J., GANprintR: Improved fakes and evaluation of the state of the art in face manipulation detection, IEEE J. Sel. Top. Sign. Proces. 14 (5) (2020) 1038–1048.
[22]
Salazar-Jurado E.H., Hernández-García R., Vilches-Ponce K., Barrientos R.J., Mora M., Jaswal G., Towards the generation of synthetic images of palm vein patterns: A review, Inf. Fusion 89 (2023) 66–90.
[23]
Boutros F., Damer N., Raja K., Ramachandra R., Kirchbuchner F., Kuijper A., Iris and periocular biometrics for head mounted displays: Segmentation, recognition, and synthetic data generation, Image Vis. Comput. 104 (2020).
[24]
Kang P., Jiang S., Shull P.B., Synthetic EMG based on adversarial style transfer can effectively attack biometric-based personal identification models, IEEE Trans. Neural Syst. Rehabil. Eng. (2023).
[25]
Murgia M., Harlow M., Who’s using your face? The ugly truth about facial recognition, in: Financial Times, Vol. 19, 2019.
[26]
Harvey, Adam. LaPlace J., Exposing.ai, 2021, https://exposing.ai.
[27]
Voigt P., Von dem Bussche A., The EU general data protection regulation (GDPR), in: A Practical Guide, Vol. 10, first ed., Springer International Publishing, Cham, 2017, pp. 10–5555.
[28]
P. Melzi, C. Rathgeb, R. Tolosana, R. Vera-Rodriguez, D. Lawatsch, F. Domin, M. Schaubert, GANDiffFace: Controllable Generation of Synthetic Datasets for Face Recognition with Realistic Variations, in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2023.
[29]
Morales A., Fierrez J., Vera-Rodriguez R., Tolosana R., SensitiveNets: Learning agnostic representations with application to face images, IEEE Trans. Pattern Anal. Mach. Intell. 43 (6) (2020) 2158–2164.
[30]
M. Kim, H. Byun, Learning texture invariant representation for domain adaptation of semantic segmentation, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 12975–12984.
[31]
Chen H., He X., Qing L., Wu Y., Ren C., Sheriff R.E., Zhu C., Real-world single image super-resolution: A brief review, Inf. Fusion 79 (2022) 124–145.
[32]
Y. Shao, L. Li, W. Ren, C. Gao, N. Sang, Domain adaptation for image dehazing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 2808–2817.
[33]
H. Qiu, B. Yu, D. Gong, Z. Li, W. Liu, D. Tao, SynFace: Face Recognition With Synthetic Data, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[34]
F. Boutros, M. Huber, P. Siebke, T. Rieber, N. Damer, Sface: Privacy-friendly and accurate face recognition using synthetic data, in: Proceedings of the IEEE International Joint Conference on Biometrics, 2022.
[35]
F. Boutros, J.H. Grebe, A. Kuijper, N. Damer, IDiff-Face: Synthetic-based Face Recognition through Fizzy Identity-Conditioned Diffusion Model, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023.
[36]
M. Kansy, A. Raël, G. Mignone, J. Naruniec, C. Schroers, M. Gross, R.M. Weber, Controllable Inversion of Black-Box Face Recognition Models via Diffusion, in: Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2023.
[37]
Terhörst P., Kolf J.N., Huber M., Kirchbuchner F., Damer N., Moreno A.M., Fierrez J., Kuijper A., A comprehensive study on face recognition biases beyond demographics, IEEE Trans. Technol. Soc. 3 (1) (2021) 16–30.
[38]
P. Melzi, R. Tolosana, R. Vera-Rodriguez, M. Kim, C. Rathgeb, X. Liu, I. DeAndres-Tame, A. Morales, J. Fierrez, J. Ortega-Garcia, et al., FRCSyn Challenge at WACV 2024: Face Recognition Challenge in the Era of Synthetic Data, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, 2024.
[39]
Rusia M.K., Singh D.K., A comprehensive survey on techniques to handle face identity threats: challenges and opportunities, Multimedia Tools Appl. 82 (2) (2023) 1669–1748.
[40]
S. Sengupta, J.-C. Chen, C. Castillo, V.M. Patel, R. Chellappa, D.W. Jacobs, Frontal to profile face verification in the wild, in: Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2016.
[41]
Oloyede M.O., Hancke G.P., Myburgh H.C., A review on face recognition systems: recent approaches and challenges, Multimedia Tools Appl. 79 (2020) 27891–27922.
[42]
De-la Torre M., Granger E., Radtke P.V., Sabourin R., Gorodnichy D.O., Partially-supervised learning from facial trajectories for face recognition in video surveillance, Inf. Fusion 24 (2015) 31–53.
[43]
Yan C., Meng L., Li L., Zhang J., Wang Z., Yin J., Zhang J., Sun Y., Zheng B., Age-invariant face recognition by multi-feature fusionand decomposition with self-attention, ACM Trans. Multimed. Comput. Commun. Appl. 18 (1s) (2022) 1–18.
[44]
Y. Wang, D. Gong, Z. Zhou, X. Ji, H. Wang, Z. Li, W. Liu, T. Zhang, Orthogonal deep features decomposition for age-invariant face recognition, in: Proceedings of the European Conference on Computer Vision, 2018.
[45]
D. Deb, L. Best-Rowden, A.K. Jain, Face recognition performance under aging, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 46–54.
[46]
Cai J., Han H., Cui J., Chen J., Liu L., Zhou S.K., Semi-supervised natural face de-occlusion, IEEE Trans. Inf. Forensics Secur. 16 (2020) 1044–1057.
[47]
J. Deng, S. Cheng, N. Xue, Y. Zhou, S. Zafeiriou, UV-GAN: Adversarial facial UV map completion for pose-invariant face recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[48]
Y. Hu, X. Wu, B. Yu, R. He, Z. Sun, Pose-guided photorealistic face rotation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
[49]
J. Zhao, Y. Cheng, Y. Xu, L. Xiong, J. Li, F. Zhao, K. Jayashree, S. Pranata, S. Shen, J. Xing, et al., Towards pose invariant face recognition in the wild, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2207–2216.
[50]
K.T. Voo, L. Jiang, C.C. Loy, Delving into high-quality synthetic face occlusion segmentation datasets, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 4711–4720.
[51]
G. Antipov, M. Baccouche, J.-L. Dugelay, Face aging with conditional generative adversarial networks, in: Proceedings of the IEEE International Conference on Image Processing, 2017.
[52]
J. Zhao, L. Xiong, P. Karlekar Jayashree, J. Li, F. Zhao, Z. Wang, P. Sugiri Pranata, P. Shengmei Shen, S. Yan, J. Feng, Dual-Agent GANs for Photorealistic and Identity Preserving Profile Face Synthesis, in: Proceedings of the Advances in Neural Information Processing Systems, Vol. 30, 2017.
[53]
J. Zhao, L. Xiong, Y. Cheng, Y. Cheng, J. Li, L. Zhou, Y. Xu, J. Karlekar, S. Pranata, S. Shen, et al., 3D-Aided Deep Pose-Invariant Face Recognition, in: Proceedings of the International Joint Conferences on Artificial Intelligence, Vol. 2, (3) 2018, p. 11.
[54]
Yi D., Lei Z., Liao S., Li S.Z., Learning face representation from scratch, 2014, arXiv preprint arXiv:1411.7923.
[55]
Q. Cao, L. Shen, W. Xie, O.M. Parkhi, A. Zisserman, VGGFace2: A dataset for recognising faces across pose and age, in: Proceedings of the International Conference on Automatic Face and Gesture Recognition, 2018.
[56]
Y. Guo, L. Zhang, Y. Hu, X. He, J. Gao, MS-Celeb-1M: A dataset and benchmark for large-scale face recognition, in: Proceedings of the European Conference on Computer Vision, 2016.
[57]
I. Sarridis, C. Koutlis, S. Papadopoulos, C. Diou, Towards Fair Face Verification: An In-depth Analysis of Demographic Biases, in: Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases Workshops, 2023.
[58]
M. Wang, W. Deng, Mitigating bias in face recognition using skewness-aware reinforcement learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[59]
Cherepanova V., Reich S., Dooley S., Souri H., Goldblum M., Goldstein T., A deep dive into dataset imbalance and bias in face identification, 2022, arXiv preprint arXiv:2203.08235.
[60]
H. Zhang, M. Grimmer, R. Ramachandra, K. Raja, C. Busch, On the applicability of synthetic data for face recognition, in: Proceedings of the IEEE International Workshop on Biometrics and Forensics, 2021.
[61]
Karras T., Aittala M., Laine S., Härkönen E., Hellsten J., Lehtinen J., Aila T., Alias-free generative adversarial networks, Adv. Neural Inf. Process. Syst. 34 (2021) 852–863.
[62]
Falkenberg M., Ottsen A.B., Ibsen M., Rathgeb C., Child face recognition at scale: Synthetic data generation and performance benchmark, 2023, arXiv preprint arXiv:2304.11685.
[63]
L. Colbois, T. de Freitas Pereira, S. Marcel, On the use of automatically generated synthetic image datasets for benchmarking face recognition, in: Proceedings of the IEEE International Joint Conference on Biometrics, 2021.
[64]
T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, T. Aila, Training Generative Adversarial Networks with Limited Data, in: Advances in Neural Information Processing Systems, 2020.
[65]
T. Karras, S. Laine, T. Aila, A Style-Based Generator Architecture for Generative Adversarial Networks, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[66]
S. Moschoglou, A. Papaioannou, C. Sagonas, J. Deng, I. Kotsia, S. Zafeiriou, AgeDB: The First Manually Collected, In-The-Wild Age Database, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017.
[67]
M.E. Erakın, U. Demir, H.K. Ekenel, On Recognizing Occluded Faces in the Wild, in: Proceedings of the International Conference of the Biometrics Special Interest Group, 2021.
[68]
K. Karkkainen, J. Joo, FairFace: Face attribute dataset for balanced race, gender, and age for bias measurement and mitigation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
[69]
of Standards N.I., (NIST) T., FRVT 1:1 Verification, NIST, 2023, https://pages.nist.gov/frvt/html/frvt11.html.
[70]
M. Ester, H.-P. Kriegel, J. Sander, X. Xu, et al., A Density-Based Algorithm for Discovering Clusters in Large Spatial Databases with Noise, in: Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, Vol. 96, (34) 1996, pp. 226–231.
[71]
Ngan M.L., Grother P.J., Hanaoka K.K., Ongoing Face Recognition Vendor Test (FRVT) Part 6B: Face recognition accuracy with face masks using post-COVID-19 algorithms, 2020.
[72]
J. Wang, Y. Liu, Y. Hu, H. Shi, T. Mei, FaceX-Zoo: A PyTorch Toolbox for Face Recognition, in: Proceedings of the 29th ACM International Conference on Multimedia, 2021.
[73]
K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016.
[74]
J. Deng, J. Guo, E. Ververas, I. Kotsia, S. Zafeiriou, RetinaFace: Single-shot Multi-level Face Localisation in the Wild, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[75]
B.H. Zhang, B. Lemoine, M. Mitchell, Mitigating unwanted biases with adversarial learning, in: Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2018.
[76]
J. Deng, J. Guo, X. An, Z. Zhu, S. Zafeiriou, Masked face recognition challenge: The insightface track report, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[77]
McNeely-White D., Sattelberg B., Blanchard N., Beveridge R., Canonical face embeddings, IEEE Trans. Biometr., Behav., Ident. Sci. 4 (2022) 197–209.
[78]
McNeely-White D., Sattelberg B., Blanchard N., Beveridge R., Canonical face embeddings, IEEE Trans. Biometr., Behav., Ident. Sci. 4 (2) (2022) 197–209.
[79]
H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, W. Liu, CosFace: Large margin cosine loss for deep face recognition, in: Proceedings IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018.
[80]
J. Deng, J. Guo, T. Liu, M. Gong, S. Zafeiriou, Sub-center ArcFace: Boosting Face Recognition by Large-scale Noisy Web Faces, in: Proceedings of the European Conference on Computer Vision, 2020.
[81]
J. Cheng, T. Liu, K. Ramamohanarao, D. Tao, Learning with bounded instance and label-dependent label noise, in: Proceedings of the International Conference on Machine Learning, 2020.
[82]
Wang M., Zhang Y., Deng W., Meta balanced network for fair face recognition, IEEE Trans. Pattern Anal. Mach. Intell. 44 (11) (2021) 8433–8448.
[83]
Zhang K., Zhang Z., Li Z., Qiao Y., Joint face detection and alignment using multitask cascaded convolutional networks, IEEE Signal Process. Lett. 23 (10) (2016) 1499–1503.
[84]
L. Chai, D. Bau, S.-N. Lim, P. Isola, What makes fake images detectable? Understanding properties that generalize, in: Proceedings of the European Conference on Computer Vision, 2020.
[85]
M. Tan, Q. Le, EfficientNetV2: Smaller Models and Faster Training, in: Proceedings of the International Conference on Machine Learning, 2021.
[86]
F. Boutros, N. Damer, F. Kirchbuchner, A. Kuijper, ElasticFace: Elastic Margin Loss for Deep Face Recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
[87]
F. Boutros, M. Klemt, M. Fang, A. Kuijper, N. Damer, Unsupervised face recognition using unlabeled synthetic data, in: Proceedings of the International Conference on Automatic Face and Gesture Recognition, 2023.
[88]
E.D. Cubuk, B. Zoph, J. Shlens, Q.V. Le, RandAugment: Practical automated data augmentation with a reduced search space, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2020.
[89]
Lumini A., Nanni L., Overview of the combination of biometric matchers, Inf. Fusion 33 (2017) 71–85.

Cited By

View all
  • (2024)FaceX: Understanding Face Attribute Classifiers through Summary Model ExplanationsProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658007(758-766)Online publication date: 30-May-2024

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Information Fusion
Information Fusion  Volume 107, Issue C
Jul 2024
522 pages

Publisher

Elsevier Science Publishers B. V.

Netherlands

Publication History

Published: 02 July 2024

Author Tags

  1. FRCSyn-onGoing
  2. Face recognition
  3. Generative AI
  4. Demographic bias
  5. Benchmark

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 19 Dec 2024

Other Metrics

Citations

Cited By

View all
  • (2024)FaceX: Understanding Face Attribute Classifiers through Summary Model ExplanationsProceedings of the 2024 International Conference on Multimedia Retrieval10.1145/3652583.3658007(758-766)Online publication date: 30-May-2024

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media