[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content
Springer Nature Link
Log in

A multi-granularity facial extreme makeup transfer and removal model with local-global collaboration

  • Published:
Applied Intelligence Aims and scope Submit manuscript

Abstract

Most existing face makeup transfer methods have limitations in accuracy, realism, and identity preservation. These methods cannot transfer extreme makeup with complex patterns, only light ones. To address these issues, we develop a multi-granularity facial makeup transfer and removal model with local-global collaboration. This model can apply and remove all makeup styles (i.e., light to extreme makeup) on a face image accurately. First, we design novel local discriminators for facial local patches divided by landmarks, in order to distinguish the accuracy of local makeup transfer. The local and global discriminators collaborate to effectively separate content and makeup style features, accurately handle transfer from coarse-grained global and fine-grained local perspectives, and adequately preserve facial identity. Then, we propose a novel loss function that ensures the consistency of local makeup styles, and maintains color, texture, and complex makeup patterns on the patches during the transfer process, generating a realistic appearance. Finally, we suggest dealing with an image’s facial and background regions independently, and separating them by introducing face parsing maps into both the generator and discriminators. This prevents the alteration of unrelated content and mitigates the negative impact of background information during makeup transfer and removal. Extensive experiments demonstrate our method’s effectiveness with light and extreme makeup styles.

Graphical abstract

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Availability of data and materials

Data cannot be shared openly due to one of the datasets I’m using is private

References

  1. Tong WS, Tang CK, Brown MS et al (2007) Example-based cosmetic transfer. In: 15th Pacific conference on computer graphics and applications (PG’07). IEEE, pp 211–218

  2. Guo D, Sim T (2009) Digital face makeup by example. In: 2009 IEEE conference on computer vision and pattern recognition. IEEE, pp 73–79

  3. Li C, Zhou K, Lin S (2015) Simulating makeup through physics-based manipulation of intrinsic image layers. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp 4621–4629

  4. Li T, Qian R, Dong C et al (2018) Beautygan: Instance-level facial makeup transfer with deep generative adversarial network. In: Proceedings of the 26th ACM international conference on multimedia. pp 645–653

  5. Jiang W, Liu S, Gao C et al (2020) Psgan: Pose and expression robust spatial-aware gan for customizable makeup transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 5194–5202

  6. Deng H, Han C, Cai H et al (2021) Spatially-invariant style-codes controlled makeup transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 6549–6557

  7. Gu Q, Wang G, Chiu MT et al (2019) Ladn: Local adversarial disentangling network for facial makeup and de-makeup. In: Proceedings of the IEEE/CVF international conference on computer vision. pp 10481–10490

  8. Nguyen T, Tran AT, Hoai M (2021) Lipstick ain’t enough: beyond color matching for in-the-wild makeup transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 13305–13314

  9. Liu L, Xing J, Liu S et al (2014) Wow! you are so beautiful today! ACM Trans Multimedia Comput, Commun. Appl (TOMM) 11(1s):1–22

    Google Scholar 

  10. Scherbaum K, Ritschel T, Hullin M et al (2011) Computer-suggested facial makeup. Comput Graph Forum 30(2):485-492

    Article  Google Scholar 

  11. Gao M, Wang P (2023) Personalized facial makeup transfer based on outline correspondence. Comput Animation Virtual Worlds 35(1):e2199

    Article  MathSciNet  Google Scholar 

  12. Tiwari H, Subramanian VK, Chen YS (2023) Real-time self-supervised achromatic face colorization. Vis Comput 39(12):6521–6536

    Article  Google Scholar 

  13. Yuan YJ, Han X, He Y, et al (2024) Munerf: Robust makeup transfer in neural radiance fields. IEEE Trans Vis Comput Graph 1–12

  14. Li M, Yu W, Liu Q et al (2023) Hybrid transformers with attention-guided spatial embeddings for makeup transfer and removal. IEEE Trans Circuits Syst Video Technol 34(4):2876–2890

    Article  Google Scholar 

  15. Zhang F, Liang X, Sun Y et al (2022) Pofmakeup: A style transfer method for peking opera makeup. Comput Electr Eng 104:108459

    Article  Google Scholar 

  16. Yan Q, Guo C, Zhao J et al (2023) Beautyrec: Robust, efficient, and component-specific makeup transfer. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 1102–1110

  17. Sun Z, Chen Y, Xiong S (2022) Ssat: A symmetric semantic-aware transformer network for makeup transfer and removal. In: Proceedings of the AAAI conference on artificial intelligence. pp 2325–2334

  18. Lu X, Liu F, Rong Y et al (2024) Makeupdiffuse: a double image-controlled diffusion model for exquisite makeup transfer. The Vis Comput 1–17

  19. Goodfellow I, Pouget-Abadie J, Mirza M et al (2020) Generative adversarial networks. Commun ACM 63(11):139–144

    Article  MathSciNet  Google Scholar 

  20. Li S, Pu Y, Zhao Z et al (2024) Dual-path hypernetworks of style and text for one-shot domain adaptation. Appl Intell

  21. Zhu X, Cao X, Wang L et al (2024) Dccmf-gan: double cycle consistently constrained multi-feature discrimination gan for makeup transfer. Multimedia Tools Appl 83(4):44009–44022

    Google Scholar 

  22. Yuan QL, Zhang HL (2022) Ramt-gan: Realistic and accurate makeup transfer with generative adversarial network. Image Vis Comput 120:104400

    Article  Google Scholar 

  23. Xu W, Wang P, Yang X (2024) Frsegan: Free-style editable facial makeup transfer based on gan combined with transformer. Comput Animat Virtual Worlds 35(3):e2235

    Article  Google Scholar 

  24. Li Y, Huang H, Cao J et al (2020) Disentangled representation learning of makeup portraits in the wild. Int J Comput Vision 128:2166–2184

    Article  Google Scholar 

  25. Liu S, Jiang W, Gao C et al (2021) Psgan++: robust detail-preserving makeup transfer and removal. IEEE Trans Pattern Anal Mach Intell 44(11):8538–8551

    Article  Google Scholar 

  26. Chen Y, Xie J, Xue J et al (2024) A robust transformer gan for unpaired data makeup transfer. Concurr Computat: Pract Experience 36(9):e7994

    Article  Google Scholar 

  27. Yang C, He W, Xu Y et al (2022) Elegant: Exquisite and locally editable gan for makeup transfer. In: European conference on computer vision. Springer, pp 737–754

  28. Hao M, Gu G, Fu H et al (2022) Cumtgan: An instance-level controllable u-net gan for facial makeup transfer. Knowl-Based Syst 249:108996

  29. Fang S, Duan M, Li K et al (2022) Facial makeup transfer with gan for different aging faces. J Vis Commun Image Represent 85:103464

    Article  Google Scholar 

  30. Xu Z, Wu S, Jiao Q et al (2022) Tsev-gan: Generative adversarial networks with target-aware style encoding and verification for facial makeup transfer. Knowl-Based Syst 257:109958

    Article  Google Scholar 

  31. Chen H, Li W, Gao X et al (2023) Aep-gan: Aesthetic enhanced perception generative adversarial network for asian facial beauty synthesis. Appl Intell 53(17):20441–20468

    Article  Google Scholar 

  32. Deng J, Guo J, Xue N et al (2019) Arcface: Additive angular margin loss for deep face recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp 4690–4699

  33. Heusel M, Ramsauer H, Unterthiner T et al (2017) Gans trained by a two time-scale update rule converge to a local nash equilibrium. Adv Neural Inf Process Syst 30

Download references

Acknowledgements

This work is supported by the National Natural Science Foundation of China (General Program)(61772309), the Shandong Province Central Government-Guided Local Science and Technology Development Fund Project (YDZX2023079), the Jinan City ‘New Universities 20 Articles’ Scientific Research Leaders Studio (2021GXRC092), the Shandong Province Higher Education Institutions Youth Innovation Science and Technology Support Plan (2020KJN007), and the Shandong Province Key Research and Development Plan (2021SFGC0102).

Author information

Authors and Affiliations

  1. School of Computer Science and Technology, Shandong University of Finance and Economics, Jinan, 250014, Shandong, China

    Yuyan Chen, Jing Chi, Tianshu Shen, Bingyi You & Yanbing Wang

  2. Shandong Provincial Key Laboratory of Digital Media Technology, Shandong University of Finance and Economics, Jinan, 250014, Shandong, China

    Yuyan Chen, Jing Chi, Tianshu Shen, Bingyi You, Yanbing Wang & Caiming Zhang

  3. School of Software, Shandong University, Jinan, 250101, Shandong, China

    Caiming Zhang

Authors
  1. Yuyan Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jing Chi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tianshu Shen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Bingyi You
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Yanbing Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Caiming Zhang
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Yuyan Chen: Methodology, Software, Validation, Visualization, Writing-original draft. Jing Chi: Conceptualization, Funding acquisition, Resources, Supervision, Writing-review and editing. Tianshu Shen: Data curation, Formal analysis, Investigation, Writing-review and editing. Bingyi You: Data curation, Formal analysis, Investigation. Yanbing Wang: Data curation, Investigation. Caiming Zhang: Investigation, Project administration.

Corresponding author

Correspondence to Jing Chi.

Ethics declarations

Ethics approval

Not applicable

Consent to participate

Not applicable

Consent for publication

Not applicable

Conflict of interest

All the authors declare that there is no confict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Chen, Y., Chi, J., Shen, T. et al. A multi-granularity facial extreme makeup transfer and removal model with local-global collaboration. Appl Intell 54, 9741–9759 (2024). https://doi.org/10.1007/s10489-024-05692-8

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10489-024-05692-8

Keywords

Search

Navigation