More Web Proxy on the site http://driver.im/

research-article

Free access

Geometry-aware face completion and editing

AUTHORs:

Ran HeAuthors Info & Claims

AAAI'19/IAAI'19/EAAI'19: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence

Article No.: 309, Pages 2506 - 2513

https://doi.org/10.1609/aaai.v33i01.33012506

Published: 27 January 2019 Publication History

PDF eReader Publisher Site

Abstract

Face completion is a challenging generation task because it requires generating visually pleasing new pixels that are semantically consistent with the unmasked face region. This paper proposes a geometry-aware Face Completion and Editing NETwork (FCENet) by systematically studying facial geometry from the unmasked region. Firstly, a facial geometry estimator is learned to estimate facial landmark heatmaps and parsing maps from the unmasked face image. Then, an encoder-decoder structure generator serves to complete a face image and disentangle its mask areas conditioned on both the masked face image and the estimated facial geometry images. Besides, since low-rank property exists in manually labeled masks, a low-rank regularization term is imposed on the disentangled masks, enforcing our completion network to manage occlusion area with various shape and size. Furthermore, our network can generate diverse results from the same masked input by modifying estimated facial geometry, which provides a flexible mean to edit the completed face appearance. Extensive experimental results qualitatively and quantitatively demonstrate that our network is able to generate visually pleasing face completion results and edit face attributes as well.

References

[1]

Arjovsky, M.; Chintala, S.; and Bottou, L. 2017. Wasserstein gan. In ICML.

[2]

Barnes, C.; Shechtman, E.; Finkelstein, A.; and Goldman, D. B. 2009. Patchmatch: A randomized correspondence algorithm for structural image editing. TOG 28(3):24.

Digital Library

[3]

Bertalmio, M.; Sapiro, G.; Caselles, V.; and Ballester, C. 2000. Image inpainting. In SIGGRAPH.

[4]

Bertalmio, M.; Vese, L.; Sapiro, G.; and Osher, S. 2003. Simultaneous structure and texture image inpainting. TIP 19(3):340-358.

[5]

Bulat, A., and Tzimiropoulos, G. 2017. How far are we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In ICCV.

[6]

Chen, Y.; Shen, C.; Wei, X.-S.; Liu, L.; and Yang, J. 2017. Adversarial posenet: A structure-aware convolutional network for human pose estimation. In ICCV.

[7]

Criminisi, A.; Pérez, P.; and Toyama, K. 2004. Region filling and object removal by exemplar-based image inpainting. TIP 13(9): 1200-1212.

Digital Library

[8]

Darabi, S.; Shechtman, E.; Barnes, C.; Goldman, D. B.; and Sen, P. 2012. Image melding: Combining inconsistent images using patch-based synthesis. TOG 31(4):82-1.

Digital Library

[9]

Elad, M.; Starck, J.-L.; Querre, P.; and Donoho, D. L. 2005. Simultaneous cartoon and texture image inpainting using morphological component analysis (mca). ACHA 19(3):340-358.

[10]

Goodfellow, I.; Pouget-Abadie, J.; Mirza, M.; Xu, B.; Warde-Farley, D.; Ozair, S.; Courville, A.; and Bengio, Y. 2014. Generative adversarial nets. In NIPS.

[11]

Gross, R.; Matthews, I.; Cohn, J.; Kanade, T.; and Baker, S. 2010. Multi-pie. IVC 28(5):807-813.

Digital Library

[12]

Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. TOG 26(3):4.

Digital Library

[13]

He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In CVPR.

[14]

Hu, Y.; Wu, X.; Yu, B.; He, R.; and Sun, Z. 2018. Poseguided photorealistic face rotation. In CVPR.

[15]

Huang, J.-B.; Kang, S. B.; Ahuja, N.; and Kopf, J. 2014. Image completion using planar structure guidance. TOG 36(4): 107.

[16]

Huang, R.; Zhang, S.; Li, T.; He, R.; et al. 2017. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In ICCV.

[17]

Iizuka, S.; Simo-Serra, E.; and Ishikawa, H. 2017. Globally and locally consistent image completion. TOG 36(4): 107.

Digital Library

[18]

Johnson, J.; Alahi, A.; and Fei-Fei, L. 2016. Perceptual losses for real-time style transfer and super-resolution. In ECCV.

[19]

Kingma, D. P., and Ba, J. 2015. Adam: A method for stochastic optimization. In ICLR.

[20]

Ledig, C.; Theis, L.; Huszár, F.; Caballero, J.; Cunningham, A.; Acosta, A.; Aitken, A.; Tejani, A.; Totz, J.; Wang, Z.; et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In CVPR.

[21]

Li, C., and Wand, M. 2016. Combining markov random fields and convolutional neural networks for image synthesis. In CVPR.

[22]

Li, Y.; Liu, S.; Yang, J.; and Yang, M.-H. 2017. Generative face completion. In CVPR.

[23]

Liu, S.; Yang, J.; Huang, C.; and Yang, M.-H. 2015a. Multiobjective convolutional learning for face labeling. In CVPR.

[24]

Liu, Z.; Luo, P.; Wang, X.; and Tang, X. 2015b. Deep learning face attributes in the wild. In ICCV.

[25]

Liu, P.; Qi, X.; He, P.; Li, Y.; Lyu, M. R.; and King, I. 2017. Semantically consistent image completion with fine-grained details. In arXiv preprint arXiv: 1711.09345.

[26]

Mao, X.; Li, Q.; Xie, H.; Lau, R. Y.; Wang, Z.; and Smolley, S. P. 2017. Least squares generative adversarial networks. In ICCV.

[27]

Newell, A.; Yang, K.; and Deng, J. 2016. Stacked hourglass networks for human pose estimation. In ECCV.

[28]

Pathak, D.; Krahenbuhl, P.; Donahue, J.; Darrell, T.; and Efros, A. A. 2016. Context encoders: Feature learning by inpainting. In CVPR.

[29]

Radford, A.; Metz, L.; and Chintala, S. 2016. Unsupervised representation learning with deep convolutional generative adversarial networks. In ICLR.

[30]

Shen, W, and Liu, R. 2017. Learning residual images for face attribute manipulation. In CVPR.

[31]

Yeh, R. A.; Chen, C.; Lim, T. Y.; Schwing, A. G.; Hasegawa-Johnson, M.; and Do, M. N. 2017. Semantic image inpainting with deep generative models. In CVPR.

[32]

Zhang, S.; He, R.; Sun, Z.; and Tan, T. 2016. Multi-task convnet for blind face inpainting with application to face verification. In ICB.

[33]

Zhang, S.; He, R.; Sun, Z.; and Tan, T. 2018. Demeshnet: Blind face inpainting for deep meshface verification. TIFS 13(3):637-647.

Cited By

Jin YWu JWang WYan YJiang JZheng J(2023)Cascading Blend Network for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/360895220:1(1-21)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3608952
Zheng HZhang ZZhang HYang YYan SWang MMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Deep Multi-Resolution Mutual Learning for Image InpaintingProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548030(6359-6367)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3548030
Jin JHu XHe KPeng TLiu JYang J(2021)Progressive Semantic Reasoning for Image InpaintingCompanion Proceedings of the Web Conference 202110.1145/3442442.3451142(68-76)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3442442.3451142

Index Terms

Geometry-aware face completion and editing
1. Computing methodologies

Index terms have been assigned to the content through auto-classification.

Recommendations

Lighting-aware face frontalization for unconstrained face recognition

Provide both lighting-recovered and lighting-normalized frontalized images.Basic frontalization with a generic 3D face model by the alignment of only five landmarks.Lighting recovered and normalized image filling by the symmetry of quotient image.LRFF ...
Facial Expression-Aware Face Frontalization
Computer Vision – ACCV 2016
Abstract
Face frontalization is a rising technique for view-invariant face analysis. It enables a non-frontal facial image to recover its general facial appearances to frontal view. A few pioneering works have been proposed very recently. However, face ...
Expression flow for 3D-aware face component transfer
SIGGRAPH '11: ACM SIGGRAPH 2011 papers

We address the problem of correcting an undesirable expression on a face photo by transferring local facial components, such as a smiling mouth, from another face photo of the same person which has the desired expression. Direct copying and blending ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Guide Proceedings

AAAI'19/IAAI'19/EAAI'19: Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence and Thirty-First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Educational Advances in Artificial Intelligence

January 2019

10088 pages

ISBN:978-1-57735-809-1

Copyright © 2019 Association for the Advancement of Artificial Intelligence.

Sponsors

Association for the Advancement of Artificial Intelligence

Publisher

AAAI Press

Publication History

Published: 27 January 2019

Qualifiers

Research-article
Research
Refereed limited

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
75
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)6

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Jin YWu JWang WYan YJiang JZheng J(2023)Cascading Blend Network for Image InpaintingACM Transactions on Multimedia Computing, Communications, and Applications10.1145/360895220:1(1-21)Online publication date: 13-Jul-2023
https://dl.acm.org/doi/10.1145/3608952
Zheng HZhang ZZhang HYang YYan SWang MMagalhães Jdel Bimbo ASatoh SSebe NAlameda-Pineda XJin QOria VToni L(2022)Deep Multi-Resolution Mutual Learning for Image InpaintingProceedings of the 30th ACM International Conference on Multimedia10.1145/3503161.3548030(6359-6367)Online publication date: 10-Oct-2022
https://dl.acm.org/doi/10.1145/3503161.3548030
Jin JHu XHe KPeng TLiu JYang J(2021)Progressive Semantic Reasoning for Image InpaintingCompanion Proceedings of the Web Conference 202110.1145/3442442.3451142(68-76)Online publication date: 19-Apr-2021
https://dl.acm.org/doi/10.1145/3442442.3451142

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents