Cited By
View all- Lu XLu RZhao WMa E(2023)Facial image inpainting for big data using an effective attention mechanism and a convolutional neural networkFrontiers in Neurorobotics10.3389/fnbot.2022.111162116Online publication date: 12-Jan-2023
Progressive Semantic Reasoning for Image Inpainting
Authors: 
Junjie Jin, Xinrong Hu, Kai He, Tao Peng, Junping Liu, Jie YangAuthors Info & Claims
Pages 68 - 76
Abstract
Image inpainting aims to reconstruct the missing or unknown region for a given image. As one of the most important topics from image processing, this task has attracted increasing research interest over the past few decades. Learning-based methods have been employed to solve this task, and achieved superior performance. Nevertheless, existing methods often produce artificial traces, due to the lack of constraints on image characterization under different semantics. To accommodate this issue, we propose a novel artistic Progressive Semantic Reasoning (PSR) network in this paper, which is composed of three shared parameters from the generation network superposition. More precisely, the proposed PSR algorithm follows a typical end-to-end training procedure, that learns low-level semantic features and further transfers them to a high-level semantic network for inpainting purposes. Furthermore, a simple but effective Cross Feature Reconstruction (CFR) strategy is proposed to tradeoff semantic information from different levels. Empirically, the proposed approach is evaluated via intensive experiments using a variety of real-world datasets. The results confirm the effectiveness of our algorithm compared with other state-of-the-art methods. The source code can be found from https://github.com/sfwyly/PSR-Net.
References
[1]
Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. 2009. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing. In ACM SIGGRAPH 2009 Papers (New Orleans, Louisiana) (SIGGRAPH ’09). Association for Computing Machinery, New York, NY, USA, Article 24, 11 pages. https://doi.org/10.1145/1576246.1531330
[2]
A. Criminisi, P. Perez, and K. Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (2004), 1200–1212. https://doi.org/10.1109/TIP.2004.833105
[3]
D. Ding, S. Ram, and J. J. Rodríguez. 2019. Image Inpainting Using Nonlocal Texture Matching and Nonlinear Filtering. IEEE Transactions on Image Processing 28, 4 (2019), 1705–1719. https://doi.org/10.1109/TIP.2018.2880681
[4]
Carl Doersch, Saurabh Singh, Abhinav Gupta, Josef Sivic, and Alexei A. Efros. 2012. What Makes Paris Look like Paris?ACM Trans. Graph. 31, 4, Article 101 (July 2012), 9 pages. https://doi.org/10.1145/2185520.2185597
[5]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020. Generative Adversarial Networks. Commun. ACM 63, 11 (Oct. 2020), 139–144. https://doi.org/10.1145/3422622
[6]
Yibing Song Wei Huang Hongyu Liu, Bin Jiangand Chao Yang. 2020. Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations. In Proceedings of the European Conference on Computer Vision. https://doi.org/10.1007/978-3-030-58536-5_43
[7]
J. Hu, L. Shen, S. Albanie, G. Sun, and E. Wu. 2020. Squeeze-and-Excitation Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8(2020), 2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372
[8]
Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and Locally Consistent Image Completion. ACM Trans. Graph. 36, 4, Article 107 (July 2017), 14 pages. https://doi.org/10.1145/3072959.3073659
[9]
Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. International Conference on Learning Representations (12 2014). https://arxiv.org/abs/1412.6980v7
[10]
J. Li, F. He, L. Zhang, B. Du, and D. Tao. 2019. Progressive Reconstruction of Visual Structure for Image Inpainting. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 5961–5970. https://doi.org/10.1109/ICCV.2019.00606
[11]
J. Li, N. Wang, L. Zhang, B. Du, and D. Tao. 2020. Recurrent Feature Reasoning for Image Inpainting. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7757–7765. https://doi.org/10.1109/CVPR42600.2020.00778
[12]
X. Li, W. Wang, X. Hu, and J. Yang. 2019. Selective Kernel Networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 510–519. https://doi.org/10.1109/CVPR.2019.00060
[13]
Guilin Liu, Fitsum A Reda, Kevin J Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. In Proceedings of the European Conference on Computer Vision (ECCV). 85–100. https://doi.org/10.1007/978-3-030-01252-6_6
[14]
H. Liu, B. Jiang, Y. Xiao, and C. Yang. 2019. Coherent Semantic Attention for Image Inpainting. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 4169–4178. https://doi.org/10.1109/ICCV.2019.00427
[15]
Z. Liu, P. Luo, X. Wang, and X. Tang. 2015. Deep Learning Face Attributes in the Wild. In 2015 IEEE International Conference on Computer Vision (ICCV). 3730–3738. https://doi.org/10.1109/ICCV.2015.425
[16]
K. Nazeri, E. Ng, T. Joseph, F. Qureshi, and M. Ebrahimi. 2019. EdgeConnect: Structure Guided Image Inpainting using Edge Prediction. In 2019 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW). 3265–3274. https://doi.org/10.1109/ICCVW.2019.00408
[17]
D. Pathak, P. Krähenbühl, J. Donahue, T. Darrell, and A. A. Efros. 2016. Context Encoders: Feature Learning by Inpainting. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2536–2544. https://doi.org/10.1109/CVPR.2016.278
[18]
Rakshith Shetty, Mario Fritz, and Bernt Schiele. 2018. Adversarial Scene Editing: Automatic Object Removal from Weak Supervision. In Proceedings of the 32nd International Conference on Neural Information Processing Systems(NIPS’18). Curran Associates Inc., Red Hook, NY, USA, 7717–7727. https://doi.org/10.5555/3327757.3327869
[19]
Linsen Song, Jie Cao, Lingxiao Song, Yibo Hu, and Ran He. 2019. Geometry-aware face completion and editing. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 2506–2513. https://doi.org/10.1609/aaai.v33i01.33012506
[20]
Ning Wang, Jingyuan Li, Lefei Zhang, and Bo Du. 2019. MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting. In IJCAI. 3748–3754. https://doi.org/10.24963/ijcai.2019/520
[21]
Jie Yang, Zhiquan Qi, and Yong Shi. 2020. Learning to incorporate structure knowledge for image inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34. 12605–12612. https://doi.org/10.20944/preprints202002.0125.v1
[22]
R. A. Yeh, C. Chen, T. Y. Lim, A. G. Schwing, M. Hasegawa-Johnson, and M. N. Do. 2017. Semantic Image Inpainting with Deep Generative Models. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 6882–6890. https://doi.org/10.1109/CVPR.2017.728
[23]
Z. Yi, Q. Tang, S. Azizi, D. Jang, and Z. Xu. 2020. Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7505–7514. https://doi.org/10.1109/CVPR42600.2020.00753
[24]
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. Huang. 2019. Free-Form Image Inpainting With Gated Convolution. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). 4470–4479. https://doi.org/10.1109/ICCV.2019.00457
[25]
J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. 2018. Generative Image Inpainting with Contextual Attention. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. 5505–5514. https://doi.org/10.1109/CVPR.2018.00577
[26]
Tao Yu, Zongyu Guo, Xin Jin, Shilin Wu, Zhibo Chen, Weiping Li, Zhizheng Zhang, and Sen Liu. 2020. Region Normalization for Image Inpainting. In AAAI. 12733–12740. https://doi.org/10.1609/aaai.v34i07.6967
[27]
Y. Zeng, J. Fu, H. Chao, and B. Guo. 2019. Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 1486–1494. https://doi.org/10.1109/CVPR.2019.00158
[28]
Yu Zeng, Zhe Lin, Jimei Yang, Jianming Zhang, Eli Shechtman, and Huchuan Lu. 2020. High-resolution image inpainting with iterative confidence feedback and guided upsampling. In European Conference on Computer Vision. Springer, 1–17. https://doi.org/10.1007/978-3-030-58529-7_1
[29]
B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba. 2018. Places: A 10 Million Image Database for Scene Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 40, 6(2018), 1452–1464. https://doi.org/10.1109/TPAMI.2017.2723009
[30]
T. Zhou, C. Ding, S. Lin, X. Wang, and D. Tao. 2020. Learning Oracle Attention for High-Fidelity Face Completion. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 7677–7686. https://doi.org/10.1109/CVPR42600.2020.00770
Recommendations
Image Retrieval Using Digital Image Inpainting Techniques
Image retrieval is an inverse problem in digital image processing. In this paper, the authors deal with restoration of image using digitally image inpainting methods. In this inpainting technique, one can extract a missing an important part or can ...
Multistage semantic‐aware image inpainting with stacked generator networks
AbstractDeep learning has been widely applied into image inpainting. However, traditional image processing methods (i.e., patch‐based and diffusion‐based methods) generally fail to produce visually natural contents and semantically reasonable structures ...
Progressive Image Inpainting Based on Wavelet Transform
Currently, the automatic image inpainting methods emphasize the inpainting techniques either globally or locally. They didn't consider the merits of global and local techniques to compensate each other. On the contrary, the artists fixed an image in ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
April 2021
726 pages
ISBN:9781450383134
DOI:10.1145/3442442
Copyright © 2021 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 03 June 2021
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
WWW '21
Sponsor:
Acceptance Rates
Overall Acceptance Rate 1,899 of 8,196 submissions, 23%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 184Total Downloads
- Downloads (Last 12 months)11
- Downloads (Last 6 weeks)1
Reflects downloads up to 01 Jan 2025
Other Metrics
Citations
Cited By
View all- Lu XLu RZhao WMa E(2023)Facial image inpainting for big data using an effective attention mechanism and a convolutional neural networkFrontiers in Neurorobotics10.3389/fnbot.2022.111162116Online publication date: 12-Jan-2023
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format