More Web Proxy on the site http://driver.im/

research-article

Multistage semantic‐aware image inpainting with stacked generator networks

Authors:

Imran MumtazAuthors Info & Claims

International Journal of Intelligent Systems, Volume 37, Issue 2

Pages 1599 - 1617

https://doi.org/10.1002/int.22687

Published: 28 December 2021 Publication History

Abstract

Deep learning has been widely applied into image inpainting. However, traditional image processing methods (i.e., patch‐based and diffusion‐based methods) generally fail to produce visually natural contents and semantically reasonable structures due to ineffectively processing the high‐level semantic information of images. To solve the problem, we propose a stacked generator networks assisted by patch discriminator for image inpainting by multistage. In the proposed method, our generator network mainly consists of three‐layer stacked encoder‐decoder architecture, which could fuse different level feature information and achieve image inpainting via a coarse‐to‐fine hierarchical representation. Meanwhile, we split the masked image into different patches in each layer, which could effectively enlarge the receptive field and extract more useful features of images. Moreover, the patch discriminator is introduced to judge the patches of inpainting image are real or fake. In this way, our network can effectively utilize the semantic information to complete a fine result. Furthermore, both perceptual loss and style loss are used to improve the inpainting results in verse. Experimental results on Places2 and Paris StreetView illustrate that our approach could generate high‐quality inpainting results, and our method is more effective than the existing image inpainting methods.

References

[1]

Guillemot C, LeMeur O. Image inpainting: Overview and recent advances. IEEE Signal Process Mag. 2013;31(1):127‐144.

[2]

Li X, Shen H, Zhang L, Zhang H, Yuan Q, Yang G. Recovering quantitative remote sensing products contaminated by thick clouds and shadows using multitemporal dictionary learning. IEEE Trans Geosci Remote Sens. 2014;52(11):7086‐7098.

[3]

Darabi S, Shechtman E, Barnes C. Image melding: combining inconsistent images using patch‐based synthesis. ACM Trans Graph. 2012;31(4):1‐10.

Digital Library

[4]

Huang JB, Kang SB, Ahuja N. Image completion using planar structure guidance. ACM Trans Graph. 2014;33(4):1‐10.

Digital Library

[5]

Barnes C, Shechtman E, Finkelstein A. PatchMatch: a randomized correspondence algorithm for structural image editing. ACM Trans Graph. 2009;28(3):24.

Digital Library

[6]

Wilczkowiak M, Brostow GJ, Tordoff B. Hole filling through photomontage.Proceedings of the British Machine Vision Conference. 2005.

[7]

Ballester C, Bertalmio M, Caselles V. Filling‐in by joint interpolation of vector fields and gray levels. IEEE Trans Image Process. 2001;10(8):1200‐1211.

Digital Library

[8]

Bertalmio M, Sapiro G, Caselles V. Image Inpainting. ACM Press/Addison‐Wesley Publishing Co. 2000:417‐424.

[9]

Esedoglu S. Digital inpainting based on the Mumford‐Shah‐Euler image model. Eur J Appl Math. 2003;13(4):353‐370.

[10]

Dong L, Sun X, Feng W. Image compression with edge‐based inpainting. IEEE Trans Circuits Syst Video Technol. 2007;17(10):1273‐1287.

[11]

Efros AA, Freeman WT. Image Quilting for Texture Synthesis and Transfer. Computer Science Division, University of California; 2001:341‐346.

[12]

Iizuka S, Simo‐Serra E, Ishikawa H. Globally and locally consistent image completion. ACM Trans Graph. 2017; 36(4): 1‐14.

Digital Library

[13]

Li Y, Liu S, Yang J, H M. Generative face inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018:5505‐5514.

[14]

Pathak D, Krahenbuhl P, Donahue J. Context encoders: feature learning by inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:2536‐2544.

[15]

Yeh R, Chen C, Lim TY. Semantic image inpainting with perceptual and contextual losses. arXiv preprint arXiv:1607.07539, 2016.

[16]

Brock A, Donahue J, Simonyan K. Large scale GAN training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096, 2018.

[17]

Goodfellow I, Pouget‐Abadie J, Mirza M, Courville A, Bengio Y. Generative adversarial nets. Advances in Neural Information Processing Systems. 2014:27.

[18]

Karras T, Aila T, Laine S, Lehtinen JProgressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.

[19]

Yan Z, Yan X, Li M, Zuo W, Shan S. Shift‐net: Image inpainting via deep feature rearrangement. Proceedings of the European Conference on Computer Vision. 2018:1‐17.

[20]

Jiahui Y, Zhe L, Jimei Y, Xiaohui S, Xin L, S HT. Generative image inpainting with contextual attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018:5505‐5514.

[21]

K He, X Zhang, S Ren, and J Sun. Deep residual learning for image recognitions. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:770‐778.

[22]

Isola P, Zhu JY, Zhou T, Efros AA. Image‐to‐image translation with conditional adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:1125‐1134.

[23]

Johnson J, Alahi A, Fei‐Fei L. Perceptual losses for real‐time style transfer and super‐resolution. European Conference on Computer Vision. 2016:694‐711.

[24]

Gatys LA, Ecker AS, Bethge M. Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016:2414‐2423.

[25]

Huang Y‐F, Shih L‐P, Tsai C‐H, Shen G‐T. Describing video scenarios using deep learning techniques. Int J Intell Syst. 2021;36(6):2465‐2490.

Digital Library

[26]

Lee CY, Chen Y‐PP. Descriptive prediction of drug side‐effects using a hybrid deep learning model. Int J Intell Syst. 2021;36(6):2491‐2510.

Digital Library

[27]

Li X, Lv J, Yi Z. An efficient representation‐based method for boundary point and outlier detection. IEEE Trans Neural Netw Learn Syst. 2016;29(1):51‐62.

[28]

Xiaojie L, Jiancheng L, Zhang Y. Outlier detection using structural scores in a high‐dimensional space. IEEE Trans Cybern. 2018;50(5):2302‐2310.

[29]

Feng G, Canghong S, Xiaojie L, Xi W, Jiliu Z, Jiancheng L. Image segmentation of nasopharyngeal carcinoma using 3D CNN with long‐range skip connection and multi‐scale feature pyramid. Soft Comput. 2020;24(16):12671‐12680.

[30]

Yang C, Lu X, Lin Z, Shechtman E, Wang O, Li H. High‐resolution image inpainting using multi‐scale neural patch synthesis. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:6721‐6729.

[31]

Simonyan K, and ZA. Very. Very deep convolutional networks for large‐scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

[32]

Liu G, Reda FA, Shih KJ, Wang T‐C, Tao A, Catanzaro B. Image inpainting for irregular holes using partial convolutions. European Conference on Computer Vision. 2018: 85‐100.

[33]

Yu J, Lin Z, Yang J, Shen X, Lu X, Huang TS. Free‐form image inpainting with gated convolution. IEEE International Conference on Computer Vision. 2019:4471‐4480.

[34]

Liu H, Jiang B, Xiao Y, Yang C. Coherent semantic attention for image inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019:4170‐4179.

[35]

Mao X, Li Q, Xie H, Lau RY, Wang Z, Smolley SP. Least squares generative adversarial networks. IEEE International Conference on Computer Vision. 2017:2813‐2821.

[36]

Zhu J‐Y, Park T, Isola P, Efros AA. Unpaired image‐to‐image translation using cycle‐consistent adversarial networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017:2223‐2232.

[37]

Zhou B, Lapedriza A, Khosla A, Oliva A, Torralba A. Places: A 10 million image database for scene recognition. IEEE Trans Pattern Anal Mach Intell. 2017;40(6):1452‐1464.

[38]

Doersch C, Singh S, Gupta A, Sivic J, Efros A. What makes Paris look like Paris? ACM Trans Graph. 2012;31(4):101.

Digital Library

[39]

Nazeri K, Ng E, Joseph T, Qureshi FZ, Ebrahimi M. Edgeconnect: Generative image inpainting with adversarial edge learning. arXiv preprint arXiv:1901.00212, 2019.

[40]

Jingyuan L, Ning W, Lefei Z, Bo D, Dacheng T. Recurrent feature reasoning for image inpainting. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2020:7760‐7768.

Cited By

Zhang YLin XYang HHe JQing LHe XLi YChen H(2024)A Multi-Attention Feature Distillation Neural Network for Lightweight Single Image Super-ResolutionInternational Journal of Intelligent Systems10.1155/2024/32552332024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/3255233
Chai XSong SGan ZLong GTian YHe X(2024)CSENMTExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122562241:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122562
Zhai HPan XYang YJiang JLi Q(2023)Two-Stage Focus Measurement Network with Joint Boundary Refinement for Multifocus Image FusionInternational Journal of Intelligent Systems10.1155/2023/41559482023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/4155948
Show More Cited By

Recommendations

Image inpainting based on deep learning: A review
Abstract
Image inpainting is an important research direction in the study of computer vision, and is widely used in image editing and photo inpainting etc. Traditional image inpainting algorithms are often difficult to deal with large-scale image deletion,...
Highlights
- Reviewing the image inpainting on deep learning of the past 15 years.
- Summarizing the prominent deep learning methods especially in feature fusion.
- Pointing out the problem and difficulty in image inpainting task.
- Providing ...
Image Retrieval Using Digital Image Inpainting Techniques

Image retrieval is an inverse problem in digital image processing. In this paper, the authors deal with restoration of image using digitally image inpainting methods. In this inpainting technique, one can extract a missing an important part or can ...
Image Inpainting: A Review
Abstract
Although image inpainting, or the art of repairing the old and deteriorated images, has been around for many years, it has recently gained even more popularity, because of the recent development in image processing techniques. With the improvement ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image International Journal of Intelligent Systems

International Journal of Intelligent Systems Volume 37, Issue 2

February 2022

749 pages

ISSN:0884-8173

DOI:10.1002/int.v37.2

Issue’s Table of Contents

© 2021 Wiley Periodicals LLC.

Publisher

John Wiley and Sons Ltd.

United Kingdom

Publication History

Published: 28 December 2021

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhang YLin XYang HHe JQing LHe XLi YChen H(2024)A Multi-Attention Feature Distillation Neural Network for Lightweight Single Image Super-ResolutionInternational Journal of Intelligent Systems10.1155/2024/32552332024Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1155/2024/3255233
Chai XSong SGan ZLong GTian YHe X(2024)CSENMTExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.122562241:COnline publication date: 1-May-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.122562
Zhai HPan XYang YJiang JLi Q(2023)Two-Stage Focus Measurement Network with Joint Boundary Refinement for Multifocus Image FusionInternational Journal of Intelligent Systems10.1155/2023/41559482023Online publication date: 1-Jan-2023
https://dl.acm.org/doi/10.1155/2023/4155948
Wei ZMin WWang QLiu QZhao H(2022)ECNFPExpert Systems with Applications: An International Journal10.1016/j.eswa.2022.118070207:COnline publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1016/j.eswa.2022.118070

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents