More Web Proxy on the site http://driver.im/

research-article

Inpainting of Dunhuang Murals by Sparsely Modeling the Texture Similarity and Structure Continuity

Authors:

Qin ZouAuthors Info & Claims

Journal on Computing and Cultural Heritage (JOCCH), Volume 12, Issue 3

Article No.: 17, Pages 1 - 21

https://doi.org/10.1145/3280790

Published: 13 June 2019 Publication History

Abstract

Ancient mural paintings often suffer from damage such as color degradation, pigment peeling, and even large-area shedding. Image inpainting techniques are widely used to virtually repair these damages. Generally, the inpainting task can be very challenging when structures are totally missing within a large area. In this article, we study mural image inpainting by incorporating structure information collected from line drawings, and propose a line-drawings-guided inpainting algorithm for repairing the damaged murals of Mogao Grottoes, Dunhuang. Unlike traditional methods that use one single patch to inpaint the target area, the proposed method constructs the target patch with a linear combination of multiple candidate patches. These candidate patches are selected by a sparse model, where two special constraints have been introduced to guarantee the texture similarity and structure continuity. Experimental results demonstrate the effectiveness of the proposed method.

References

[1]

Michal Aharon, Michael Elad, and Alfred Bruckstein. 2006. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Transactions on Signal Processing 54, 11 (2006), 4311--4322.

Digital Library

[2]

Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B. Goldman. 2009. PatchMatch: A randomized correspondence algorithm for structural image editing. In SIGGRAPH. 1--11.

Digital Library

[3]

Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. 2000. Image inpainting. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’00). ACM Press/Addison-Wesley Publishing Co., New York, NY, 417--424.

Digital Library

[4]

Raphaël Bornard, Emmanuelle Lecan, Louis Laborelli, and Jean-Hugues Chenot. 2002. Missing data correction in still images and image sequences. In Proceedings of the 10th ACM International Conference on Multimedia (MULTIMEDIA’02). ACM, New York, NY, 355--361.

Digital Library

[5]

Folkmar Bornemann and Tom März. 2007. Fast image inpainting based on coherence transport. Journal of Mathematical Imaging and Vision 28, 3 (July 2007), 259--278.

Digital Library

[6]

Tony F. Chan and Jianhong Shen. 2001. Nontexture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation 12, 4 (2001), 436--449.

Digital Library

[7]

Long Chen, Jianda Chen, Qin Zou, Kai Huang, and Qingquan Li. 2017. Multi-view feature combination for ancient paintings chronological classification. Journal on Computing and Cultural Heritage 10, 2, Article 7 (March 2017), 15 pages.

Digital Library

[8]

B. Cornelis, T. Ruić, E. Gezels, A. Dooms, A. Piurica, L. Platića, J. Cornelis, M. Martens, M. De Mey, and I. Daubechies. 2013. Crack detection and inpainting for virtual restoration of paintings: The case of the Ghent Altarpiece. Signal Processing 93, 3 (2013), 605--619.

Digital Library

[9]

A. Criminisi, P. Perez, and K. Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on Image Processing 13, 9 (Sept. 2004), 1200--1212.

Digital Library

[10]

M. Daisy, P. Buyssens, D. Tschumperlé, and O. Lézoray. 2015. A smarter exemplar-based inpainting algorithm using local and global heuristics for more geometric coherence. In IEEE International Conference on Image Processing. 4622--4626.

[11]

Connelly Barnes, Ivan Belaunde, Dan Goldman, Eli Shechtman. 2010. Adobe Systems Inc. Retrieved April 4, 2010 from www.adobe.com/ technology/ graphics/content_aware_fill.html.

[12]

T. T. Dang, A. Beghdadi, and M. C. Larabi. 2014. A perceptual image completion approach based on a hierarchical optimization scheme. Signal Processing 103 (2014), 127--141.

Digital Library

[13]

Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B. Goldman, and Pradeep Sen. 2012. Image Melding: Combining inconsistent images using patch-based synthesis. ACM Transactions on Graphics 31, 4 (2012), 82:1--82:10.

Digital Library

[14]

Iddo Drori, Daniel Cohen-Or, and Hezy Yeshurun. 2003. Fragment-based image completion. ACM Transactions on Graphics 22, 3 (July 2003), 303--312.

Digital Library

[15]

A. A. Efros and T. K. Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the 7th IEEE International Conference on Computer Vision, Vol. 2. 1033--1038.

Digital Library

[16]

M. Elad and M. Aharon. 2006. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Transactions on Image Processing 15, 12 (2006), 3736--3745.

Digital Library

[17]

M. J. Fadili, J.-L. Starck, and F. Murtagh. 2009. Inpainting and zooming using sparse representations. The Computer Journal 52, 1 (Jan. 2009), 64--79.

Digital Library

[18]

Ankur G. Patel, Shashwat Kumar, and Ankit D. Prajapati. 2014. Improved exemplar based image inpainting using structure tensor. International Journal of Computer Applications 96, 15 (2014), 9--14.

[19]

Zhi Gao, Mo Shan, and Qingquan Li. 2015. Adaptive sparse representation for analyzing artistic style of paintings. Journal on Computing and Cultural Heritage 8, 4 (Aug. 2015), Article 22, 15 pages.

Digital Library

[20]

Shiming Ge, Kaixuan Xie, Shuying Li, and Rui Yang. 2016. Global image completion with joint sparse patch selection and optimal seam synthesis. Signal Processing 124, (2016), 147--155.

Digital Library

[21]

C. Guillemot, M. Turkan, O. Le Meur, and M. Ebdelli. 2013. Object removal and loss concealment using neighbor embedding methods. Image Communication 28, 10 (2013), 1405--1419.

Digital Library

[22]

Onur G. Guleryuz. 2006. Nonlinear approximation based image recovery using adaptive sparse reconstructions and iterated denoising-part I: Theory. IEEE Transactions on Image Processing 15, 3 (2006), 555--571.

Digital Library

[23]

Kaiming He and Jian Sun. 2014. Image completion approaches using the statistics of similar patches. IEEE Transactions on Pattern Analysis and Machine Intelligence 36, 12 (2014), 2423--2435.

[24]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2017) 36, 4 (2017), 107:1--107:14.

Digital Library

[25]

N. Komodakis and G. Tziritas. 2007. Image completion using efficient belief propagation via priority scheduling and dynamic pruning. IEEE Transactions on Image Processing 16, 11 (2007), 2649--2661.

Digital Library

[26]

Chuan Li and Michael Wand. 2016. Combining Markov random fields and convolutional neural networks for image synthesis. CoRR abs/1601.04589 (2016). http://arxiv.org/abs/1601.04589

[27]

Qingquan Li, Qin Zou, De Ma, Qian Wang, and Song Wang. 2018. Dating ancient paintings of Mogao Grottoes using deeply learnt visual codes. Science China Information Sciences 15, 6 (2018), 1--13.

[28]

Z. Li, H. He, H. Tai, Z. Yin, and F. Chen. 2015. Color-direction patch-sparsity-based image inpainting using multidirection features. IEEE Transactions on Image Processing 24, 3 (2015), 1138--1152.

[29]

Zhidan Li, Hongjie He, Zhongke Yin, and Fan Chen. 2014. A color-gradient patch sparsity based image inpainting algorithm with structure coherence and neighborhood consistency. Signal Processing 99, 6 (2014), 116--128.

Digital Library

[30]

Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, and Bryan Catanzaro. 2018. Image inpainting for irregular holes using partial convolutions. CoRR abs/1804.07723 (2018). http://arxiv.org/abs/1804.07723

[31]

Julien Mairal, Francis Bach, Jean Ponce, and Guillermo Sapiro. 2010. Online learning for matrix factorization and sparse coding. Journal of Machine Learning Research 11 (March 2010), 19--60. http://dl.acm.org/citation.cfm?id=1756006.1756008

[32]

J. Mairal, M. Elad, and G. Sapiro. 2007. Sparse representation for color image restoration. IEEE Transactions on Image Processing 17, 1 (2007), 53--69.

Digital Library

[33]

R. Martínez-Noriega, A. Roumy, and G. Blanchard. 2012. Exemplar-based image inpainting: Fast priority and coherent nearest neighbor search. In IEEE International Workshop on Machine Learning for Signal Processing. 1--6.

[34]

S. Masnou. 2002. Disocclusion: A variational approach using level lines. IEEE Transactions on Image Processing 11, 2 (Feb. 2002), 68--76.

Digital Library

[35]

Olivier Le Meur, Josselin Gautier, and Christine Guillemot. 2011. Examplar-based inpainting based on local geometry. In IEEE International Conference on Image Processing. 3401--3404.

[36]

Jayesh Patel and Tanuja K. Sarode. 2014. Exemplar based image inpainting with reduced search region. International Journal of Computer Applications 92, 12 (2014), 27--33.

[37]

Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei A. Efros. 2016. Context encoders: Feature learning by inpainting. CoRR abs/1604.07379 (2016). http://arxiv.org/abs/1604.07379

[38]

Peyr and Gabriel. 2009. Sparse modeling of textures. Journal of Mathematical Imaging and Vision 34, 1 (2009), 17--31.

Digital Library

[39]

S. Roth and M. J. Black. 2005. Fields of experts: A framework for learning image priors. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), Vol. 2. 860--867.

Digital Library

[40]

Leonid I. Rudin, Stanley Osher, and Emad Fatemi. 1992. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena 60 (1992), 259--268.

Digital Library

[41]

Bin Shen, Wei Hu, Yimin Zhang, and Yu Jin Zhang. 2009. Image inpainting via sparse representation. In IEEE International Conference on Acoustics, Speech and Signal Processing. 697--700.

Digital Library

[42]

Jianhong Shen and Tony F. Chan. 2002. Mathematical models for local nontexture inpaintings. SIAM Journal on Applied Mathematics 62, 3 (2002), 1019--1043.

Digital Library

[43]

S. Zahra Siadati, Farzin Yaghmaee, and Peyman Mahdavi. 2016. A new exemplar-based image inpainting algorithm using image structure tensors. In Electrical Engineering. 995--1001.

[44]

Yuhang Song, Chao Yang, Zhe Lin, Hao Li, Qin Huang, and C.-C. Jay Kuo. 2017. Image inpainting using multi-scale feature image translation. CoRR abs/1711.08590 (2017). http://arxiv.org/abs/1711.08590

[45]

Michael Strobel, Julia Diebold, and Daniel Cremers. 2014. Flow and Color Inpainting for Video Completion. 293--304.

[46]

Richard Szeliski, Ramin Zabih, Daniel Scharstein, Olga Veksler, Vladimir Kolmogorov, Aseem Agarwala, Marshall Tappen, and Carsten Rother. 2008. A comparative study of energy minimization methods for Markov random fields with smoothness-based priors. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 6 (2008), 1068--1080.

Digital Library

[47]

Tomohiro Takahashi, Katsumi Konishi, and Toshihiro Furukawa. 2012. Structured matrix rank minimization approach to image inpainting. Midwest Symposium on Circuits and Systems (2012), 860--863.

[48]

Mehmet Türkan and Christine Guillemot. 2013. Locally linear embedding based texture synthesis for image prediction and error concealment. In IEEE International Conference on Image Processing. 3009--3012.

[49]

Jing Wang, Ke Lu, Daru Pan, Ning He, and Bing Kun Bao. 2014. Letters: Robust object removal with an exemplar-based image inpainting approach. Neurocomputing 123 (2014), 150--155.

Digital Library

[50]

Zhou Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (2004), 600--612.

Digital Library

[51]

Li-Yi Wei and Marc Levoy. 2000. Fast texture synthesis using tree-structured vector quantization. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH’00). ACM Press/Addison-Wesley Publishing Co., New York, NY, 479--488.

Digital Library

[52]

Yonatan Wexler, Eli Shechtman, and Michal Irani. 2007. Space-time completion of video. IEEE Transactions on Pattern Analysis and Machine Intelligence 29, 3 (2007), 463--476.

Digital Library

[53]

Mang Xiao, Guangyao Li, Yunlan Tan, and Jie Qin. 2015. Image completion using similarity analysis and transformation. International Journal of Multimedia and Ubiquitous Engineering 10, 4 (2015), 193--204.

[54]

Z. Xu and J. Sun. 2010. Image inpainting by patch propagation using patch sparsity. IEEE Transactions on Image Processing 19, 5 (May 2010), 1153--1165.

Digital Library

[55]

Chao Yang, Xin Lu, Zhe Lin, Eli Shechtman, Oliver Wang, and Hao Li. 2016. High-resolution image inpainting using multi-scale neural patch synthesis. CoRR abs/1611.09969 (2016). http://arxiv.org/abs/1611.09969

[56]

Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S. Huang. 2018. Generative image inpainting with contextual attention. CoRR abs/1801.07892 (2018). http://arxiv.org/abs/1801.07892

[57]

M. Zontak and M. Irani. 2011. Internal statistics of a single natural image. In CVPR 2011. 977--984.

Digital Library

[58]

Qin Zou, Yu Cao, Qingquan Li, Chuanhe Huang, and Song Wang. 2014a. Chronological classification of ancient paintings using appearance and shape features. Pattern Recognition Letters 49 (2014), 146--154.

Digital Library

[59]

Qin Zou, Yu Cao, Qingquan Li, Qingzhou Mao, and Song Wang. 2014b. Automatic inpainting by removing fence-like structures in RGBD images. Machine Vision and Applications 25, 7 (2014), 1841--1858.

Digital Library

Cited By

Sari IDu W(2024)Weighted Similarity-Confidence Laplacian Synthesis for High-Resolution Art Painting CompletionApplied Sciences10.3390/app1406239714:6(2397)Online publication date: 12-Mar-2024
https://doi.org/10.3390/app14062397
Sumathi GUma Devi M(2024)Inpainting of damaged temple murals using edge- and line-guided diffusion patch GANFrontiers in Artificial Intelligence10.3389/frai.2024.14538477Online publication date: 6-Nov-2024
https://doi.org/10.3389/frai.2024.1453847
Yan YGouet-Zhang RHe HLei TZhang XJiang C(2024)Image Restoration Technology of Tang Dynasty Tomb Murals Using Adversarial Edge LearningJournal on Computing and Cultural Heritage10.1145/3674984Online publication date: 15-Jul-2024
https://doi.org/10.1145/3674984
Show More Cited By

Index Terms

Inpainting of Dunhuang Murals by Sparsely Modeling the Texture Similarity and Structure Continuity
1. Applied computing
  1. Arts and humanities
    1. Fine arts
    2. Performing arts

Recommendations

Inpainting Digital Dunhuang Murals with Structure-Guided Deep Network
Inpainting deteriorated regions in digital Dunhuang murals is important for Dunhuang mural content preservation. Algorithms of mural image inpainting help simplify the digital restoration process of the deteriorated murals. Most of the existing algorithms ...
Image inpainting by patch propagation using patch sparsity

This paper introduces a novel examplar-based inpainting algorithm through investigating the sparsity of natural image patches. Two novel concepts of sparsity at the patch level are proposed for modeling the patch priority and patch representation, which ...
Progressive Dunhuang murals inpainting based on recurrent feature reasoning network

The Dunhuang murals, notably the paintings on the interior walls of China’s Dunhuang Grottoes, are considered international cultural treasure. The Dunhuang murals were ruined to varied degrees after a lengthy period of erosion. Deep learning networks were ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal on Computing and Cultural Heritage

Journal on Computing and Cultural Heritage Volume 12, Issue 3

October 2019

158 pages

ISSN:1556-4673

EISSN:1556-4711

DOI:10.1145/3340676

Editor:
Franco Niccolucci
VAST-LAB at PIN, University of Florence, Italy

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 June 2019

Accepted: 01 November 2018

Revised: 01 September 2018

Received: 01 June 2018

Published in JOCCH Volume 12, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

Natural Science Foundation of Hubei Province
National Basic Research Program of China
Major Program of Key Research Institute on Humanities and Social Science of the Chinese Ministry of Education
National Natural Science Foundation of China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
511
Total Downloads

Downloads (Last 12 months)44
Downloads (Last 6 weeks)9

Reflects downloads up to 01 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Sari IDu W(2024)Weighted Similarity-Confidence Laplacian Synthesis for High-Resolution Art Painting CompletionApplied Sciences10.3390/app1406239714:6(2397)Online publication date: 12-Mar-2024
https://doi.org/10.3390/app14062397
Sumathi GUma Devi M(2024)Inpainting of damaged temple murals using edge- and line-guided diffusion patch GANFrontiers in Artificial Intelligence10.3389/frai.2024.14538477Online publication date: 6-Nov-2024
https://doi.org/10.3389/frai.2024.1453847
Yan YGouet-Zhang RHe HLei TZhang XJiang C(2024)Image Restoration Technology of Tang Dynasty Tomb Murals Using Adversarial Edge LearningJournal on Computing and Cultural Heritage10.1145/3674984Online publication date: 15-Jul-2024
https://doi.org/10.1145/3674984
Chen YDu WZhang S(2024)A text guided cross modal joint inpainting algorithm for ancient muralsExpert Systems with Applications: An International Journal10.1016/j.eswa.2024.125152258:COnline publication date: 15-Dec-2024
https://dl.acm.org/doi/10.1016/j.eswa.2024.125152
Sari IDu W(2023)Structure-Texture Consistent Painting Completion for ArtworksIEEE Access10.1109/ACCESS.2023.325289211(27369-27381)Online publication date: 2023
https://doi.org/10.1109/ACCESS.2023.3252892
Moral-Andrés FMerino-Gómez EReviriego PLombardi F(2023)Can Artificial Intelligence Reconstruct Ancient Mosaics?Studies in Conservation10.1080/00393630.2023.222779869:5(313-326)Online publication date: 2-Jul-2023
https://doi.org/10.1080/00393630.2023.2227798
Shao XYe HYang BCao F(2023)Two-stream coupling network with bidirectional interaction between structure and texture for image inpaintingExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.120700231:COnline publication date: 30-Nov-2023
https://dl.acm.org/doi/10.1016/j.eswa.2023.120700
Sun XJia JXu PNi JShi WLi B(2023)Structure-guided virtual restoration for defective silk cultural relicsJournal of Cultural Heritage10.1016/j.culher.2023.05.01662(78-89)Online publication date: Jul-2023
https://doi.org/10.1016/j.culher.2023.05.016
Zhou ZLiu XShang JHuang JLi ZJia H(2022)Inpainting Digital Dunhuang Murals with Structure-Guided Deep NetworkJournal on Computing and Cultural Heritage10.1145/3532867Online publication date: 25-Jul-2022
https://doi.org/10.1145/3532867
Zeng ZSun SSun JYin JShen Y(2022)Constructing a mobile visual search framework for Dunhuang murals based on fine-tuned CNN and ontology semantic distanceThe Electronic Library10.1108/EL-09-2021-017340:3(121-139)Online publication date: 16-Feb-2022
https://doi.org/10.1108/EL-09-2021-0173
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents