Inpainting Applied to Facade Images: A Comparison of Algorithms
Authors: 
Willy Fritzsche, Steffen Goebbels, Simon Hensel, Marco Rußinski, Nils SchuchAuthors Info & Claims
Pages 410 - 422
Abstract
Many municipalities provide textured 3D city models for planning and simulation purposes. Usually, the textures are automatically taken from oblique aerial images. In these images, walls may be occluded by building parts, vegetation and other objects such as cars, traffic signs, etc. To obtain high quality models, these objects have to be segmented and then removed from facade textures. In this study, we investigate the ability of different non-specialized inpainting algorithms to continue facade patterns in occluded facade areas. To this end, non-occluded facade textures of a 3D city model are equipped with various masks simulating occlusions. Then, the performance of the algorithms is evaluated by comparing their results with the original images. In particular, very useful results are obtained with the neural network “DeepFill v2” trained with transfer learning on freely available facade datasets and the “Shift-Map” algorithm.
References
[1]
Bertalmio, M., Bertozzi, A., Shapiro, G.: Navier-Stokes, fluid dynamics, and image and video inpainting. In: Proceedings of CVPR 2001 (2001).
[2]
Cohen, A., Oswald, M.R., Liu, Y., Pollefeys, M.: Symmetry-aware façade parsing with occlusions. In: Proceedings of 2017 International Conference on 3D Vision (3DV), pp. 393–401 (2017).
[3]
Dai, D., Riemenschneider, H., Schmitt, G., Van, L.: Example-based facade texture synthesis. In: Proceedings of 2013 IEEE International Conference on Computer Vision (ICCV), pp. 1065–1072. IEEE Computer Society, Los Alamitos, CA (2013)
[4]
Doersch, C., Singh, S., Gupta, A., Sivic, J., Efros, A.: What makes Paris look like Paris? ACM Trans. Graph. 31(4:101), 1–9 (2012)
[5]
Genser, N., Seiler, J., Schilling, F., Kaup, A.: Signal and loss geometry aware frequency selective extrapolation for error concealment. In: Proceedings of 2018 Picture Coding Symposium (PCS), pp. 159–163 (2018)
[6]
Goebbels, S., Pohle-Fröhlich, R.: Roof reconstruction from airborne laser scanning data based on image processing methods. ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. III-3, 407–414 (2016)
[7]
Goebbels S and Pohle-Fröhlich R Automatic unfolding of CityGML buildings to paper models Geographies 2021 1 3 333-345
[8]
He K and Sun J Fitzgibbon A, Lazebnik S, Perona P, Sato Y, and Schmid C Statistics of patch offsets for image completion Computer Vision – ECCV 2012 2012 Heidelberg Springer 16-29
[9]
Hensel, S., Goebbels, S., Kada, M.: LSTM architectures for facade structure completion. In: Proceedings of GRAPP 2021, pp. 15–24 (2021)
[10]
Huang JB, Kang SB, Ahuja N, and Kopf J Image completion using planar structure guidance ACM Trans. Graph. 2014 33 4 1-10
[11]
Kaup A, Meisinger K, and Aach T Frequency selective signal extrapolation with applications to error concealment in image communication AEUE - Int. J. Electron. Commun. 2005 59 147-156
[12]
Korah T and Rasmussen C Spatiotemporal inpainting for recovering texture maps of occluded building facades IEEE Trans. Image Process. 2007 16 9 2262-2271
[13]
Kottler, B., Bulatov, D., Zhang, X.: Context-aware patch-based method for façade inpainting. In: Proceedings of GRAPP 2020, pp. 210–218 (2020)
[14]
Kottler, B., List, L., Bulatov, D., Weinmann, M.: 3GAN: a three-GAN-based approach for image inpainting applied to the reconstruction of occluded parts of building walls. In: Proceedings of VISAPP 2022 (2022)
[15]
Mehra S, Dogra A, Goyal B, Sharma AM, and Chandra R From textural inpainting to deep generative models: an extensive survey of image inpainting techniques J. Comput. Sci. 2020 16 1 35-49
[16]
Nazeri, K., Ng, E., Joseph, T., Qureshi, F.Z., Ebrahimi, M.: EdgeConnect: generative image inpainting with adversarial edge learning. arXiv Preprint: arXiv: 1901.00212 (2019)
[17]
Pritch, Y., Kav-Venaki, E., Peleg, S.: Shift-map image editing. In: Proceedings of 2009 IEEE International Conference on Computer Vision (ICCV), pp. 151–158. IEEE Computer Society, Los Alamitos, CA (2009)
[18]
Seiler J, Jonscher M, Schöberl M, and Kaup A Resampling images to a regular grid from a non-regular subset of pixel positions using frequency selective reconstruction IEEE Trans. Image Process. 2015 24 11 4540-4555
[19]
Shen H et al. Missing information reconstruction of remote sensing data: a technical review IEEE Geosci. Remote Sens. Mag. 2015 3 3 61-85
[20]
Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: Proceedings of 3rd International Conference on Learning Representations (ICLR) 2015, San Diego, CA, USA (2015)
[21]
Telea A An image inpainting technique based on the fast marching method J. Graph. Tools 2004 9 1 23-34
[22]
Tyleček R and Šára R Weickert J, Hein M, and Schiele B Spatial pattern templates for recognition of objects with regular structure Pattern Recognition 2013 Heidelberg Springer 364-374
[23]
Wang, Y., Tao, X., Qi, X., Shen, X., Jia, J.: Image inpainting via generative multi-column convolutional neural networks. In: Proceedings of 32nd International Conference on Neural Information Processing Systems, pp. 329–338. NIPS 2018. Curran Associates Inc., Red Hook, NY (2018)
[24]
Wonka P, Wimmer M, Sillion F, and Ribarsky W Instant architecture ACM Trans. Graph. 2003 22 3 669-677
[25]
Yeh YT, Breeden K, Yang L, Fisher M, and Hanrahan P Synthesis of tiled patterns using factor graphs ACM Trans. Graph. 2013 32 1 1-13
[26]
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Generative image inpainting with contextual attention. In: Proceedings of CVPR 2018, arXiv preprint arXiv:1801.07892 (2018)
[27]
Yu, J., Lin, Z., Yang, J., Shen, X., Lu, X., Huang, T.: Free-form image inpainting with gated convolution. In: Proceedings of CVPR 2019, arXiv preprint arXiv:1806.03589 (2019)
[28]
Zhang J, Fukuda T, and Yabuki N Automatic object removal with obstructed façades completion using semantic segmentation and generative adversarial inpainting IEEE Access 2021 9 117486-117495
[29]
Zhou B, Lapedriza A, Khosla A, Oliva A, and Torralba A A 10 million image database for scene recognition IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI) 2018 40 6 1452-14649
Index Terms
- Inpainting Applied to Facade Images: A Comparison of Algorithms
Index terms have been assigned to the content through auto-classification.
Recommendations
Inpainting on Digital Images Using Convolution based Method: A Comparison Study
ICBEB 2017: Proceedings of the 2017 International Conference on Biomedical Engineering and BioinformaticsThe goals and applications of inpainting are numerous, from the restoration of damaged paintings and photographs to the removal/replacement of selected object. Reconstruction of missing or damaged portions of images is an ancient practice used ...
The Conception of an Urban Energy Dashboard using 3D City Models
e-Energy '20: Proceedings of the Eleventh ACM International Conference on Future Energy SystemsIn this paper, we present a concept for an urban energy dashboard built using different Open Geospatial Consortium (OGC) standards integrated with 3D city models. With cities continuously pressed to cut upon their carbon emissions, it becomes vital to ...
Puzzle facade: a site-specific urban technological intervention
DIS Companion '14: Proceedings of the 2014 companion publication on Designing interactive systemsMedia façades are becoming part of our urban landscapes, challenging media artists and designers to create content for them. The diversity of resolution and spatial properties of these façades hasn't stopped content creators to adapt their projects to ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Jun 2022
718 pages
ISBN:978-3-031-09036-3
DOI:10.1007/978-3-031-09037-0
© Springer Nature Switzerland AG 2022.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 01 June 2022
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 22 Feb 2025