Abstract
With well-selected data, homogeneous diffusion inpainting can reconstruct images from sparse data with high quality. While 4K colour images of size \(3840\, \times \,2160\) can already be inpainted in real time, optimising the known data for applications like image compression remains challenging: Widely used stochastic strategies can take days for a single 4K image. Recently, a first neural approach for this so-called mask optimisation problem offered high speed and good quality for small images. It trains a mask generation network with the help of a neural inpainting surrogate. However, these mask networks can only output masks for the resolution and mask density they were trained for. We solve these problems and enable mask optimisation for high-resolution images through a neuroexplicit coarse-to-fine strategy. Additionally, we improve the training and interpretability of mask networks by including a numerical inpainting solver directly into the network. This allows to generate masks for 4K images in around 0.6 s while exceeding the quality of stochastic methods on practically relevant densities. Compared to popular existing approaches, this is an acceleration of up to four orders of magnitude.
This work has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement no. 741215, ERC Advanced Grant INCOVID).
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Agustsson, E., Timofte, R.: NTIRE 2017 challenge on single image super-resolution: dataset and study. In: Proceedings of the 2017 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, Honolulu, vol. 1, pp. 1122–1131 (2017)
Alt, T., Peter, P., Weickert, J.: Learning sparse masks for diffusion-based image inpainting. In: Pinho, A.J., Georgieva, P., Teixeira, L.F., Sánchez, J.A. (eds.) IbPRIA 2022. LNCS, vol. 13256, pp. 528–539. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-04881-4_42
Arbelaez, P., Maire, M., Fowlkes, C., Malik, J.: Contour detection and hierarchical image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 33(5), 898–916 (2011)
Belhachmi, Z., Bucur, D., Burgeth, B., Weickert, J.: How to choose interpolation data in images. SIAM J. Appl. Math. 70(1), 333–352 (2009)
Bonettini, S., Loris, I., Porta, F., Prato, M., Rebegoldi, S.: On the convergence of a linesearch based proximal-gradient method for nonconvex optimization. Inverse Prob. 33(5), 055005 (2017)
Carlsson, S.: Sketch based coding of grey level images. Signal Process. 15, 57–83 (1988)
Chizhov, V., Weickert, J.: Efficient data optimisation for harmonic inpainting with finite elements. In: Tsapatsoulis, N., Panayides, A., Theocharides, T., Lanitis, A., Pattichis, C., Vento, M. (eds.) CAIP 2021. LNCS, vol. 13053, pp. 432–441. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-89131-2_40
Dai, Q., Chopp, H., Pouyet, E., Cossairt, O., Walton, M., Katsaggelos, A.K.: Adaptive image sampling using deep learning and its application on X-ray fluorescence image reconstruction. IEEE Trans. Multimedia 22(10), 2564–2578 (2019)
Daropoulos, V., Augustin, M., Weickert, J.: Sparse inpainting with smoothed particle hydrodynamics. SIAM J. Appl. Math. 14(4), 1669–1704 (2021)
Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: ImageNet: a large-scale hierarchical image database. In: Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Miami, pp. 248–255 (2009)
Floyd, R.W., Steinberg, L.: An adaptive algorithm for spatial grey scale. In: Proceedings of the Society of Information Display, vol. 17, pp. 75–77 (1976)
Galić, I., Weickert, J., Welk, M., Bruhn, A., Belyaev, A., Seidel, H.P.: Image compression with anisotropic diffusion. J. Math. Imaging Vis. 31(2–3), 255–269 (2008)
Guillemot, C., Le Meur, O.: Image inpainting: overview and recent advances. IEEE Signal Process. Mag. 31(1), 127–144 (2014)
Hoeltgen, L., Setzer, S., Weickert, J.: An optimal control approach to find sparse data for Laplace interpolation. In: Heyden, A., Kahl, F., Olsson, C., Oskarsson, M., Tai, X.C. (eds.) Energy Minimisation Methods in Computer Vision and Pattern Recognition. Lecture Notes in Computer Science, vol. 8081, pp. 151–164. Springer, Heidelberg (2013). https://doi.org/10.1007/978-3-642-40395-8_12
Kämper, N., Weickert, J.: Domain decomposition algorithms for real-time homogeneous diffusion inpainting in 4K. In: Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing, Singapore, pp. 1680–1684 (2022)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Proceedings of the 3rd International Conference on Learning Representations, San Diego (2015)
Mainberger, M., et al.: Optimising spatial and tonal data for homogeneous diffusion inpainting. In: Bruckstein, A.M., ter Haar Romeny, B., Bronstein, A.M., Bronstein, M.M. (eds.) SSVM 2011. LNCS, vol. 6667, pp. 26–37. Springer, Heidelberg (2012). https://doi.org/10.1007/978-3-642-24785-9_3
Ochs, P., Chen, Y., Brox, T., Pock, T.: iPiano: inertial proximal algorithm for nonconvex optimization. SIAM J. Imag. Sci. 7(2), 1388–1419 (2014)
Pennebaker, W.B., Mitchell, J.L.: JPEG: Still Image Data Compression Standard. Springer, New York (1992)
Peter, P.: A Wasserstein GAN for joint learning of inpainting and its spatial optimisation. arXiv:2202.05623 [eess.IV] (2022)
Peter, P., Schrader, K., Alt, T., Weickert, J.: Deep spatial and tonal data optimisation for homogeneous diffusion inpainting. arXiv:2208.14371 [eess.IV] (2022)
Vašata, D., Halama, T., Friedjungová, M.: Image inpainting using Wasserstein generative adversarial imputation network. In: Farkaš, I., Masulli, P., Otte, S., Wermter, S. (eds.) ICANN 2021. LNCS, vol. 12892, pp. 575–586. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-86340-1_46
Weickert, J., Welk, M.: Tensor field interpolation with PDEs. In: Weickert, J., Hagen, H. (eds.) Visualization and Processing of Tensor Fields, pp. 315–325. Springer, Heidelberg (2006). https://doi.org/10.1007/3-540-31272-2_19
Wendland, H.: Numerical Linear Algebra: An Introduction. Cambridge University Press, Cambridge (2017)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Schrader, K., Peter, P., Kämper, N., Weickert, J. (2023). Efficient Neural Generation of 4K Masks for Homogeneous Diffusion Inpainting. In: Calatroni, L., Donatelli, M., Morigi, S., Prato, M., Santacesaria, M. (eds) Scale Space and Variational Methods in Computer Vision. SSVM 2023. Lecture Notes in Computer Science, vol 14009. Springer, Cham. https://doi.org/10.1007/978-3-031-31975-4_2
Download citation
DOI: https://doi.org/10.1007/978-3-031-31975-4_2
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-31974-7
Online ISBN: 978-3-031-31975-4
eBook Packages: Computer ScienceComputer Science (R0)