More Web Proxy on the site http://driver.im/

research-article

Intensity-guided edge-preserving depth upsampling through weighted L0 gradient minimization

Authors:

Joongkyu KimAuthors Info & Claims

Journal of Visual Communication and Image Representation, Volume 42, Issue C

Pages 132 - 144

https://doi.org/10.1016/j.jvcir.2016.11.009

Published: 01 January 2017 Publication History

Abstract

Display Omitted Intensity-guided edge-preserving depth upsampling.Weighted L0 gradient minimization.Alternating minimization and half-quadratic splitting.Suppressing edge blurring and texture copying artifacts. Depth is an important visual cue to perceive real-world scenes. Although a time-of-flight (ToF) depth camera can provide depth information in dynamic scenes, captured depth images are often noisy and of low resolution. In this paper, we propose an intensity-guided edge-preserving depth upsampling method through weighted L0 gradient minimization to enhance both resolution and visual quality of depth images. Guided by the high-resolution intensity image, we perform optimization to preserve boundaries of objects. We apply L0 gradient to the regularization term, and compute its weight from both intensity and depth images. We optimize the objective function using alternating minimization and half-quadratic splitting. Experimental results on Middlebury 2005, 2014, and real-world scene datasets demonstrate that the proposed method produces boundary-preserving depth upsampling results and outperforms state-of-the-art ones in terms of accuracy.

References

[1]

D. Scharstein, R. Szeliski, A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, Int. J. Comput. Vis., 47 (2002) 7-42.

Digital Library

[2]

P. Thanusutiyabhorn, P. Kanongchaiyos, W. Mohammed, Image-based 3D laser scanner, in: Proc. Int. Conf. Elect. Eng./Electron., Comput., Telecommun., Inform. Technol., 2011, pp. 975978.

[3]

A. Kolb, E. Barth, R. Koch, R. Larsen, Time-of-flight cameras in computer graphics, Comput. Graph. Forum, 29 (2010) 141-159.

[4]

S. Gokturk, H. Yalcin, C. Bamji, A time-of-flight depth sensor-system description, issues and solutions, in: Proc. IEEE CVPR Workshop, 2004, pp. 35.

Digital Library

[5]

Available: <http://www.mesa-imaging.ch/>.

[6]

K. Guo, X. Yang, H. Zha, W. Lin, S. Yu, Multiscale semilocal interpolation with antialiasing, IEEE Trans. Image Process., 21 (2012) 1176-1190.

Digital Library

[7]

X. Jiang, B. Sheng, W. Lin, P. Li, L. Ma, R. Shen, Antialiased super-resolution with parallel high-frequency synthesis, Multimed. Tools Appl. (2015) 1-18.

Digital Library

[8]

K. Guo, X. Yang, W. Lin, R. Zhang, S. Yu, Learning-based super-resolution method with a combining of both global and local constraints, IET Image Process., 6 (2012) 337-344.

[9]

C. Dong, C.C. Loy, K. He, X. Tang, Learning a Deep Convolutional Network for Image Super-Resolution, in: Proc. ECCV, 2014, pp. 184-199.

[10]

G. Zhong, L. Yu, P. Zhou, Edge-preserving single depth image interpolation, in: Proc. IEEE VCIP, 2013, pp. 1-6.

[11]

S. Schuon, C. Theobalt, J. Davis, S. Thrun, High-quality scanning using time-of-flight depth superresolution, in: Proc. IEEE CVPR Workshop, 2008, pp. 1-7.

[12]

O.M. Aodha, N.D.F. Campbell, A. Nair, G.J. Brostow, Patch based synthesis for single depth image super-resolution, in: Proc. ECCV Workshop, 2012, pp. 71-84.

Digital Library

[13]

J. Xie, R. Feris, S.-S. Yu, M.-T. Sun, Joint Super Resolution and Denoising From a Single Depth Image, IEEE Trans. Multimedia, 17 (2015) 1525-1537.

[14]

J. Xie, R. Feris, M.-T. Sun, Edge-guided single depth image super resolution, IEEE Trans. Image Process., 25 (2016) 428-438.

[15]

T.-W. Hui, C.C. Loy, X. Tang, Depth map super resolution by deep multi-scale guidance, in: Proc. ECCV, 2016, pp. 353369.

[16]

J. Kopf, M.F. Cohen, D. Lischinski, M. Uyttendaele, Joint bilateral upsampling, ACM Trans. Graph., 26 (2007) 96:1-96:5.

Digital Library

[17]

A.K. Riemens, O.P. Gangwal, R-P.M. Berretty, Multistep joint bilateral depth upsampling, in: Proc. SPIE, 2009, pp. 1-12.

[18]

Q. Yang, R. Yang, J. Davis, D. Nister, Spatial-depth super resolution for range images, in: Proc. IEEE CVPR, 2007, pp. 1-8.

[19]

Q. Yang, K.-H. Tan, B. Culbertson, J. Apostolopoulos, Fusion of active and passive sensors for Fast 3-D capture, in: Proc. IEEE Int. Workshop MMSP, 2010, pp. 6974.

[20]

D. Chan, H. Buisman, C. Theobalt, S. Thrun, A noise-aware filter for real-time depth upsampling, in: Proc. ECCV Workshop, 2008, pp. 112.

[21]

F. Garcia, D. Aouada, B. Mirbach, T. Solignac, B. Ottersten, A new multi-lateral filter for real-time depth enhancement, in: Proc. AVSS, 2011, pp. 42-47.

Digital Library

[22]

Q. Yang, Fusion of median and bilateral filtering for range image upsampling, IEEE Trans. Image Process., 22 (2013) 4841-4852.

Digital Library

[23]

K. He, J. Sun, X. Tang, Guided image filtering, IEEE Trans. Pattern Anal. Mach. Intell., 35 (2013) 1397-1409.

Digital Library

[24]

D. Min, J. Lu, M. Do, Depth video enhancement based on weighted mode filtering, IEEE Trans. Image Process., 21 (2012) 1176-1190.

Digital Library

[25]

X. Zuo, J. Zheng, A refined weighted mode filtering approach for depth video enhancement, in: Proc. ICVRV, 2013, pp. 138144.

Digital Library

[26]

O. Choi, S.-W. Jung, A consensus-driven approach for structure and texture aware depth map upsampling, IEEE Trans. Image Process., 23 (2014) 3321-3335.

[27]

Z. Ma, K. He, Y. Wei, J. Sun, E. Wu, Constant time weighted median filtering for stereo matching and beyond, in: Proc. IEEE ICCV, 2013, pp. 49-56.

Digital Library

[28]

Q. Yang, Stereo matching using tree filtering, IEEE Trans. Pattern Anal. Mach. Intell., 37 (2015) 834-846.

[29]

L. Dai, F. Zhang, X. Mei, X. Zhang, Fast minimax path-based joint depth interpolation, IEEE Signal Process. Lett., 22 (2015) 623-627.

[30]

M.-Y. Liu, O. Tuzel, Y. Taguchi, Joint geodesic upsampling of depth images, in: Proc. IEEE CVPR, 2013, pp. 169-176.

Digital Library

[31]

J. Diebel, S. Thrun, An application of Markov random fields to range sensing, in: Proc. NIPS, 2005, pp. 291-298.

Digital Library

[32]

J. Lu, D. Min, R. Pahwa, M. Do, A revisit to MRF-based depth map super-resolution and enhancement, in: Proc. IEEE ICASSP, 2011, pp. 985-988.

[33]

D. Kim, K.-J. Yoon, High-quality depth map up-sampling robust to edge noise of range sensors, in: Proc. IEEE ICIP, 2012, pp. 553-556.

[34]

J. Zhu, L. Wang, J. Gao, R. Yang, Spatial-temporal fusion for high accuracy depth maps using dynamic MRFs, IEEE Trans. Pattern Anal. Mach. Intell., 32 (2010) 899-909.

Digital Library

[35]

J. Park, H. Kim, Y.-W. Tai, M.S. Brown, I. Kweon, High quality depth map upsampling for 3D-TOF cameras, in: Proc. IEEE ICCV, 2011, pp. 1623-1630.

Digital Library

[36]

L. Dai, H. Wang, X. Mai, X. Zhang, Depth map upsampling via compressive sensing, in: Proc. ACPR, 2013, pp. 90-94.

Digital Library

[37]

X. Gong, J. Ren, B. Lai, C. Yan, H. Qian, Guided depth upsampling via a cosparse analysis model, in: Proc. IEEE CVPR, 2014, pp. 738-745.

Digital Library

[38]

J. Yang, X. Ye, K. Li, C. Hou, Y. Wang, Color-guided depth recovery from rgb-d data using an adaptive auto-regressive model, IEEE Trans. Image Process., 23 (2014) 3443-3458.

[39]

S. Schwarz, M. Sjstrm, R. Olsson, A weighted optimization approach to time-of-flight sensor fusion, IEEE Trans. Image Process., 23 (2014) 214-225.

Digital Library

[40]

D. Ferstl, C. Reinbacher, R. Ranftl, M. Rther, H. Bischof, Image guided depth upsampling using anisotropic total generalized variation, in: Proc. IEEE ICCV, 2013, pp. 993-1000.

Digital Library

[41]

A. Zomet, S. Peleg, Multi-sensor super-resolution, in: Proc. IEEE WACV, 2002, pp. 27-31.

Digital Library

[42]

C. Tomasi, R. Manduchi, Bilateral filtering for gray and color images, in: Proc. IEEE ICCV, 1998, pp. 836-846.

Digital Library

[43]

D. Barash, A fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation, IEEE Trans. Pattern Anal. Mach. Intell., 24 (2002) 844-847.

Digital Library

[44]

P. Perona, J. Malik, Scale-space and edge detection using anisotropic diffusion, IEEE Trans. Pattern Anal. Mach. Intell., 12 (1990) 629-639.

Digital Library

[45]

P. Saint-Marc, J.S. Chen, G. Medioni, Adaptive smoothing: a general tool for early vision, IEEE Trans. Pattern Anal. Mach. Intell., 13 (1991) 514-529.

Digital Library

[46]

G. Petschnigg, M. Agrawala, H. Hoppe, R. Szeliski, M. Cohen, K. Toyama, Digital photography with flash and no-flash image pairs, ACM Trans. Graph., 23 (2007) 664-672.

Digital Library

[47]

P. Choudhury, J. Tumblin, The trilateral filter for high contrast images and meshes, in: Proc. Eurograph. Symp. Rendering, 2003, pp. 186196.

Digital Library

[48]

C. Zhang, W. Lin, W. Li, B. Zhou, J. Xie, J. Li, Improved image deblurring based on salient-region segmentation, Signal Process.: Image Commun., 28 (2013) 1171-1186.

Digital Library

[49]

Z. Li, J. Zheng, Z. Zhu, W. Yao, S. Wu, Weighted guided image filtering, IEEE Trans. Image Process., 24 (2015) 120-129.

[50]

L.I. Rudin, S. Osher, E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D, Nonlinear Phenom., 60 (1992) 259-268.

Digital Library

[51]

Z. Farbman, R. Fattal, D. Lischinski, R. Szeliski, Edge-preserving decompositions for multi-scale tone and detail manipulation, ACM Trans. Graph., 27 (2008) 249-256.

Digital Library

[52]

L. Xu, C.W. Lu, Y. Xu, J. Jia, Image smoothing via L0 gradient minimization, ACM Trans. Graph., 30 (2011) 174:1-174:12.

Digital Library

[53]

R.C. Gonzalez, R.E. Woods, Digital Image Processing, Prentice-Hall, Upper Saddle River, NJ, USA, 2002.

Digital Library

[54]

L. Itti, C. Koch, E. Niebur, A model of saliency-based visual attention for rapid scene analysis, IEEE Trans. Pattern Anal. Mach. Intell., 20 (1998) 1254-1259.

Digital Library

[55]

D. Scharstein, C. Pal, Learning conditional random fields for stereo, in: Proc. CVPR, 2007, pp. 1-8.

[56]

D. Scharstein, H. Hirschmller, Y. Kitajima, G. Krathwohl, N. Nesic, X. Wang, P. Westling, High-resolution stereo datasets with subpixel-accurate ground truth, in: Proc. GCPR, 2014, pp. 31-42.

[57]

X. Li, M.T. Orchard, New edge-directed interpolation, IEEE Trans. Image Process., 10 (2001) 1251-1257.

Digital Library

[58]

J. Sun, J. Sun, Z. Xu, Image super-resolution using gradient profile prior, in: Proc. IEEE CVPR, 2008, pp. 1-8.

[59]

Available: <http://docs.nvidia.com/cuda/thrust/index.html>.

Cited By

Chen LFu G(2020)Structure-preserving image smoothing with semantic cuesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-020-01950-136:10-12(2017-2027)Online publication date: 1-Oct-2020
https://dl.acm.org/doi/10.1007/s00371-020-01950-1
Yang HZhang Z(2020)Depth image upsampling based on guided filter with low gradient minimizationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-019-01748-w36:7(1411-1422)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.1007/s00371-019-01748-w
Altantawy DSaleh AKishk S(2020)Texture-guided depth upsampling using Bregman split: a clustering graph-based approachThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-018-1611-x36:2(333-359)Online publication date: 1-Feb-2020
https://dl.acm.org/doi/10.1007/s00371-018-1611-x
Show More Cited By

Intensity-guided edge-preserving depth upsampling through weighted L0 gradient minimization
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
  2. Computer graphics

Recommendations

Image-guided ToF depth upsampling: a survey

Recently, there has been remarkable growth of interest in the development and applications of time-of-flight (ToF) depth cameras. Despite the permanent improvement of their characteristics, the practical applicability of ToF cameras is still limited by ...
Depth image upsampling based on guided filter with low gradient minimization
Abstract
In this paper, we present a novel upsampling framework to enhance the spatial resolution of the depth image. In our framework, the upscaling of a low-resolution depth image is guided by a corresponding intensity images; we formulate it as a cost ...
Edge-preserving image decomposition using L1 fidelity with L0 gradient
SA '12: SIGGRAPH Asia 2012 Technical Briefs

We present an image decomposition method using L1 fidelity term with L0 norm of gradient to decompose an image into base layer and detail layer. Generally, the L1 fidelity should be preferable to the L2 norm when the erroneous measurements exist. It is ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Journal of Visual Communication and Image Representation

Journal of Visual Communication and Image Representation Volume 42, Issue C

January 2017

180 pages

ISSN:1047-3203

Issue’s Table of Contents

Copyright © Elsevier Inc.

Publisher

Academic Press, Inc.

United States

Publication History

Published: 01 January 2017

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 11 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Chen LFu G(2020)Structure-preserving image smoothing with semantic cuesThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-020-01950-136:10-12(2017-2027)Online publication date: 1-Oct-2020
https://dl.acm.org/doi/10.1007/s00371-020-01950-1
Yang HZhang Z(2020)Depth image upsampling based on guided filter with low gradient minimizationThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-019-01748-w36:7(1411-1422)Online publication date: 1-Jul-2020
https://dl.acm.org/doi/10.1007/s00371-019-01748-w
Altantawy DSaleh AKishk S(2020)Texture-guided depth upsampling using Bregman split: a clustering graph-based approachThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-018-1611-x36:2(333-359)Online publication date: 1-Feb-2020
https://dl.acm.org/doi/10.1007/s00371-018-1611-x
Li YSun JWang BZhao Y(2018)Depth Super-Resolution Using Joint Adaptive Weighted Least Squares And Patching Gradient2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2018.8462667(1458-1462)Online publication date: 15-Apr-2018
https://dl.acm.org/doi/10.1109/ICASSP.2018.8462667

View Options

View options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents