More Web Proxy on the site http://driver.im/

research-article

Image-based rendering for scenes with reflections

Authors:

Sudipta N. Sinha,

Michael Goesele,

Daniel Scharstein,

Richard SzeliskiAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 31, Issue 4

Article No.: 100, Pages 1 - 10

https://doi.org/10.1145/2185520.2185596

Published: 01 July 2012 Publication History

Abstract

We present a system for image-based modeling and rendering of real-world scenes containing reflective and glossy surfaces. Previous approaches to image-based rendering assume that the scene can be approximated by 3D proxies that enable view interpolation using traditional back-to-front or z-buffer compositing. In this work, we show how these can be generalized to multiple layers that are combined in an additive fashion to model the reflection and transmission of light that occurs at specular surfaces such as glass and glossy materials. To simplify the analysis and rendering stages, we model the world using piecewise-planar layers combined using both additive and opaque mixing of light. We also introduce novel techniques for estimating multiple depths in the scene and separating the reflection and transmission components into different layers. We then use our system to model and render a variety of real-world scenes with reflections.

Supplementary Material

ZIP File (a100-sinha.zip)

Supplemental material.

Download
376.70 MB

References

[1]

Beery, E., and Yeredor, A. 2008. Blind separation of superimposed shifted images using parameterized joint diagonalization. IEEE Transactions on Image Processing 17, 3 (March), 340--353.

Digital Library

[2]

Bergen, J. R., Burt, P. J., Hingorani, R., and Peleg, S. 1992. A three-frame algorithm for estimating two-component image motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 9 (September), 886--896.

Digital Library

[3]

Bhat, D. N., and Nayar, S. K. 1998. Stereo and specular reflection. International Journal of Computer Vision 26, 2 (February), 91--106.

Digital Library

[4]

Bhat, P., Zitnick, C. L., Snavely, N., Agarwala, A., Agrawala, M., Cohen, M., Curless, B., and Kang, S. B. 2007. Using photographs to enhance videos of a static scene. In Eurographics Symposium on Rendering, 327--338.

Digital Library

[5]

Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization via graph cuts. IEEE Transactions on Pattern Analysis and Machine Intelligence 23, 11 (November), 1222--1239.

Digital Library

[6]

Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured Lumigraph rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, ACM, SIGGRAPH '01, 425--432.

Digital Library

[7]

Carceroni, R. L., and Kutulakos, K. N. 2002. Multi-view scene capture by surfel sampling: From video streams to nonrigid 3d motion, shape and reflectance. International Journal of Computer Vision 49, 2/3, 175--214.

Digital Library

[8]

Chen, S., and Williams, L. 1993. View interpolation for image synthesis. In Proceedings of the 20th annual conference on Computer graphics and interactive techniques, SIGGRAPH '93, 279--288.

Digital Library

[9]

Cohen-Steiner, D., Alliez, P., and Desbrun, M. 2004. Variational shape approximation. ACM Trans. Graph. 23 (August), 905--914.

Digital Library

[10]

Criminisi, A., Kang, S. B., Swaminathan, R., Szeliski, R., and Anandan, P. 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. Computer Vision and Image Understanding 97, 1 (January), 51--85.

Digital Library

[11]

Darrell, T., and Pentland, A. 1995. Cooperative robust estimation using layers of support. IEEE Transactions on Pattern Analysis and Machine Intelligence 17, 5 (May), 474--487.

Digital Library

[12]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry- and image-based approach. In ACM SIGGRAPH 1996 Conference Proceedings, 11--20.

Digital Library

[13]

Diamant, Y., and Schechner, Y. Y. 2008. Overcoming visual reverberations. In Computer Vision and Pattern Recognition (CVPR'08), 1--8.

[14]

Furukawa, Y., Curless, B., Seitz, S. M., and Szeliski, R. 2009. Manhattan-world stereo. In Computer Vision and Pattern Recognition (CVPR 2009), 1422--1429.

[15]

Gallup, D., Frahm, J.-M., and Pollefeys, M. 2010. Piece-wise planar and non-planar stereo for urban scene reconstruction. In Computer Vision and Pattern Recognition (CVPR'10), 1418--1425.

[16]

Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The Lumigraph. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH '96, 43--54.

Digital Library

[17]

Hirschmüller, H. 2008. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2 (February), 328--341.

Digital Library

[18]

Irani, M., Rousso, B., and Peleg, S. 1994. Computing occluding and transparent motions. International Journal of Computer Vision 12, 1 (January), 5--16.

Digital Library

[19]

Ju, S. X., Black, M. J., and Jepson, A. D. 1996. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. In Computer Vision and Pattern Recognition (CVPR'96), 307--314.

Digital Library

[20]

Levin, A., Zomet, A., and Weiss, Y. 2002. Learning to perceive transparency from the statistics of natural scenes. In (NIPS), MIT Press, 1247--1254.

[21]

Levoy, M., and Hanrahan, P. 1996. Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, ACM, SIGGRAPH '96, 31--42.

Digital Library

[22]

Nocedal, J., and Wright, S. J. 2006. Numerical Optimization, second ed. Springer, New York.

[23]

Popescu, V., Mei, C., Dauble, J., and Sacks, E. 2006. Reflected-scene impostors for realistic reflections at interactive rates. Computer Graphics Forum 25, 3 (Sept.), 313--322.

[24]

Schechner, Y. Y., Shamir, J., and Kiryati, N. 1999. Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface. In International Conference on Computer Vision (ICCV'99), 814--819.

Digital Library

[25]

Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. In ACM SIGGRAPH 1998 Conference Proceedings, 231--242.

Digital Library

[26]

Shizawa, M., and Mase, K. 1991. A unified computational theory of motion transparency and motion boundaries based on eigenenergy analysis. In Computer Vision and Pattern Recognition (CVPR), 289--295.

[27]

Sinha, S. N., Steedly, D., and Szeliski, R. 2009. Piece-wise planar stereo for image-based rendering. In International Conference on Computer Vision (ICCV 2009), 1881--1888.

[28]

Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3D. ACM Transactions on Graphics (Proc. SIGGRAPH 2006) 25, 3 (August), 835--846.

Digital Library

[29]

Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. In Computer Vision and Pattern Recognition (CVPR'2000), vol. 1, 246--253.

[30]

Szeliski, R. 2010. Computer Vision: Algorithms and Applications. Springer, New York.

[31]

Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. IEEE Trans. on Pattern Analysis and Machine Intelligence 28, 2 (February), 290--301.

Digital Library

[32]

Weingarten, J. W., Gruener, G., and Dorf, A. 2004. Probabilistic plane fitting in 3D and an application to robotic mapping. In International Conference on Robotics and Automation (ICRA), 927--932.

[33]

Zebedin, L., Bauer, J., Karner, K. F., and Bischof, H. 2008. Fusion of feature- and area-based information for urban buildings modeling from aerial imagery. In European Conference on Computer Vision (ECCV'08), 873--886.

Digital Library

[34]

Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics (Proc. SIGGRAPH 2004) 23, 3 (August), 600--608.

Digital Library

Cited By

Lu ZWang WGuo TWang F(2025)Single-image reflection removal via self-supervised diffusion modelsThe Journal of Supercomputing10.1007/s11227-024-06837-981:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11227-024-06837-9
Tang JZhou WLiu HMo YKe W(2024)Single Image Reflection Separation in Scenes with Light Source Only on One Side of the Camera2024 International Conference on Cloud and Network Computing (ICCNC)10.1109/ICCNC63989.2024.00015(36-41)Online publication date: 31-May-2024
https://doi.org/10.1109/ICCNC63989.2024.00015
Zhu YFu XJiang PZhang HSun QChen JZha ZLi B(2024)Revisiting Single Image Reflection Removal in the Wild2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02406(25468-25478)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02406
Show More Cited By

Index Terms

Image-based rendering for scenes with reflections
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        3D imaging
  2. Computer graphics

Index terms have been assigned to the content through auto-classification.

Recommendations

Sample-Based Cameras for Feed Forward Reflection Rendering

This paper presents sample-based cameras for rendering high quality reflections on convex reflectors at interactive rates. The method supports change of view, moving objects and reflectors, higher order reflections, view-dependent lighting of reflected ...
An introduction to image-based rendering
Integrated image and graphics technologies

In this chapter, we review the techniques for image-based rendering. Unlike traditional 3D computer graphics in which 3D geometry of the scene is known, image-based rendering (IBR) techniques render novel views directly from input images. IBR techniques ...
Real-time multiply recursive reflections and refractions using hybrid rendering

We present a new method for real-time rendering of multiple recursions of reflections and refractions. The method uses the strengths of real-time ray tracing for objects close to the camera, by storing them in a per-frame constructed bounding volume ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 31, Issue 4

July 2012

935 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2185520

Issue’s Table of Contents

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2012

Published in TOG Volume 31, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

91
Total Citations
View Citations
926
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)6

Reflects downloads up to 09 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lu ZWang WGuo TWang F(2025)Single-image reflection removal via self-supervised diffusion modelsThe Journal of Supercomputing10.1007/s11227-024-06837-981:1Online publication date: 1-Jan-2025
https://dl.acm.org/doi/10.1007/s11227-024-06837-9
Tang JZhou WLiu HMo YKe W(2024)Single Image Reflection Separation in Scenes with Light Source Only on One Side of the Camera2024 International Conference on Cloud and Network Computing (ICCNC)10.1109/ICCNC63989.2024.00015(36-41)Online publication date: 31-May-2024
https://doi.org/10.1109/ICCNC63989.2024.00015
Zhu YFu XJiang PZhang HSun QChen JZha ZLi B(2024)Revisiting Single Image Reflection Removal in the Wild2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02406(25468-25478)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02406
Alzayer HZhang KFeng BMetzler CHuang J(2024)Seeing the World through Your Eyes2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00465(4864-4873)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00465
Wu WJin LQi BLi GLi J(2024)Anti-noise light field depth measurement using specular highlight removalOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108079176(108079)Online publication date: May-2024
https://doi.org/10.1016/j.optlaseng.2024.108079
Qiu JYin ZCheng MRen B(2024)Rendering real-world unbounded scenes with cars by learning positional biasThe Visual Computer: International Journal of Computer Graphics10.1007/s00371-023-03070-y40:6(4085-4098)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s00371-023-03070-y
Chen RZhao JZhang FChalmers ARhee T(2024)Neural Radiance Fields for Dynamic View Synthesis Using Local Temporal PriorsComputational Visual Media10.1007/978-981-97-2095-8_5(74-90)Online publication date: 10-Apr-2024
https://dl.acm.org/doi/10.1007/978-981-97-2095-8_5
Xie MCai HShah SXu YFeng BHuang JMetzler C(2024)Flash-Splat: 3D Reflection Removal with Flash Cues and Gaussian SplatsComputer Vision – ECCV 202410.1007/978-3-031-73007-8_8(122-139)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73007-8_8
Guo YLu WLi XHuang Q(2023)Single Image Reflection Removal Based on Residual Attention MechanismApplied Sciences10.3390/app1303161813:3(1618)Online publication date: 27-Jan-2023
https://doi.org/10.3390/app13031618
Paliwal ANguyen BTsarov AKalantari N(2023)ReShader: View-Dependent Highlights for Single Image View-SynthesisACM Transactions on Graphics10.1145/361839342:6(1-9)Online publication date: 5-Dec-2023
https://dl.acm.org/doi/10.1145/3618393
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents