More Web Proxy on the site http://driver.im/

research-article

Reconstructing Reflection Maps Using a Stacked-CNN for Mixed Reality Rendering

Authors:

Andrew Chalmers,

Daniel Medeiros,

Taehyun RheeAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 27, Issue 10

Pages 4073 - 4084

https://doi.org/10.1109/TVCG.2020.3001917

Published: 01 October 2021 Publication History

Abstract

Corresponding lighting and reflectance between real and virtual objects is important for spatial presence in augmented and mixed reality (AR and MR) applications. We present a method to reconstruct real-world environmental lighting, encoded as a reflection map (RM), from a conventional photograph. To achieve this, we propose a stacked convolutional neural network (SCNN) that predicts high dynamic range (HDR) 360<inline-formula><tex-math notation="LaTeX">${}^\circ$</tex-math><alternatives><mml:math><mml:msup><mml:mrow/><mml:mo>∘</mml:mo></mml:msup></mml:math><inline-graphic xlink:href="chalmers-ieq1-3001917.gif"/></alternatives></inline-formula> RMs with varying roughness from a limited field of view, low dynamic range photograph. The SCNN is progressively trained from high to low roughness to predict RMs at varying roughness levels, where each roughness level corresponds to a virtual object’s roughness (from diffuse to glossy) for rendering. The predicted RM provides high-fidelity rendering of virtual objects to match with the background photograph. We illustrate the use of our method with indoor and outdoor scenes trained on separate indoor/outdoor SCNNs showing plausible rendering and composition of virtual objects in AR/MR. We show that our method has improved quality over previous methods with a comparative user study and error metrics.

References

[1]

G. S. Miller and C. D. Hoffman, “Illumination and reflection maps: Simulated objects in simulated and real environments,” Course Notes Adv. Comput. Graphics Animation, SIGGRAPH, 1984.

[2]

M. Garon, K. Sunkavalli, S. Hadap, N. Carr, and J.-F. Lalonde, “Fast spatially-varying indoor lighting estimation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 6908–6917.

[3]

Y. Hold-Geoffroy, A. Athawale, and J.-F. Lalonde, “Deep sky modeling for single image outdoor lighting estimation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 6927–6935.

[4]

Y. Hold-Geoffroy, K. Sunkavalli, S. Hadap, E. Gambaretto, and J.-F. Lalonde, “Deep outdoor illumination estimation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 7312–7321.

[5]

C. LeGendre, et al., “DeepLight: Learning illumination for unconstrained mobile mixed reality,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 5918–5928.

[6]

J. Zhang, K. Sunkavalli, Y. Hold-Geoffroy, S. Hadap, J. Eisenman, and J.-F. Lalonde, “All-weather deep outdoor lighting estimation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 10 158–10 166.

[7]

S. Song and T. Funkhouser, “Neural illumination: Lighting prediction for indoor environments,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 6918–6926.

[8]

M.-A. Gardner, et al., “Learning to predict indoor illumination from a single image,” ACM Trans. Graph., vol. 36, no. 6, 2017, Art. no.

Digital Library

[9]

T. Rhee, L. Petikam, B. Allen, and A. Chalmers, “MR360: Mixed reality rendering for 360 panoramic videos,” IEEE Trans. Vis. Comput. Graphics, vol. 23, no. 4, pp. 1379–1388, Apr. 2017.

Digital Library

[10]

P. E. Debevec and J. Malik, “Recovering high dynamic range radiance maps from photographs,” in Proc. 24th Annu. Conf. Comput. Graph. Interactive Techn., 1997, pp. 369–378.

[11]

P. Debevec, “Rendering synthetic objects into real scenes: Bridging traditional and image-based graphics with global illumination and high dynamic range photography,” in Proc. 25th Annu. Conf. Comput. Graph. Interactive Techn., 1998, pp. 189–198.

[12]

T. Rhee, S. Thompson, D. Medeiros, R. Dos Anjos, and A. Chalmers, “Augmented virtual teleportation for high-fidelity telecollaboration,” IEEE Trans. Vis. Comput. Graphics, vol. 26, no. 5, pp. 1923–1933, May 2020.

[13]

K. Rohmer, J. Jendersie, and T. Grosch, “Natural environment illumination: Coherent interactive augmented reality for mobile and non-mobile devices,” IEEE Trans. Vis. Comput. Graphics, vol. 23, no. 11, pp. 2474–2484, Nov. 2017.

Digital Library

[14]

K. Rohmer, W. Büschel, R. Dachselt, and T. Grosch, “Interactive near-field illumination for photorealistic augmented reality with varying materials on mobile devices,” IEEE Trans. Vis. Comput. Graphics, vol. 21, no. 12, pp. 1349–1362, Dec. 2015.

Digital Library

[15]

S. R. Marschner and D. P. Greenberg, Inverse Rendering for Computer Graphics, PhD thesis, Stanford University, 1998.

Digital Library

[16]

E. A. Khan, E. Reinhard, R. W. Fleming, and H. H. Bülthoff, “Image-based material editing,” ACM Trans. Graph., vol. 25, no. 3, pp. 654–663, 2006.

Digital Library

[17]

H. Landis, “Production-ready global illumination,” SIGGRAPH Course Notes, vol. 16, no. 2002, 2002, Art. no.

[18]

K. Karsch, et al., “Automatic scene inference for 3D object compositing,” ACM Trans. Graph., vol. 33, no. 3, Jun. 2014, Art. no.

[19]

S. Georgoulis, K. Rematas, T. Ritschel, M. Fritz, T. Tuytelaars, and L. Van Gool, “What is around the camera?” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 5170–5178.

[20]

K. Rematas, T. Ritschel, M. Fritz, E. Gavves, and T. Tuytelaars, “Deep reflectance maps,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 4508–4516.

[21]

M. Maximov, L. Leal-Taixe, M. Fritz, and T. Ritschel, “Deep appearance maps,” in Proc. IEEE Int. Conf. Comput. Vis., 2019, pp. 8729–8738.

[22]

S. Georgoulis, et al., “Reflectance and natural illumination from single-material specular objects using deep learning,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 8, pp. 1932–1947, Aug. 2018.

[23]

J.-F. Lalonde, A. A. Efros, and S. G. Narasimhan, “Estimating natural illumination from a single outdoor image,” in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 183–190.

[24]

Y. Liu, X. Qin, S. Xu, E. Nakamae, and Q. Peng, “Light source estimation of outdoor scenes for mixed reality,” The Vis. Comput., vol. 25, no. 5/7, pp. 637–646, 2009.

Digital Library

[25]

L. Hosek and A. Wilkie, “An analytic model for full spectral sky-dome radiance,” ACM Trans. Graph., vol. 31, no. 4, 2012, Art. no.

[26]

M.-A. Gardner, Y. Hold-Geoffroy, K. Sunkavalli, C. Gagne, and J.-F. Lalonde, “Deep parametric indoor lighting estimation,” in Proc. IEEE Int. Conf. Comput. Vis., 2019.

[27]

R. Ramamoorthi and P. Hanrahan, “On the relationship between radiance and irradiance: Determining the illumination from images of a convex lambertian object,” J. Opt. Soc. America A, vol. 18, no. 10, pp. 2448–2459, 2001.

[28]

J. L. Elman, “Learning and development in neural networks: The importance of starting small,” Cognition, vol. 48, no. 1, pp. 71–99, 1993.

[29]

Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in Proc. 26th Annu. Int. Conf. Mach. Learn., 2009, pp. 41–48.

[30]

G. Hacohen and D. Weinshall, “On the power of curriculum learning in training deep networks,” in Proc. Int. Conf. Mach. Learn., 2019.

[31]

B. T. Phong, “Illumination for computer generated pictures,” Commun. ACM, vol. 18, no. 6, pp. 311–317, 1975.

Digital Library

[32]

B. Burley and W. D. A. Studios, “Physically-based shading at disney,” ACM SIGGRAPH Courses, vol. 2012, pp. 1–7, 2012.

[33]

B. Walter, S. R. Marschner, H. Li, and K. E. Torrance, “Microfacet models for refraction through rough surfaces,” in Proc. 18th Eurographics Conf. Rendering Techn., 2007, pp. 195–206.

[34]

K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 770–778.

[35]

D.-A. Clevert, T. Unterthiner, and S. Hochreiter, “Fast and accurate deep network learning by exponential linear units (ELUs),” in Proc. Int. Conf. Learn. Representations, 2016.

[36]

S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. Int. Conf. Mach. Learn., 2015.

[37]

J. Xiao, K. A. Ehinger, A. Oliva, and A. Torralba, “Recognizing scene viewpoint using panoramic place representation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 2695–2702.

[38]

J. Zhang and J.-F. Lalonde, “Learning high dynamic range from outdoor panoramas,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 4529–4538.

[39]

H. Weber, D. Prévost, and J.-F. Lalonde, “Learning to estimate indoor lighting from 3D objects,” in Proc. Int. Conf. 3D Vis., 2018, pp. 199–207.

[40]

D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. Int. Conf. Learn. Representations, 2015.

[41]

L. Williams, “Pyramidal parametrics,” in Proc. 10th Annu. Conf. Comput. Graph. Interactive Techn., 1983, vol. 17, pp. 1–11.

[42]

M. Abadi, et al., “TensorFlow: A system for large-scale machine learning,” in Proc. 12th USENIX Symp. Operating Syst. Des. Implementation, 2016, pp. 265–283.

[43]

M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “MobileNetV2: Inverted residuals and linear bottlenecks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 4510–4520.

Cited By

Zhao JXue BZhang M(2024)SALENet: Structure-Aware Lighting Estimations From a Single Image for Indoor EnvironmentsIEEE Transactions on Image Processing10.1109/TIP.2024.351238133(6806-6820)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3512381
Cortes DBermejo BJuiz C(2024)The use of CNNs in VR/AR/MR/XR: a systematic literature reviewVirtual Reality10.1007/s10055-024-01044-628:3Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1007/s10055-024-01044-6
Liu CWang LLi ZQuan SXu Y(2023)Real-Time Lighting Estimation for Augmented Reality via Differentiable Screen-Space RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.314194329:4(2132-2145)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3141943
Show More Cited By

Index Terms

Reconstructing Reflection Maps Using a Stacked-CNN for Mixed Reality Rendering
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
  2. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Index terms have been assigned to the content through auto-classification.

Recommendations

Sample-Based Cameras for Feed Forward Reflection Rendering

This paper presents sample-based cameras for rendering high quality reflections on convex reflectors at interactive rates. The method supports change of view, moving objects and reflectors, higher order reflections, view-dependent lighting of reflected ...
Radially-symmetric reflection maps
SIGGRAPH '09: SIGGRAPH 2009: Talks

When designing lighting for outdoor scenes, area lights can represent distant sources such as the sun and sky with greater shading fidelity than is possible with simple directional or hemispherical lights, but the combination of dynamic area light ...
Photorealistic rendering of mixed reality scenes

Photo-realistic rendering of virtual objects into real scenes is one of the most important research problems in computer graphics. Methods for capture and rendering of mixed reality scenes are driven by a large number of applications, ranging from ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 27, Issue 10

Oct. 2021

246 pages

ISSN:1077-2626

Issue’s Table of Contents

1077-2626 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 October 2021

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 24 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Zhao JXue BZhang M(2024)SALENet: Structure-Aware Lighting Estimations From a Single Image for Indoor EnvironmentsIEEE Transactions on Image Processing10.1109/TIP.2024.351238133(6806-6820)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3512381
Cortes DBermejo BJuiz C(2024)The use of CNNs in VR/AR/MR/XR: a systematic literature reviewVirtual Reality10.1007/s10055-024-01044-628:3Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1007/s10055-024-01044-6
Liu CWang LLi ZQuan SXu Y(2023)Real-Time Lighting Estimation for Augmented Reality via Differentiable Screen-Space RenderingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2022.314194329:4(2132-2145)Online publication date: 1-Apr-2023
https://dl.acm.org/doi/10.1109/TVCG.2022.3141943
Zhang FZhao JZhang YZollmann S(2023)A Survey on 360° Images and Videos in Mixed Reality: Algorithms and ApplicationsJournal of Computer Science and Technology10.1007/s11390-023-3210-138:3(473-491)Online publication date: 1-Jun-2023
https://dl.acm.org/doi/10.1007/s11390-023-3210-1
Rhee TJung S(2022)TeleverseSIGGRAPH Asia 2022 Courses10.1145/3550495.3558217(1-134)Online publication date: 6-Dec-2022
https://dl.acm.org/doi/10.1145/3550495.3558217
Chalmers AZickler TRhee T(2022)Illumination BrowserComputers and Graphics10.1016/j.cag.2022.01.006103:C(101-108)Online publication date: 1-Apr-2022
https://dl.acm.org/doi/10.1016/j.cag.2022.01.006

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents