More Web Proxy on the site http://driver.im/

research-article

Differentiable Heightfield Path Tracing with Accelerated Discontinuities

Authors:

Hsueh-Ti Derek Liu,

Alec JacobsonAuthors Info & Claims

SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

Article No.: 19, Pages 1 - 9

https://doi.org/10.1145/3588432.3591530

Published: 23 July 2023 Publication History

Abstract

We investigate the problem of accelerating a physically-based differentiable renderer for heightfields based on path tracing with global illumination. On a heightfield with 1 million vertices (1024 × 1024 resolution), our differentiable renderer requires only 4 ms per sample per pixel when differentiating direct illumination, orders of magnitude faster than most existing general 3D mesh differentiable renderers. It is well-known that one can leverage spatial hierarchical data structures (e.g., the maximum mipmaps) to accelerate the forward pass of heightfield rendering. The key idea of our approach is to further utilize the hierarchy to speed up the backward pass—differentiable heightfield rendering. Specifically, we use the maximum mipmaps to accelerate the process of identifying scene discontinuities, which is crucial for obtaining accurate derivatives. Our renderer supports global illumination. we are able to optimize global effects, such as shadows, with respect to the geometry and the material parameters. Our differentiable renderer achieves real-time frame rates and unlocks interactive inverse rendering applications. We demonstrate the flexibility of our method with terrain optimization, geometric illusions, shadow optimization, and text-based shape generation.

Supplemental Material

MP4 File

presentation

Download
154.72 MB

PDF File

Appendix

Download
920.00 KB

References

[1]

Marc Alexa and Wojciech Matusik. 2010. Reliefs as images. ACM Trans. Graph. 29, 4 (2010), 60:1–60:7.

Digital Library

[2]

John Amanatides and Andrew Woo. 1987. A Fast Voxel Traversal Algorithm for Ray Tracing. In 8th European Computer Graphics Conference and Exhibition, Eurographics 1987, Amsterdam, The Netherlands, August 24-28, 1987, Proceedings, Guy Maréchal (Ed.). North-Holland / Eurographics Association.

[3]

Sai Praveen Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased warped-area sampling for differentiable rendering. ACM Trans. Graph. 39, 6 (2020), 245:1–245:18.

Digital Library

[4]

Sai Praveen Bangaru, Jesse Michel, Kevin Mu, Gilbert Bernstein, Tzu-Mao Li, and Jonathan Ragan-Kelley. 2021. Systematically differentiating parametric discontinuities. ACM Trans. Graph. 40, 4 (2021), 107:1–107:18.

Digital Library

[5]

Chengqian Che, Fujun Luan, Shuang Zhao, Kavita Bala, and Ioannis Gkioulekas. 2018. Inverse Transport Networks. CoRR abs/1809.10820 (2018).

[6]

Wenzheng Chen, Huan Ling, Jun Gao, Edward J. Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. 2019. Learning to Predict 3D Objects with an Interpolation-based Differentiable Renderer. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d’Alché-Buc, Emily B. Fox, and Roman Garnett (Eds.). 9605–9616.

[7]

Daniel Cohen-Or, Eran Rich, Uri Lerner, and Victor Shenkar. 1996. A Real-Time Photo-Realistic Visual Flythrough. IEEE Trans. Vis. Comput. Graph. 2, 3 (1996), 255–265.

Digital Library

[8]

Daniel Cohen-Or and Amit Shaked. 1993. Photo-Realistic Imaging of Digital Terrains. Comput. Graph. Forum 12, 3 (1993), 363–373.

[9]

Mark Duchaineau, Murray Wolinsky, David E Sigeti, Mark C Miller, Charles Aldrich, and Mark B Mineev-Weinstein. 1997. ROAMing terrain: Real-time optimally adapting meshes. In Proceedings. Visualization’97 (Cat. No. 97CB36155). IEEE, 81–88.

[10]

Jun Gao, Wenzheng Chen, Tommy Xiang, Alec Jacobson, Morgan McGuire, and Sanja Fidler. 2020. Learning Deformable Tetrahedral Meshesfor 3D Reconstruction. In NeurIPS.

[11]

Yotam I. Gingold and Denis Zorin. 2008. Shading-based surface editing. ACM Trans. Graph. 27, 3 (2008), 95.

Digital Library

[12]

Christian Henning and Peter Stephenson. 2004. Accelerating the ray tracing of height fields. In Proceedings of the 2nd international conference on Computer graphics and interactive techniques in Australasia and South East Asia. 254–258.

Digital Library

[13]

Hugues Hoppe. 1998. Smooth view-dependent level-of-detail control and its application to terrain rendering. In 9th IEEE Visualization Conference, IEEE Vis 1998, Research Triangle Park, North Carolina, USA, October 18-23, 1998, Proceedings, David S. Ebert, Holly E. Rushmeier, and Hans Hagen (Eds.). IEEE Computer Society and ACM, 35–42.

[14]

Kai-Wen Hsiao, Jia-Bin Huang, and Hung-Kuo Chu. 2018. Multi-view wire art. ACM Trans. Graph. 37, 6 (2018), 242:1–242:11.

Digital Library

[15]

Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, Merlin Nimier-David, Delio Vicini, Tizian Zeltner, Baptiste Nicolet, Miguel Crespo, Vincent Leroy, and Ziyi Zhang. 2022b. Mitsuba 3 renderer. https://mitsuba-renderer.org.

[16]

Wenzel Jakob, Sébastien Speierer, Nicolas Roussel, and Delio Vicini. 2022a. Dr.Jit: A Just-In-Time Compiler for Differentiable Rendering. Transactions on Graphics (Proceedings of SIGGRAPH) 41, 4 (July 2022). https://doi.org/10.1145/3528223.3530099

Digital Library

[17]

Dawoon Jung, Fridger Schrempp, and Seunghee Son. 2020. Optimally Fast Soft Shadows on Curved Terrain with Dynamic Programming and Maximum Mipmaps. CoRR abs/2005.06671 (2020).

[18]

Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. 2020. Differentiable Rendering: A Survey. CoRR abs/2006.12057 (2020). arXiv:2006.12057https://arxiv.org/abs/2006.12057

[19]

Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. 2018. Neural 3D Mesh Renderer. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018. Computer Vision Foundation / IEEE Computer Society, 3907–3916.

[20]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, Yoshua Bengio and Yann LeCun (Eds.).

[21]

Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. 2020. Modular primitives for high-performance differentiable rendering. ACM Trans. Graph. 39, 6 (2020), 194:1–194:14.

Digital Library

[22]

Tzu-Mao Li, Miika Aittala, Frédo Durand, and Jaakko Lehtinen. 2018. Differentiable Monte Carlo ray tracing through edge sampling. ACM Trans. Graph. 37, 6 (2018), 222:1–222:11.

Digital Library

[23]

Tzu-Mao Li, Michal Lukác, Michaël Gharbi, and Jonathan Ragan-Kelley. 2020. Differentiable vector graphics rasterization for editing and learning. ACM Trans. Graph. 39, 6 (2020), 193:1–193:15.

Digital Library

[24]

Peter Lindstrom, David Koller, William Ribarsky, Larry F. Hodges, Nickolas Faust, and Gregory A. Turner. 1996. Real-Time, Continuous Level of Detail Rendering of Height Fields. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 1996, New Orleans, LA, USA, August 4-9, 1996, John Fujii (Ed.). ACM, 109–118.

Digital Library

[25]

Hsueh-Ti Derek Liu, Michael Tao, and Alec Jacobson. 2018. Paparazzi: surface editing by way of multi-view image processing. ACM Trans. Graph. 37, 6 (2018), 221:1–221:11.

Digital Library

[26]

Hsueh-Ti Derek Liu, Michael Tao, Chun-Liang Li, Derek Nowrouzezahrai, and Alec Jacobson. 2019b. Beyond Pixel Norm-Balls: Parametric Adversaries using an Analytically Differentiable Renderer. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.

[27]

Shichen Liu, Weikai Chen, Tianye Li, and Hao Li. 2019a. Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019. IEEE, 7707–7716.

[28]

Matthew M. Loper and Michael J. Black. 2014. OpenDR: An Approximate Differentiable Renderer. In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part VII(Lecture Notes in Computer Science, Vol. 8695), David J. Fleet, Tomás Pajdla, Bernt Schiele, and Tinne Tuytelaars (Eds.). Springer, 154–169.

[29]

Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob. 2019. Reparameterizing discontinuous integrands for differentiable rendering. ACM Trans. Graph. 38, 6 (2019), 228:1–228:14.

Digital Library

[30]

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. 2021. Text2Mesh: Text-Driven Neural Stylization for Meshes. CoRR abs/2112.03221 (2021).

[31]

Niloy J. Mitra and Mark Pauly. 2009. Shadow art. ACM Trans. Graph. 28, 5 (2009), 156.

Digital Library

[32]

Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. 2021. Extracting Triangular 3D Models, Materials, and Lighting From Images. CoRR abs/2111.12503 (2021).

[33]

F Kenton Musgrave. 1988. Grid tracing: Fast ray tracing for height fields. Yale University Dept. of Computer Science Research (1988).

[34]

Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. 2021. Large steps in inverse rendering of geometry. ACM Trans. Graph. 40, 6 (2021), 248:1–248:13.

Digital Library

[35]

Merlin Nimier-David, Sébastien Speierer, Benoît Ruiz, and Wenzel Jakob. 2020. Radiative backpropagation: an adjoint method for lightning-fast differentiable rendering. ACM Trans. Graph. 39, 4 (2020), 146.

Digital Library

[36]

Derek Nowrouzezahrai and John M. Snyder. 2009. Fast Global Illumination on Dynamic Height Fields. Comput. Graph. Forum 28, 4 (2009), 1131–1139.

Digital Library

[37]

Christopher Oat and Pedro V. Sander. 2007. Ambient aperture lighting. In Proceedings of the 2007 Symposium on Interactive 3D Graphics, SI3D 2007, April 30 - May 2, 2007, Seattle, Washington, USA, Bruce Gooch and Peter-Pike J. Sloan (Eds.). ACM, 61–64.

Digital Library

[38]

Daniele Panozzo and Olga Sorkine-Hornung. 2014. Appearance-mimicking surfaces. ACM Trans. Graph. 33, 6 (2014), 216:1–216:10.

[39]

Felix Petersen, Amit H. Bermano, Oliver Deussen, and Daniel Cohen-Or. 2019. Pix2Vex: Image-to-Geometry Reconstruction using a Smooth Differentiable Renderer. CoRR abs/1903.11149 (2019).

[40]

Matt Pharr, Wenzel Jakob, and Greg Humphreys. 2016. Physically based rendering: From theory to implementation. Morgan Kaufmann.

[41]

Huamin Qu, Feng Qiu, Nan Zhang, Arie E. Kaufman, and Ming Wan. 2003. Ray Tracing Height Fields. In 2003 Computer Graphics International (CGI 2003), 9-11 July 2003, Tokyo, Japan. IEEE Computer Society, 202–209.

[42]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning Transferable Visual Models From Natural Language Supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event(Proceedings of Machine Learning Research, Vol. 139), Marina Meila and Tong Zhang (Eds.). PMLR, 8748–8763.

[43]

Ravi Ramamoorthi and Pat Hanrahan. 2001. An efficient representation for irradiance environment maps. In Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH 2001, Los Angeles, California, USA, August 12-17, 2001, Lynn Pocock (Ed.). ACM, 497–500.

Digital Library

[44]

Helge Rhodin, Nadia Robertini, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. 2015. A Versatile Scene Model with Differentiable Visibility Applied to Generative Pose Estimation. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015. IEEE Computer Society, 765–773.

Digital Library

[45]

Stefan Röttger, Wolfgang Heidrich, Philipp Slusallek, and Hans-Peter Seidel. 1998. Real-time generation of continuous levels of detail for height fields. (1998).

[46]

Kaustubh Sadekar, Ashish Tiwari, and Shanmuganathan Raman. 2022. Shadow Art Revisited: A Differentiable Rendering Based Approach. In IEEE/CVF Winter Conference on Applications of Computer Vision, WACV 2022, Waikoloa, HI, USA, January 3-8, 2022. IEEE, 628–636.

[47]

Steven M. Seitz and Kiriakos N. Kutulakos. 1998. Plenoptic Image Editing. In Proceedings of the Sixth International Conference on Computer Vision (ICCV-98), Bombay, India, January 4-7, 1998. IEEE Computer Society, 17–24.

[48]

Silvia Sellán, Noam Aigerman, and Alec Jacobson. 2020. Developability of heightfields via rank minimization. ACM Trans. Graph. 39, 4 (2020), 109.

Digital Library

[49]

John M. Snyder and Derek Nowrouzezahrai. 2008. Fast Soft Self-Shadowing on Dynamic Height Fields. Comput. Graph. Forum 27, 4 (2008), 1275–1283.

Digital Library

[50]

David C Taylor. 1994. An algorithm for continuous resolution polygonalizations of a discrete surface. In Proc. Graphics Interface’94.

[51]

Art Tevs, Ivo Ihrke, and Hans-Peter Seidel. 2008. Maximum mipmaps for fast, accurate, and scalable dynamic height field rendering. In Proceedings of the 2008 Symposium on Interactive 3D Graphics, SI3D 2008, February 15-17, 2008, Redwood City, CA, USA, Eric Haines and Morgan McGuire (Eds.). ACM, 183–190.

Digital Library

[52]

Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason M. Saragih, Matthias Nießner, Rohit Pandey, Sean Ryan Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, and Michael Zollhöfer. 2020. State of the Art on Neural Rendering. Comput. Graph. Forum 39, 2 (2020), 701–727.

[53]

Theo Thonat, Francois Beaune, Xin Sun, Nathan Carr, and Tamy Boubekeur. 2021. Tessellation-Free Displacement Mapping for Ray Tracing. 40, 6 (2021).

[54]

Ville Timonen and Jan Westerholm. 2010. Scalable Height Field Self-Shadowing. Comput. Graph. Forum 29, 2 (2010), 723–731.

[55]

Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021a. Path replay backpropagation: differentiating light paths using constant memory and linear time. ACM Trans. Graph. 40, 4 (2021), 108:1–108:14.

Digital Library

[56]

Delio Vicini, Sébastien Speierer, and Wenzel Jakob. 2021b. Path Replay Backpropagation: Differentiating Light Paths using Constant Memory and Linear Time. Transactions on Graphics (Proceedings of SIGGRAPH) 40, 4 (Aug. 2021), 108:1–108:14. https://doi.org/10.1145/3450626.3459804

Digital Library

[57]

Yifan Wang, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Hornung. 2019. Differentiable surface splatting for point-based geometry processing. ACM Trans. Graph. 38, 6 (2019), 230:1–230:14.

[58]

Sebastian Weiss, Florian Bayer, and Rüdiger Westermann. 2020. Triplanar Displacement Mapping for Terrain Rendering. In 41st Annual Conference of the European Association for Computer Graphics, Eurographics 2020 - Short Papers, Norrköping, Sweden, May 25-29, 2020 [online only], Alexander Wilkie and Francesco Banterle (Eds.). Eurographics Association, 53–56.

[59]

Kai Yan, Christoph Lassner, Brian Budge, Zhao Dong, and Shuang Zhao. 2022. Efficient estimation of boundary integrals for path-space differentiable rendering. ACM Trans. Graph. 41, 4 (2022), 123:1–123:13.

Digital Library

[60]

Cem Yuksel and John Keyser. 2009. Fast real-time caustics from height fields. Vis. Comput. 25, 5-7 (2009), 559–564.

Digital Library

[61]

Tizian Zeltner, Sébastien Speierer, Iliyan Georgiev, and Wenzel Jakob. 2021. Monte Carlo estimators for differential light transport. ACM Trans. Graph. 40, 4 (2021), 78:1–78:16.

Digital Library

[62]

Cheng Zhang, Zihan Yu, and Shuang Zhao. 2021. Path-space differentiable rendering of participating media. ACM Trans. Graph. 40, 4 (2021), 76:1–76:15.

Digital Library

[63]

Shaokun Zheng, Zhiqian Zhou, Xin Chen, Difei Yan, Chuyan Zhang, Yuefeng Geng, Yan Gu, and Kun Xu. 2022. LuisaRender: A High-Performance Rendering Framework with Layered and Unified Interfaces on Stream Architectures. ACM Trans. Graph. 41, 6, Article 232 (nov 2022), 19 pages. https://doi.org/10.1145/3550454.3555463

Digital Library

[64]

Yang Zhou, Lifan Wu, Ravi Ramamoorthi, and Ling-Qi Yan. 2021. Vectorization for Fast, Analytic, and Differentiable Visibility. ACM Trans. Graph. 40, 3 (2021), 27:1–27:21.

Digital Library

[65]

Jacek Zienkiewicz, Andrew J. Davison, and Stefan Leutenegger. 2016. Real-time height map fusion using differentiable rendering. In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016, Daejeon, South Korea, October 9-14, 2016. IEEE, 4280–4287.

Digital Library

Cited By

Tong XHachisuka T(2024)Efficient Image-Space Shape Splatting for Monte Carlo RenderingACM Transactions on Graphics10.1145/368794343:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687943
Zheng SChen XShi ZYan LXu K(2024)GPU Coroutines for Flexible Splitting and Scheduling of Rendering TasksACM Transactions on Graphics10.1145/368776643:6(1-24)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687766

Index Terms

Differentiable Heightfield Path Tracing with Accelerated Discontinuities
1. Computing methodologies
  1. Computer graphics
    1. Rendering

Recommendations

Reparameterizing discontinuous integrands for differentiable rendering

Differentiable rendering has recently opened the door to a number of challenging inverse problems involving photorealistic images, such as computational material design and scattering-aware reconstruction of geometry and materials from photographs. ...
PSDR-Room: Single Photo to Scene using Differentiable Rendering
SA '23: SIGGRAPH Asia 2023 Conference Papers

A 3D digital scene contains many components: lights, materials and geometries, interacting to reach the desired appearance. Staging such a scene is time-consuming and requires both artistic and technical skills. In this work, we propose PSDR-Room, a ...
Reconstructing Translucent Objects using Differentiable Rendering
SIGGRAPH '22: ACM SIGGRAPH 2022 Conference Proceedings

Inverse rendering is a powerful approach to modeling objects from photographs, and we extend previous techniques to handle translucent materials that exhibit subsurface scattering. Representing translucency using a heterogeneous bidirectional ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '23: ACM SIGGRAPH 2023 Conference Proceedings

July 2023

911 pages

ISBN:9798400701597

DOI:10.1145/3588432

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Adobe Inc.
NSERC Discovery
DSI Catalyst Grant program
Ontario Early Research Award program
Adobe Systems
Canada Research Chairs Program
Sloan Research Fellowship

Conference

SIGGRAPH '23

Sponsor:

SIGGRAPH

SIGGRAPH '23: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 6 - 10, 2023

CA, Los Angeles, USA

Acceptance Rates

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
157
Total Downloads

Downloads (Last 12 months)79
Downloads (Last 6 weeks)4

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Tong XHachisuka T(2024)Efficient Image-Space Shape Splatting for Monte Carlo RenderingACM Transactions on Graphics10.1145/368794343:6(1-11)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687943
Zheng SChen XShi ZYan LXu K(2024)GPU Coroutines for Flexible Splitting and Scheduling of Rendering TasksACM Transactions on Graphics10.1145/368776643:6(1-24)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687766

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents