Creating realistic images has been a major focus in the study of computer graphics for much of its history. This effort has led to mathematical models and algorithms that can compute predictive, or physically realistic, images from known camera positions and scene descriptions that include the geometry of objects, the reflectance of surfaces, and the lighting used to illuminate the scene. These images accurately describe the physical quantities that would be measured from a real scene. Because these algorithms can predict real images, they can also be used in inverse problems to work backward from photographs to attributes of the scene.
Work on three such inverse rendering problems is described. The first, inverse lighting, assumes knowledge of geometry, reflectance, and the recorded photograph and solves for the lighting in the scene. A technique using a linear least-squares system is proposed and demonstrated. Also demonstrated is an application of inverse lighting, called re-lighting, which modifies lighting in photographs.
The second two inverse rendering problems solve for unknown reflectance, given images with known geometry, lighting, and camera positions. Photographic texture measurement concentrates on capturing the spatial variation in an object's reflectance. The resulting system begins with scanned 3D models of real objects and uses photographs to construct accurate, high-resolution textures suitable for physically realistic rendering. The system is demonstrated on two complex natural objects with detailed surface textures.
Image-based BRDF measurement takes the opposite approach to reflectance measurement, capturing the directional characteristics of a surface's reflectance by measuring the bidirectional reflectance distribution function, or BRDF. Using photographs of an object with spatially uniform reflectance, the BRDFs of paints and papers are measured with completeness and accuracy that rival that of measurements obtained using specialized devices. The image-based approach and novel light source positioning technique require only general-purpose equipment, so the cost of the apparatus is low compared to conventional approaches. In addition, very densely sampled data can be measured very quickly, when the wavelength spectrum of the BRDF does not need to be measured in detail.
Cited By
- Chen H, Miller B and Gkioulekas I (2024). 3D Reconstruction with Fast Dipole Sums, ACM Transactions on Graphics, 43:6, (1-19), Online publication date: 19-Dec-2024.
- Mariyanayagam D and Bartoli A (2024). The shading isophotes, Computer Vision and Image Understanding, 248:C, Online publication date: 1-Nov-2024.
- Kuang Z, Zhang Y, Yu H, Agarwala S, Wu S and Wu J Stanford-ORB Proceedings of the 37th International Conference on Neural Information Processing Systems, (46938-46957)
- Cheng T, Ma W, Guan K, Torralba A and Wang S Structure from duplicates Proceedings of the 37th International Conference on Neural Information Processing Systems, (17195-17207)
- Ramamoorthi R and Hanrahan P A Signal-Processing Framework for Inverse Rendering Seminal Graphics Papers: Pushing the Boundaries, Volume 2, (327-338)
- Whitton M (2023). Seminal Graphics Papers: Pushing the Boundaries, Volume 2, 10.1145/3596711, Online publication date: 1-Aug-2023.
- Mubarik M, Kanungo R, Zirr T and Kumar R Hardware Acceleration of Neural Graphics Proceedings of the 50th Annual International Symposium on Computer Architecture, (1-12)
- Li Q, Guo J, Fei Y, Li F and Guo Y NeuLighting: Neural Lighting for Free Viewpoint Outdoor Scene Relighting with Unconstrained Photo Collections SIGGRAPH Asia 2022 Conference Papers, (1-9)
- Lyu L, Tewari A, Leimkühler T, Habermann M and Theobalt C Neural Radiance Transfer Fields for Relightable Novel-View Synthesis with Global Illumination Computer Vision – ECCV 2022, (153-169)
- Zhao Y, Chen H, Sahli H, Lu K and Jiang D Uncertainty-Aware Semi-Supervised Learning of 3D Face Rigging from Single Image Proceedings of the 30th ACM International Conference on Multimedia, (170-179)
- Zhang X, Srinivasan P, Deng B, Debevec P, Freeman W and Barron J (2021). NeRFactor, ACM Transactions on Graphics, 40:6, (1-18), Online publication date: 1-Dec-2021.
- Pediredla A, Veeraraghavan A and Gkioulekas I (2019). Ellipsoidal path connections for time-gated rendering, ACM Transactions on Graphics, 38:4, (1-12), Online publication date: 31-Aug-2019.
- Li D, Langlois T and Zheng C (2018). Scene-aware audio for 360° videos, ACM Transactions on Graphics, 37:4, (1-12), Online publication date: 31-Aug-2018.
- Tetzlaff M and Meyer G Using flash photography and image-based rendering to document cultural heritage artifacts Proceedings of the 14th Eurographics Workshop on Graphics and Cultural Heritage, (137-146)
- Mallett I and Yuksel C Parameterization of Tabulated BRDFs Proceedings of the 33rd Computer Graphics International, (45-48)
- Hegedus R, Lucat A, Redon J and Pacanowski R Isotropic BRDF measurements with quantified uncertainties Proceedings of the Eurographics 2016 Workshop on Material Appearance Modeling, (9-14)
- Nagano K, Collins T, Chen C and Nakano A GPU-based inverse rendering with multi-objective particle swarm optimization SIGGRAPH Asia 2015 Visualization in High Performance Computing, (1-4)
- Aittala M, Weyrich T and Lehtinen J (2015). Two-shot SVBRDF capture for stationary materials, ACM Transactions on Graphics, 34:4, (1-13), Online publication date: 27-Jul-2015.
- Dong Y, Chen G, Peers P, Zhang J and Tong X (2014). Appearance-from-motion, ACM Transactions on Graphics, 33:6, (1-12), Online publication date: 19-Nov-2014.
- Herbort S and Wöhler C (2011). An introduction to image-based 3D surface reconstruction and a survey of photometric stereo methods, 3D Research, 2:3, (1-17), Online publication date: 1-Sep-2011.
- Bai J, Chandraker M, Ng T and Ramamoorthi R A dual theory of inverse and forward light transport Proceedings of the 11th European conference on Computer vision: Part II, (294-307)
- Holroyd M, Lawrence J and Zickler T A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance ACM SIGGRAPH 2010 papers, (1-12)
- Holroyd M, Lawrence J and Zickler T (2010). A coaxial optical scanner for synchronous acquisition of 3D geometry and surface reflectance, ACM Transactions on Graphics, 29:4, (1-12), Online publication date: 26-Jul-2010.
- Zinke A, Rump M, Lay T, Weber A, Andriyenko A and Klein R A practical approach for photometric acquisition of hair color ACM SIGGRAPH Asia 2009 papers, (1-9)
- Gutierrez D, Jensen H, Jarosz W and Donner C Scattering ACM SIGGRAPH ASIA 2009 Courses, (1-620)
- Zinke A, Rump M, Lay T, Weber A, Andriyenko A and Klein R (2009). A practical approach for photometric acquisition of hair color, ACM Transactions on Graphics, 28:5, (1-9), Online publication date: 1-Dec-2009.
- Gutierrez D, Jarosz W, Donner C and Narasimhan S Scattering ACM SIGGRAPH 2009 Courses, (1-397)
- Ghosh A, Chen T, Peers P, Wilson C and Debevec P Estimating specular roughness and anisotropy from second order spherical gradient illumination Proceedings of the Twentieth Eurographics conference on Rendering, (1161-1170)
- Dorsey J and Rushmeier H Light and materials in virtual cities ACM SIGGRAPH 2008 classes, (1-4)
- Rushmeier H The perception of simulated materials ACM SIGGRAPH 2008 classes, (1-12)
- Theobalt C, Ahmed N, Lensch H, Magnor M and Seidel H (2007). Seeing People in Different Light-Joint Shape, Motion, and Reflectance Capture, IEEE Transactions on Visualization and Computer Graphics, 13:4, (663-674), Online publication date: 1-Jul-2007.
- Patow G, Pueyo X and Vinacua A (2007). Technical Section, Computers and Graphics, 31:3, (501-515), Online publication date: 1-Jun-2007.
- Zickler T, Ramamoorthi R, Enrique S and Belhumeur P (2006). Reflectance Sharing, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28:8, (1287-1302), Online publication date: 1-Aug-2006.
- Narasimhan S, Gupta M, Donner C, Ramamoorthi R, Nayar S and Jensen H Acquiring scattering properties of participating media by dilution ACM SIGGRAPH 2006 Papers, (1003-1012)
- Narasimhan S, Gupta M, Donner C, Ramamoorthi R, Nayar S and Jensen H (2006). Acquiring scattering properties of participating media by dilution, ACM Transactions on Graphics, 25:3, (1003-1012), Online publication date: 1-Jul-2006.
- Chandra K, Adabala N and Toyama K Aerial image relighting Proceedings of the 24th international conference on Advances in Computer Graphics, (594-605)
- Magallon J, Patow G, Seron F and Pueyo X Parallelization of inverse design of luminaire reflectors Proceedings of the 6th Eurographics conference on Parallel Graphics and Visualization, (99-107)
- Mahajan D, Ramamoorthi R and Curless B A theory of spherical harmonic identities for BRDF/Lighting transfer and image consistency Proceedings of the 9th European conference on Computer Vision - Volume Part IV, (41-55)
- Birkbeck N, Cobzas D, Sturm P and Jagersand M Variational shape and reflectance estimation under changing light and viewpoints Proceedings of the 9th European conference on Computer Vision - Volume Part I, (536-549)
- Malassiotis S and Strintzis M (2005). Robust face recognition using 2D and 3D data, Pattern Recognition, 38:12, (2537-2548), Online publication date: 1-Dec-2005.
- Magnor M, Kindlmann G, Hansen C and Duric N (2005). Reconstruction and Visualization of Planetary Nebulae, IEEE Transactions on Visualization and Computer Graphics, 11:5, (485-496), Online publication date: 1-Sep-2005.
- Lensch H, Goesele M, Chuang Y, Hawkins T, Marschner S, Matusik W and Mueller G Realistic materials in computer graphics ACM SIGGRAPH 2005 Courses, (1-es)
- Zickler T, Enrique S, Ramamoorthi R and Belhumeur P Reflectance sharing: image-based rendering from a sparse set of images Proceedings of the Sixteenth Eurographics conference on Rendering Techniques, (253-264)
- Ngan A, Durand F and Matusik W Experimental analysis of BRDF models Proceedings of the Sixteenth Eurographics conference on Rendering Techniques, (117-126)
- Hawkins T, Einarsson P and Debevec P A dual light stage Proceedings of the Sixteenth Eurographics conference on Rendering Techniques, (91-98)
- Stark M, Arvo J and Smits B (2005). Barycentric Parameterizations for Isotropic BRDFs, IEEE Transactions on Visualization and Computer Graphics, 11:2, (126-138), Online publication date: 1-Mar-2005.
- Magnor M, Kindlmann G and Hansen C Constrained Inverse Volume Rendering for Planetary Nebulae Proceedings of the conference on Visualization '04, (83-90)
- Tsumura N, Ojima N, Sato K, Shiraishi M, Shimizu H, Nabeshima H, Akazaki S, Hori K and Miyake Y Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin ACM SIGGRAPH 2003 Papers, (770-779)
- Gardner A, Tchou C, Hawkins T and Debevec P Linear light source reflectometry ACM SIGGRAPH 2003 Papers, (749-758)
- Tsumura N, Ojima N, Sato K, Shiraishi M, Shimizu H, Nabeshima H, Akazaki S, Hori K and Miyake Y (2003). Image-based skin color and texture analysis/synthesis by extracting hemoglobin and melanin information in the skin, ACM Transactions on Graphics, 22:3, (770-779), Online publication date: 1-Jul-2003.
- Gardner A, Tchou C, Hawkins T and Debevec P (2003). Linear light source reflectometry, ACM Transactions on Graphics, 22:3, (749-758), Online publication date: 1-Jul-2003.
- Luong Q, Fua P and Leclerc Y (2002). The Radiometry of Multiple Images, IEEE Transactions on Pattern Analysis and Machine Intelligence, 24:1, (19-33), Online publication date: 1-Jan-2002.
- Hawkins T, Cohen J and Debevec P A photometric approach to digitizing cultural artifacts Proceedings of the 2001 conference on Virtual reality, archeology, and cultural heritage, (333-342)
- Bernardini F, Martin I and Rushmeier H (2001). High-Quality Texture Reconstruction from Multiple Scans, IEEE Transactions on Visualization and Computer Graphics, 7:4, (318-332), Online publication date: 1-Oct-2001.
- Ramamoorthi R and Hanrahan P A signal-processing framework for inverse rendering Proceedings of the 28th annual conference on Computer graphics and interactive techniques, (117-128)
- Debevec P (1999). Image-based modeling and lighting, ACM SIGGRAPH Computer Graphics, 33:4, (46-50), Online publication date: 4-Nov-1999.
- Yu Y, Debevec P, Malik J and Hawkins T Inverse global illumination Proceedings of the 26th annual conference on Computer graphics and interactive techniques, (215-224)
- Marschner S, Westin S, Lafortune E, Torrance K and Greenberg D Image-based BRDF measurement including human skin Proceedings of the 10th Eurographics conference on Rendering, (131-144)
- Rocchini C, Cignoni P, Montani C and Scopigno R Multiple textures stitching and blending on 3D objects Proceedings of the 10th Eurographics conference on Rendering, (119-130)
Recommendations
Real-time multi-perspective rendering on graphics hardware
SIGGRAPH '06: ACM SIGGRAPH 2006 SketchesMulti-perspective rendering has a variety of applications; examples include lens refraction, curved mirror reflection, caustics, as well depiction and visualization. However, multi-perspective rendering is not yet practical on polygonal graphics ...