More Web Proxy on the site http://driver.im/

Article

Performance relighting and reflectance transformation with time-multiplexed illumination

Authors:

Andreas Wenger,

Andrew Gardner,

Paul DebevecAuthors Info & Claims

SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

Pages 756 - 764

https://doi.org/10.1145/1186822.1073258

Published: 01 July 2005 Publication History

Abstract

We present a technique for capturing an actor's live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-multiplexed basis lighting conditions, and to record these conditions with a high-speed video camera so that many conditions are recorded in the span of the desired output frame interval. We investigate several lighting bases for representing the sphere of incident illumination using a set of discrete LED light sources, and we estimate and compensate for subject motion using optical flow and image warping based on a set of tracking frames inserted into the lighting basis. To composite the illuminated performance into a new background, we include a time-multiplexed matte within the basis. We also show that the acquired data enables time-varying surface normals, albedo, and ambient occlusion to be estimated, which can be used to transform the actor's reflectance to produce both subtle and stylistic effects.

Supplementary Material

MP4 File (pps043.mp4)

Download
42.41 MB

References

[1]

Black, M. J., and Anandan, P. 1993. A framework for the robust estimation of optical flow. In Fourth International Conf. on Computer Vision, 231--236.

[2]

Brostow, G. J., and Essa. I. 2001. Image-based motion blur for stop motion animation. In Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, 561--566.

Digital Library

[3]

Calahan, S. 2000. Advanced Renderman: Creating CGI for Motion Pictures. Morgan Kaufman Publishers, San Francisco, ch. Storytelling through lighting, a computer perspective, 337382.

[4]

Debevec. P., Hawkins, T., Tchou. C., Duiker, H.-P., Sarokin, W., and Sagar, M. 2000. Acquiring the reflectance field of a human face. Proceedings of SIGGRAPH 2000 (July), 145--156.

Digital Library

[5]

Debevec, P., Wenger, A., Tchou, C., Gardner, A., Waese, J., and Hawkins, T. 2002. A lighting reproduction approach to live-action compositing. ACM Transactions on Graphics 21, 3 (July), 547--556.

Digital Library

[6]

Fielding, R. 1985. The Technique of Special Effects Cinematography, 4th ed. Hastings House, New York.

[7]

Georghiades. A., Belhumeur. P., and Kriegman, D. 1999. Illumination-based image synthesis: Creating novel images of human faces under differing pose and lighting. In IEEE Workshop on Multi-View Modeling and Analysis of Visual Scenes, 47--54.

Digital Library

[8]

Georghiades. A. S. 2003. Recovering 3-d shape and reflectance from a small number of photographs. In Eurographics Symposium on Rendering: 14th Eurographics Workshop on Rendering, 230--240.

Digital Library

[9]

Gershbein, R., and Hanrahan, P. M. 2000. A fast relighting engine for interactive cinematic lighting design. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings, Annual Conference Series, 353--358.

Digital Library

[10]

Greene, N. 1986. Environment mapping and other application of world projections. IEEE Computer Graphics and Applications 6, 11 (November), 21--29.

Digital Library

[11]

Guenter, B., Grimm. C., Wood, D., Malvar, H., and Pighin, F. July 1998. Making faces. Proceedings of SIGGRAPH 98, 55--66.

Digital Library

[12]

Haeberli, P. 1992. Synthetic lighting for photography. Available at http://www.sgi.com/grafica/synth/index.html, January.

[13]

Hager, G. D., and Belhumeur, P. N. 1996. Real-time tracking of image regions with changes in geometry and illumination. In Proc. IEEE Conf. on Comp. Vision and Patt. Recog., 403--410.

Digital Library

[14]

Hall, M. 1998. Combinatorial Theory, 2nd ed. Wiley, New York.

Digital Library

[15]

Harwit, M., and Sloane, N. J. A. 1979. Hadamard transform optics. Academic Press, New York.

[16]

Hawkins, T., Wenger, A., Tchou, C., and Debevec, A. G. F. G. P. 2004. Animatable facial reflectance fields. In Eurographics Symposium on Rendering: 15th Eurographics Workshop on Rendering.

Digital Library

[17]

Hewlett-Packard Components Group. 1998. Noise sources in cmos image sensors. Tech. rep., Hewlett-Packard.

[18]

Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video. ACM Transactions on Graphics 22, 3 (July), 319--325.

Digital Library

[19]

Landis, H., 2002. Production-ready global illumination. Course Notes for SIGGRAPH 2002 Course 16, RenderMan in Production.

[20]

Malzbender, T., Gelb, D., and Wolters, H. 2001. Polynomial texture maps. Proceedings of SIGGRAPH 2001 (August), 519--528.

Digital Library

[21]

Marks, J., Andalman, B., Beardsley, P. A., Freeman, W., Gibson, S., Hodgins, J. K., Kang. T., Mirtich, B., Pfister, H., Ruml, W., Ryall, K., Seims, J., and Shieber, S. 1997. Design galleries: A general approach to setting parameters for computer graphics and animation. In Proceedings of SIGGRAPH 97, Computer Graphics Proceedings, Annual Conference Series, 389--400.

Digital Library

[22]

Marschner, S., Guenter, B., and Raghupathy, S. 2000. Modeling and rendering for realistic facial animation. In Rendering Techniques 2000: 11th Eurographics Workshop on Rendering, 231--242.

Digital Library

[23]

Masselus, V., Peers, P., Dutre, P., and Willems, Y. D. 2004. Smooth reconstruction and compact representation of reflectance functions for image-based relighting. In 15th Eurographics Symposium on Rendering, no. Norrkoping, Sweden.

Digital Library

[24]

Miller, G. S., and Hoffman, C. R. 1984. Illumination and reflection maps: Simulated objects in simulated and real environments. In SIGGRAPH 84 Course Notes for Advanced Computer Graphics Animation.

[25]

Ng, R., Ramamoorthi, R., and Hanrahan, P. 2003. All-frequency shadows using non-linear wavelet lighting approximation. ACM Transactions on Graphics 22, 3 (July). 376--381.

Digital Library

[26]

Nimeroff, J. S., Simoncelli, E., and Dorsey, J. 1994, Efficient re-rendering of naturally illuminated environments. In Fifth Eurographics Workshop on Rendering, 359--373.

[27]

Nishino, K., and Nayar, S. K. 2004. Eyes for relighting. ACM Transactions on Graphics 23, 3 (Aug.), 704--711.

Digital Library

[28]

Petrović, L., Fujito, B., Williams, L., and Finkelstein, A. 2000. Shadows for cel animation. In Proceedings of ACM SIGGRAPH 2000, Computer Graphics Proceedings. Annual Conference Series, 511--516.

Digital Library

[29]

Porter, T., and Duff, T. 1984. Compositing digital images. In Computer Graphics (Proceedings of SIGGRAPH 84), vol. 18, 253--259.

Digital Library

[30]

Raskar, R., Tan. K.-H., Feris, R., Yu. J., and Turk, M. 2004. Non photorealistic camera: depth edge detection and stylized rendering using multiflash imaging. ACM Transactions on Graphics 23, 3 (Aug.), 679--688.

Digital Library

[31]

Rusinkiewicz, S., Hall-Holt. O., and Levoy, M. 2002. Real-time 3d model acquisition. ACM Transactions on Graphics 21, 3 (July), 438--446.

Digital Library

[32]

Schechner, Y. Y., Nayar, S. K., and Belhumeur, P. 2003. A theory of multiplexed illumination. In International Conference on Computer Vision.

Digital Library

[33]

Sloan, P.-P., Kautz, J., and Snyder, J. 2002. Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments. ACM Transactions on Graphics 21, 3 (July), 527--536.

Digital Library

[34]

Sloan, N. J. A., 1999. A library of hadamard matrices. http://www.research.att.com/~njas/hadamard/.

[35]

Smith, A. R., and Blinn, J. F. 1996. Blue screen matting. In Proceedings of SIGGRAPH 96, 259--268.

Digital Library

[36]

Trumbull, D. 2000. Personal communication. January.

[37]

Wen, Z., Liu, Z., and Huang, T. S. 2003. Face relighting with radiance environment maps. In 2003 Conference on Computer Vision and Pattern Recognition (CVPR 2003), 158--165.

[38]

Williams, L. 1991. Shading in two dimensions. In Graphics Interface '91, 143--151.

[39]

Woodham, R. J. 1980. Photometric method for determining surface orientation from multiple images. Optical Engineering 19, 1, 139--144.

[40]

Wu, X. 1991. An efficient antialiasing technique. In SIGGRAPH 91: Proceedings of the 18th annual conference on Computer graphics and interactive techniques, ACM Press, 143--152.

Digital Library

[41]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: high resolution capture for modeling and animation. ACM Transactions on Graphics 23, 3 (Aug.), 548--558.

Digital Library

[42]

Zongker, D. E., Werner, D. M., Curless, B., and Salesin, D. H. 1999. Environment matting and compositing. Proceedings of SIGGRAPH 99 (August), 205--214.

Digital Library

Cited By

Pandey REscolano SLegendre CHäne CBouaziz SRhemann CDebevec PFanello S(2021)Total relightingACM Transactions on Graphics10.1145/3450626.345987240:4(1-21)Online publication date: Aug-2021
https://doi.org/10.1145/3450626.3459872
Huang BLing H(2021)DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.306777127:5(2725-2735)Online publication date: May-2021
https://doi.org/10.1109/TVCG.2021.3067771
Meka APandey RHäne COrts-Escolano SBarnum PDavid-Son PErickson DZhang YTaylor JBouaziz SLegendre CMa WOverbeck RBeeler TDebevec PIzadi STheobalt CRhemann CFanello S(2020)Deep relightable texturesACM Transactions on Graphics10.1145/3414685.341781439:6(1-21)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417814
Show More Cited By

Index Terms

Performance relighting and reflectance transformation with time-multiplexed illumination
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Tracking
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Image manipulation
      1. Texturing

Recommendations

Total relighting: learning to relight portraits for background replacement

We propose a novel system for portrait relighting and background replacement, which maintains high-frequency boundary details and accurately synthesizes the subject's appearance as lit by novel illumination, thereby producing realistic composite images ...
Performance relighting and reflectance transformation with time-multiplexed illumination

We present a technique for capturing an actor's live-action performance in such a way that the lighting and reflectance of the actor can be designed and modified in postproduction. Our approach is to illuminate the subject with a sequence of time-...
Relighting with the Reflected Irradiance Field: Representation, Sampling and Reconstruction

Image-based relighting (IBL) is a technique to change the illumination of an image-based object/scene. In this paper, we define a representation called the reflected irradiance field which records the light reflected from a scene as viewed at a fixed ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

July 2005

826 pages

ISBN:9781450378253

DOI:10.1145/1186822

Editor:
Markus Gross
Eidgenössische Technische Hochschule Zürich

Copyright © 2005 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2005

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGGRAPH05

Sponsor:

SIGGRAPH

SIGGRAPH05: Special Interest Group on Computer Graphics and Interactive Techniques Conference

July 31 - August 4, 2005

California, Los Angeles

Acceptance Rates

SIGGRAPH '05 Paper Acceptance Rate 98 of 461 submissions, 21%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
1,719
Total Downloads

Downloads (Last 12 months)5
Downloads (Last 6 weeks)0

Reflects downloads up to 07 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Pandey REscolano SLegendre CHäne CBouaziz SRhemann CDebevec PFanello S(2021)Total relightingACM Transactions on Graphics10.1145/3450626.345987240:4(1-21)Online publication date: Aug-2021
https://doi.org/10.1145/3450626.3459872
Huang BLing H(2021)DeProCams: Simultaneous Relighting, Compensation and Shape Reconstruction for Projector-Camera SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2021.306777127:5(2725-2735)Online publication date: May-2021
https://doi.org/10.1109/TVCG.2021.3067771
Meka APandey RHäne COrts-Escolano SBarnum PDavid-Son PErickson DZhang YTaylor JBouaziz SLegendre CMa WOverbeck RBeeler TDebevec PIzadi STheobalt CRhemann CFanello S(2020)Deep relightable texturesACM Transactions on Graphics10.1145/3414685.341781439:6(1-21)Online publication date: 27-Nov-2020
https://dl.acm.org/doi/10.1145/3414685.3417814
Yu ZShin Yoon JLee IVenkatesh PPark JYu JPark H(2020)HUMBI: A Large Multiview Dataset of Human Body Expressions2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR42600.2020.00306(2987-2997)Online publication date: Jun-2020
https://doi.org/10.1109/CVPR42600.2020.00306
Meka AHäne CPandey RZollhöfer MFanello SFyffe GKowdle AYu XBusch JDourgarian JDenny PBouaziz SLincoln PWhalen MHarvey GTaylor JIzadi STagliasacchi ADebevec PTheobalt CValentin JRhemann C(2019)Deep reflectance fieldsACM Transactions on Graphics10.1145/3306346.332302738:4(1-12)Online publication date: 12-Jul-2019
https://dl.acm.org/doi/10.1145/3306346.3323027
Sun TBarron JTsai YXu ZYu XFyffe GRhemann CBusch JDebevec PRamamoorthi R(2019)Single image portrait relightingACM Transactions on Graphics10.1145/3306346.332300838:4(1-12)Online publication date: 12-Jul-2019
https://dl.acm.org/doi/10.1145/3306346.3323008
Sagawa RSatoh Y(2017)Illuminant-Camera Communication to Observe Moving Objects under Strong External Light by Spread Spectrum Modulation2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR.2017.249(2317-2325)Online publication date: Jul-2017
https://doi.org/10.1109/CVPR.2017.249
Zhou WPan JLi PWei XLiu Z(2017)A Distributed Stream Computing Architecture for Dynamic Light-Field Acquisition and Rendering SystemLNCS on Transactions on Edutainment XIII - Volume 1009210.1007/978-3-662-54395-5_11(123-132)Online publication date: 1-Jan-2017
https://dl.acm.org/doi/10.1007/978-3-662-54395-5_11
Maître H(2017)BibliographyFrom Photon to Pixel10.1002/9781119402442.biblio(417-437)Online publication date: 11-Feb-2017
https://doi.org/10.1002/9781119402442.biblio
Gotardo PSimon TSheikh YMatthews I(2015)Photogeometric Scene Flow for High-Detail Dynamic 3D ReconstructionProceedings of the 2015 IEEE International Conference on Computer Vision (ICCV)10.1109/ICCV.2015.103(846-854)Online publication date: 7-Dec-2015
https://dl.acm.org/doi/10.1109/ICCV.2015.103
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents