More Web Proxy on the site http://driver.im/

research-article

Intrinsic Harmonization for Illumination-Aware Image Compositing

Authors:

S. Mahdi H. Miangoleh,

Yağız AksoyAuthors Info & Claims

SA '23: SIGGRAPH Asia 2023 Conference Papers

Article No.: 100, Pages 1 - 10

https://doi.org/10.1145/3610548.3618178

Published: 11 December 2023 Publication History

Abstract

Despite significant advancements in network-based image harmonization techniques, there still exists a domain disparity between typical training pairs and real-world composites encountered during inference. Most existing methods are trained to reverse global edits made on segmented image regions, which fail to accurately capture the lighting inconsistencies between the foreground and background found in composited images. In this work, we introduce a self-supervised illumination harmonization approach formulated in the intrinsic image domain. First, we estimate a simple global lighting model from mid-level vision representations to generate a rough shading for the foreground region. A network then refines this inferred shading to generate a harmonious re-shading that aligns with the background scene. In order to match the color appearance of the foreground and background, we utilize ideas from prior harmonization approaches to perform parameterized image edits in the albedo domain. To validate the effectiveness of our approach, we present results from challenging real-world composites and conduct a user study to objectively measure the enhanced realism achieved compared to state-of-the-art harmonization methods.

Supplemental Material

ZIP File - Code for performing inference as described in "Intrinsic Harmonization for Illumination-Aware Compositing"

This repository contains the code for performing inference of the method described in the paper, published in SIGGRAPH Asia 2023. For additional information, visit https://github.com/compphoto/IntrinsicCompositing

Download
108.58 MB

ZIP File

Supplementary PDF, intrinsic decomposition method tech report, and user study comparisons

Download
69.73 MB

References

[1]

Zhongyun Bao, Chengjiang Long, Gang Fu, Daquan Liu, Yuanzhen Li, Jiaming Wu, and Chunxia Xiao. 2022. Deep Image-based Illumination Harmonization. In Proc. CVPR.

[2]

Anand Bhattad and David A. Forsyth. 2022. Cut-and-Paste Object Insertion by Enabling Deep Image Prior for Reshading. Proc. 3DV (2022).

[3]

Ralph Allan Bradley and Milton E Terry. 1952. Rank Aanalysis of Incomplete Block Designs: I. The Method of Paired Comparisons. Biometrika 39, 3/4 (1952), 324–345.

[4]

Chris Careaga and Yağız Aksoy. 2023. Intrinsic Image Decomposition via Ordinal Shading. SFU Tech. Rep. (2023).

[5]

Wenyan Cong, Xinhao Tao, Li Niu, Jing Liang, Xuesong Gao, Qihao Sun, and Liqing Zhang. 2022. High-resolution image harmonization via collaborative dual transformations. In Proc. CVPR.

[6]

Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and Liqing Zhang. 2020. DoveNet: Deep Image Harmonization via Domain Verification. In Proc. CVPR.

[7]

Ainaz Eftekhar, Alexander Sax, Jitendra Malik, and Amir Zamir. 2021. Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets From 3D Scans. In Proc. ICCV.

[8]

Mathieu Garon, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, and Jean-Francois Lalonde. 2019. Fast Spatially-Varying Indoor Lighting Estimation. In Proc. CVPR.

[9]

David Griffiths, Tobias Ritschel, and Julien Philip. 2022. OutCast: Single Image Relighting with Cast Shadows. Comput. Graph. Forum (2022).

[10]

Zonghui Guo, Zhaorui Gu, Bing Zheng, Junyu Dong, and Haiyong Zheng. 2022. Transformer for Image Harmonization and Beyond. IEEE Trans. Pattern Anal. Mach. Intell. (2022).

[11]

Zonghui Guo, Haiyong Zheng, Yufeng Jiang, Zhaorui Gu, and Bing Zheng. 2021. Intrinsic Image Harmonization. In Proc. CVPR.

[12]

Zhongyun Hu, Ntumba Elie Nsampi, Xue Wang, and Qing Wang. 2021. NeurSF: Neural Shading Field for Image Harmonization. arxiv:2112.01314 [cs.CV]

[13]

Kevin Karsch, Varsha Hedau, David Forsyth, and Derek Hoiem. 2011. Rendering Synthetic Objects into Legacy Photographs. ACM Trans. Graph. 30, 6 (2011).

Digital Library

[14]

Kevin Karsch, Kalyan Sunkavalli, Sunil Hadap, Nathan Carr, Hailin Jin, Rafael Fonte, Michael Sittig, and David Forsyth. 2014. Automatic Scene Inference for 3D Object Compositing. ACM Trans. Graph. 33, 3 (2014).

Digital Library

[15]

Zhanghan Ke, Chunyi Sun, Lei Zhu, Ke Xu, and Rynson W.H. Lau. 2022. Harmonizer: Learning to Perform White-Box Image and Video Harmonization. In Proc. ECCV.

Digital Library

[16]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. Proc. ICLR.

[17]

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. 2023. Segment Anything. (2023). arxiv:2304.02643 [cs.CV]

[18]

Zhengqin Li, Mohammad Shafiei, Ravi Ramamoorthi, Kalyan Sunkavalli, and Manmohan Chandraker. 2020. Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image. In Proc. CVPR.

[19]

Zhengqin Li, Jia Shi, Sai Bi, Rui Zhu, Kalyan Sunkavalli, Miloš Hašan, Zexiang Xu, Ravi Ramamoorthi, and Manmohan Chandraker. 2022. Physically-Based Editing of Indoor Scene Lighting from a Single Image. In Proc. ECCV.

Digital Library

[20]

Zhengqi Li and Noah Snavely. 2018. MegaDepth: Learning Single-View Depth Prediction from Internet Photos. In Proc. CVPR.

[21]

Zicheng Liao, Kevin Karsch, Hongyi Zhang, and David Forsyth. 2019. An Approximate Shading Model with Detail Decomposition for Object Relighting. Int. J. Comput. Vision 127 (2019).

Digital Library

[22]

G. Lin, A. Milan, C. Shen, and I. Reid. 2017. RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation. In Proc. CVPR.

[23]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. 2014. Microsoft COCO: Common Objects in Context. In Proc. ECCV.

[24]

Jorge Lopez-Moreno, Sunil Hadap, Erik Reinhard, and Diego Gutierrez. 2010. Compositing Images through Light Source Detection. Computers & Graphics 34, 6 (2010), 698–707.

Digital Library

[25]

S. Mahdi H. Miangoleh, Zoya Bylinskii, Eric Kee, Eli Shechtman, and Yağız Aksoy. 2023. Realistic Saliency Guided Image Enhancement. Proc. CVPR.

[26]

S. Mahdi H. Miangoleh, Sebastian Dille, Long Mai, Sylvain Paris, and Yağız Aksoy. 2021. Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging. In Proc. CVPR.

[27]

Lukas Murmann, Michael Gharbi, Miika Aittala, and Fredo Durand. 2019. A Multi-Illumination Dataset of Indoor Object Appearance. In Proc. ICCV.

[28]

Baptiste Nicolet, Julien Philip, and George Drettakis. 2020. Repurposing a Relighting Network for Realistic Compositions of Captured Scenes. In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games.

Digital Library

[29]

Rohit Pandey, Sergio Orts-Escolano, Chloe LeGendre, Christian Haene, Sofien Bouaziz, Christoph Rhemann, Paul Debevec, and Sean Fanello. 2021. Total Relighting: Learning to Relight Portraits for Background Replacement. ACM Trans. Graph. (2021).

[30]

F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. 2016. A Benchmark Dataset and Evaluation Methodology for Video Object Segmentation. In Proc. CVPR.

[31]

Julien Philip, Michaël Gharbi, Tinghui Zhou, Alexei A. Efros, and George Drettakis. 2019. Multi-View Relighting Using a Geometry-Aware Network. ACM Trans. Graph. 38, 4 (2019).

Digital Library

[32]

Julien Philip, Sébastien Morgenthaler, Michaël Gharbi, and George Drettakis. 2021. Free-viewpoint Indoor Neural Relighting from Multi-view Stereo. ACM Trans. Graph. (2021).

[33]

René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. 2020. Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-shot Cross-dataset Transfer. IEEE Trans. Pattern Anal. Mach. Intell. (2020).

[34]

Ke Wang, Michaël Gharbi, He Zhang, Zhihao Xia, and Eli Shechtman. 2023. Semi-supervised Parametric Real-world Image Harmonization. In Proc. CVPR.

[35]

Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017. Aggregated Residual Transformations for Deep Neural Networks. In Proc. CVPR.

[36]

Ben Xue, Shenghui Ran, Quan Chen, Rongfei Jia, Binqiang Zhao, and Binqiang Zhao. 2022. DCCF: Deep Comprehensible Color Filter Learning Framework for High-Resolution Image Harmonization. In Proc. ECCV.

Digital Library

[37]

Yu-Ying Yeh, Koki Nagano, Sameh Khamis, Jan Kautz, Ming-Yu Liu, and Ting-Chun Wang. 2022. Learning to Relight Portrait Images via a Virtual Light Stage and Synthetic-to-Real Adaptation. ACM Trans. Graph. (2022).

[38]

Jinsong Zhang, Kalyan Sunkavalli, Yannick Hold-Geoffroy, Sunil Hadap, Jonathan Eisenmann, and Jean-François Lalonde. 2019. All-Weather Deep Outdoor Lighting Estimation. In Proc. CVPR.

Cited By

Dille SCareaga CAksoy Y(2024)Intrinsic Single-Image HDR ReconstructionComputer Vision – ECCV 202410.1007/978-3-031-73247-8_10(161-177)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73247-8_10
Liu SRen ZGupta SWang S(2024)PhysGen: Rigid-Body Physics-Grounded Image-to-Video GenerationComputer Vision – ECCV 202410.1007/978-3-031-73007-8_21(360-378)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73007-8_21
Zhong XHuang XYang XLin GWu Q(2024)DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion ConsistencyComputer Vision – ECCV 202410.1007/978-3-031-72784-9_20(352-370)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72784-9_20

Index Terms

Intrinsic Harmonization for Illumination-Aware Image Compositing
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation

Recommendations

Shadow Harmonization for Realistic Compositing
SA '23: SIGGRAPH Asia 2023 Conference Papers

Compositing virtual objects into real background images requires one to carefully match the scene’s camera parameters, surface geometry, textures, and lighting to obtain plausible renderings. Recent learning approaches have shown many scene properties ...
Spatially-Varying Illumination-Aware Indoor Harmonization
Abstract
In this paper, we address the problem of spatially-varying illumination-aware indoor harmonization. Existing image harmonization works either focus on extracting no more than 2D information (e.g., low-level statistics or image filters) from the ...
Low-Resolution Image Rendering Method Based on Reflected Radiance and Intrinsic Decomposition
CISAI '24: Proceedings of the 2024 7th International Conference on Computer Information Science and Artificial Intelligence
Neural Radiance Fields (NeRF) utilizes neural networks to generate high-quality novel view images, but their rendering speed is slow. Reducing the resolution of input images can accelerate the rendering process, but it affects the quality of the ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Conferences

SA '23: SIGGRAPH Asia 2023 Conference Papers

December 2023

1113 pages

ISBN:9798400703157

DOI:10.1145/3610548

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 December 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Results Reproduced / v1.1

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Natural Sciences and Engineering Research Council of Canada

Conference

SA '23

Sponsor:

SIGGRAPH

SA '23: SIGGRAPH Asia 2023

December 12 - 15, 2023

NSW, Sydney, Australia

Acceptance Rates

Overall Acceptance Rate 178 of 869 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
169
Total Downloads

Downloads (Last 12 months)143
Downloads (Last 6 weeks)10

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Dille SCareaga CAksoy Y(2024)Intrinsic Single-Image HDR ReconstructionComputer Vision – ECCV 202410.1007/978-3-031-73247-8_10(161-177)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73247-8_10
Liu SRen ZGupta SWang S(2024)PhysGen: Rigid-Body Physics-Grounded Image-to-Video GenerationComputer Vision – ECCV 202410.1007/978-3-031-73007-8_21(360-378)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-73007-8_21
Zhong XHuang XYang XLin GWu Q(2024)DeCo: Decoupled Human-Centered Diffusion Video Editing with Motion ConsistencyComputer Vision – ECCV 202410.1007/978-3-031-72784-9_20(352-370)Online publication date: 29-Sep-2024
https://dl.acm.org/doi/10.1007/978-3-031-72784-9_20

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten