More Web Proxy on the site http://driver.im/

research-article

Public Access

Deep high dynamic range imaging of dynamic scenes

Authors:

Nima Khademi Kalantari,

Ravi RamamoorthiAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 36, Issue 4

Article No.: 144, Pages 1 - 12

https://doi.org/10.1145/3072959.3073609

Published: 20 July 2017 Publication History

Abstract

Producing a high dynamic range (HDR) image from a set of images with different exposures is a challenging process for dynamic scenes. A category of existing techniques first register the input images to a reference image and then merge the aligned images into an HDR image. However, the artifacts of the registration usually appear as ghosting and tearing in the final HDR images. In this paper, we propose a learning-based approach to address this problem for dynamic scenes. We use a convolutional neural network (CNN) as our learning model and present and compare three different system architectures to model the HDR merge process. Furthermore, we create a large dataset of input LDR images and their corresponding ground truth HDR images to train our system. We demonstrate the performance of our system by producing high-quality HDR images from a set of three LDR images. Experimental results show that our method consistently produces better results than several state-of-the-art approaches on challenging scenes.

Supplementary Material

ZIP File (a144-kalantari.zip)

Supplemental files.

Download
324.08 MB

MP4 File (papers-0141.mp4)

Download
508.41 MB

References

[1]

A. Badki, N. Khademi Kalantari, and P. Sen. 2015. Robust Radiometric Calibration for Dynamic Scenes in the Wild. In IEEE ICCP. 1--10.

[2]

Luca Bogoni. 2000. Extending Dynamic Range of Monochrome and Color Images through Fusion. In IEEE ICPR. 3007--3016.

[3]

Z. Chen, H. Jin, Z. Lin, S. Cohen, and Y. Wu. 2013. Large Displacement Optical Flow from Nearest Neighbor Fields. In IEEE CVPR. 2443--2450.

Digital Library

[4]

Z. Cheng, Q. Yang, and B. Sheng. 2015. Deep Colorization. In IEEE ICCV. 415--423.

Digital Library

[5]

Paul E. Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In ACM SIGGRAPH. 369--378.

[6]

A. Dosovitskiy, P. Fischery, E. Ilg, P. HÃd'usser, C. Hazirbas, V. Golkov, P. v. d. Smagt, D. Cremers, and T. Brox. 2015. FlowNet: Learning Optical Flow with Convolutional Networks. In ICCV. 2758--2766.

[7]

John Flynn, Ivan Neulander, James Philbin, and Noah Snavely. 2016. DeepStereo: Learning to Predict New Views from the WorldâĂ&Zacute;s Imagery. In IEEE CVPR. 5515--5524.

[8]

Jan Froehlich, Stefan Grandinetti, Bernd Eberhardt, Simon Walter, Andreas Schilling, and Harald Brendel. 2014. Creating cinematic wide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays. SPIE 9023 (2014), 90230X--90230X-10.

[9]

Brian Funt and Lilong Shi. 2010. The Rehabilitation of MaxRGB. Color and Imaging Conference 2010, 1 (2010), 256--259.

[10]

O. Gallo, N. Gelfand, W. Chen, M. Tico, and K. Pulli. 2009. Artifact-free High Dynamic Range Imaging. In IEEE ICCP. 1--7.

[11]

O. Gallo, A. Troccoli, J. Hu, K. Pulli, and J. Kautz. 2015. Locally non-rigid registration for mobile HDR photography. In IEEE CVPRW. 48--55.

[12]

Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, Vol. 9. 249--256.

[13]

I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014. Generative Adversarial Networks. ArXiv e-prints (June 2014). arXiv:stat.ML/1406.2661

[14]

M. Granados, B. Ajdin, M. Wand, C. Theobalt, H. P. Seidel, and H. P. A. Lensch. 2010. Optimal HDR reconstruction with linear digital cameras. In IEEE CVPR. 215--222.

[15]

Miguel Granados, Kwang In Kim, James Tompkin, and Christian Theobalt. 2013. Automatic Noise Modeling for Ghost-free HDR Reconstruction. ACM TOG 32, 6, Article 201 (2013), 10 pages.

[16]

T. Grosch. 2006. Fast and Robust High Dynamic Range Image Generation with Camera and Object Movement. In Vision, Modeling and Visualization. 277--284.

[17]

Michael D. Grossberg and Shree K. Nayar. 2003. Determining the Camera Response from Images: What Is Knowable? IEEE PAMI 25, 11 (Nov. 2003), 1455--1467.

Digital Library

[18]

Yoav HaCohen, Eli Shechtman, Dan B. Goldman, and Dani Lischinski. 2011. Non-rigid dense correspondence with applications for image enhancement. ACM TOG 30, 4, Article 70 (2011), 10 pages.

[19]

Saghi Hajisharif, Joel Kronander, and Jonas Unger. 2015. Adaptive dualISO HDR reconstruction. EURASIP Journal on Image and Video Processing 2015, 1 (2015), 41.

[20]

S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In CVPR. 553--560.

[21]

Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst Photography for High Dynamic Range and Low-light Imaging on Mobile Cameras. ACM TOG 35, 6, Article 192 (Nov. 2016), 12 pages.

[22]

Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, Karen Egiazarian, Jan Kautz, and Kari Pulli. 2014. FlexISP: A Flexible Camera Image Processing Framework. ACM TOG 33, 6, Article 231 (Nov. 2014), 13 pages.

[23]

Yong Seok Heo, Kyoung Mu Lee, Sang Uk Lee, Youngsu Moon, and Joonhyuk Cha. 2010. Ghost-free high dynamic range imaging. In ACCV, Vol. 4. 486--500.

[24]

Jun Hu, Orazio Gallo, and Kari Pulli. 2012. Exposure Stacks of Live Scenes with Hand-Held Cameras. Springer Berlin Heidelberg, Berlin, Heidelberg, 499--512.

[25]

J. Hu, O. Gallo, K. Pulli, and X. Sun. 2013. HDR Deghosting: How to Deal with Saturation?. In IEEE CVPR. 1163--1170.

[26]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2016. Let There Be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification. ACM TOG 35, 4, Article 110 (July 2016), 11 pages.

[27]

Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. 2016. FlowNet 2.0: Evolution of Optical Flow Estimation with Deep Networks. CoRR abs/1612.01925 (2016).

[28]

K. Jacobs, C. Loscos, and G. Ward. 2008. Automatic High-Dynamic Range Image Generation for Dynamic Scenes. IEEE Computer Graphics and Applications 28, 2 (Mar.-Apr. 2008), 84--93.

Digital Library

[29]

T. Jinno and M. Okuda. 2008. Motion blur free HDR image acquisition using multiple exposures. In IEEE ICIP. 1304--1307.

[30]

Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. 2016. Learning-based View Synthesis for Light Field Cameras. ACM TOG 35, 6, Article 193 (Nov. 2016), 10 pages.

[31]

Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. 2003. High dynamic range video. ACM TOG 22, 3 (2003), 319--325.

Digital Library

[32]

Kanita Karaduzovic-Hadziabdic, Jasminka Hasic Telalovic, and Rafal Mantiuk. 2016. Subjective and Objective Evaluation of Multi-exposure High Dynamic Range Image Deghosting Methods. In EG, T. Bashford-Rogers and L. P. Santos (Eds.). The Eurographics Association.

[33]

E.A. Khan, A.O. Akyüz, and E. Reinhard. 2006. Ghost Removal in High Dynamic Range Images. In IEEE ICIP. 2005--2008.

[34]

C. Lee, Y. Li, and V. Monga. 2014. Ghost-Free High Dynamic Range Imaging via Rank Minimization. IEEE SPL 21, 9 (Sept 2014), 1045--1049.

[35]

Z. Li, J. Zheng, Z. Zhu, and S. Wu. 2014. Selectively Detail-Enhanced Fusion of Differently Exposed Images With Moving Objects. IEEE Transactions on Image Processing 23, 10 (Oct 2014), 4372--4382.

[36]

C. Liu. 2009. Beyond Pixels: Exploring New Representations and Applications for Motion Analysis. Doctoral thesis. Massachusetts Institute of Technology.

[37]

Bruce D. Lucas and Takeo Kanade. 1981. An Iterative Image Registration Technique with an Application to Stereo Vision. In IJCAI. 674--679.

[38]

Steve Mann and Rosalind W. Picard. 1995. On Being 'undigital' With Digital Cameras: Extending Dynamic Range By Combining Differently Exposed Pictures. In Proc. of Society for Imaging Science and Technology. 442--448.

[39]

Rafat Mantiuk, Kil Joong Kim, Allan G. Rempel, and Wolfgang Heidrich. 2011. HDR-VDP-2: A Calibrated Visual Metric for Visibility and Quality Predictions in All Luminance Conditions. ACM TOG 30, 4, Article 40 (July 2011), 14 pages.

[40]

M. McGuire, W. Matusik, H. Pfister, B. Chen, J.F. Hughes, and S.K. Nayar. 2007. Optical Splitting Trees for High-Precision Monocular Imaging. IEEE Computer Graphics and Applications 27, 2 (march-april 2007), 32--42.

Digital Library

[41]

T. H. Oh, J. Y. Lee, Y. W. Tai, and I. S. Kweon. 2015. Robust High Dynamic Range Imaging by Rank Minimization. IEEE PAMI 37, 6 (2015), 1219--1232.

[42]

F. Pece and J. Kautz. 2010. Bitmap Movement Detection: HDR for Dynamic Scenes. In CVMP. 1--8.

[43]

Photomatix. 2017. Commercially-available HDR processing software. (2017). http://www.hdrsoft.com/.

[44]

Shanmuganathan Raman and Subhasis Chaudhuri. 2011. Reconstruction of high contrast images for dynamic scenes. The Visual Computer 27, 12 (Dec. 2011), 1099--1114.

Digital Library

[45]

David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1986. Learning Representations by Back-propagating Errors. Nature 323 (1986), 533--536.

[46]

Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM TOG 31, 6, Article 203 (Nov. 2012), 11 pages.

[47]

Ana Serrano, Felix Heide, Diego Gutierrez, Gordon Wetzstein, and Belen Masia. 2016. Convolutional Sparse Coding for High Dynamic Range Imaging. CGF 35, 2 (2016), 153--163.

Digital Library

[48]

Michael D. Tocci, Chris Kiser, Nora Tocci, and Pradeep Sen. 2011. A versatile HDR video production system. ACM TOG 30, 4, Article 41 (July 2011), 10 pages.

[49]

A. Tomaszewska and R. Mantiuk. 2007. Image Registration for Multi-exposure High Dynamic Range Image Acquisition. In WSCG.

[50]

Okan Tarhan Tursun, Ahmet Oğuz Akyüz, Aykut Erdem, and Erkut Erdem. 2016. An Objective Deghosting Quality Metric for HDR Images. CGF 35, 2 (2016), 139--152.

Digital Library

[51]

Okan Tarhan Tursun, Ahmet OÄ§uz AkyAijz, Aykut Erdem, and Erkut Erdem. 2015. The State of the Art in HDR Deghosting: A Survey and Evaluation. CGF 34, 2 (2015).

[52]

Andrea Vedaldi and Karel Lenc. 2015. MatConvNet: Convolutional neural networks for Matlab. In ACMMM. 689--692.

[53]

Greg Ward. 2003. Fast, Robust Image Registration for Compositing High Dynamic Range Photographs from Hand-Held Exposures. Journal of Graphics, GPU, and Game Tools 8, 2 (2003), 17--30.

[54]

L. Zhang, A. Deshpande, and X. Chen. 2010. Denoising vs. deblurring: HDR imaging techniques using moving cameras. In CVPR. 522--529.

[55]

Wei Zhang and Wai-Kuen Cham. 2012. Gradient-Directed Multiexposure Composition. IEEE TIP 21, 4 (April 2012), 2318--2323.

Digital Library

[56]

H. Zhao, B. Shi, C. Fernandez-Cull, S. K. Yeung, and R. Raskar. 2015. Unbounded High Dynamic Range Photography Using a Modulo Camera. In ICCP 2015. 1--10.

[57]

J. Zheng, Z. Li, Z. Zhu, S. Wu, and S. Rahardja. 2013. Hybrid Patching for a Sequence of Differently Exposed Images With Moving Objects. IEEE TIP 22, 12 (Dec 2013), 5190--5201.

Digital Library

[58]

Henning Zimmer, AndrÃl's Bruhn, and Joachim Weickert. 2011. Freehand HDR Imaging of Moving Scenes with Simultaneous Resolution Enhancement. CGF 30, 2 (April 2011), 405--414.

Cited By

Yang XZheng HSu Y(2025)High dynamic range image tone mapping based on variational image decomposition and color correctionOptics & Laser Technology10.1016/j.optlastec.2024.111873181(111873)Online publication date: Feb-2025
https://doi.org/10.1016/j.optlastec.2024.111873
C.R. JS. V(2025)A multi-exposure image fusion using adaptive color dissimilarity and dynamic equalization techniquesJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104350107(104350)Online publication date: Mar-2025
https://doi.org/10.1016/j.jvcir.2024.104350
Park YKang S(2024)Effective Ghost Artifact Removal through Transformer-based HDR Image ReconstructionJOURNAL OF BROADCAST ENGINEERING10.5909/JBE.2024.29.3.23529:3(235-241)Online publication date: 31-May-2024
https://doi.org/10.5909/JBE.2024.29.3.235
Show More Cited By

Index Terms

Deep high dynamic range imaging of dynamic scenes
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Computational photography

Recommendations

Comparison of Deghosting Algorithms for Multi-exposure High Dynamic Range Imaging
SCCG '13: Proceedings of the 29th Spring Conference on Computer Graphics

High dynamic range (HDR) images can be generated by capturing a sequence of low dynamic range (LDR) images of the same scene with different exposures and then merging those images to create an HDR image. During capturing of LDR images, any changes in ...
Robust artifact-free high dynamic range imaging of dynamic scenes

The irradiance range of the real-world scene is often beyond the capability of digital cameras. Therefore, High Dynamic Range (HDR) images can be generated by fusing images with different exposure of the same scene. However, moving objects pose the most ...
High dynamic range imaging by sparse representation

High dynamic range (HDR) imaging technology is becoming increasingly popular in various applications. A common approach to get an HDR image is the multiple exposed images fusion. However, the phenomenon of ghosting artifacts is brought in for the scene ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 36, Issue 4

August 2017

2155 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3072959

Issue’s Table of Contents

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 July 2017

Published in TOG Volume 36, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

UC San Diego Center for Visual Computing
Office of Naval Research
National Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

423
Total Citations
View Citations
3,071
Total Downloads

Downloads (Last 12 months)671
Downloads (Last 6 weeks)139

Reflects downloads up to 24 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yang XZheng HSu Y(2025)High dynamic range image tone mapping based on variational image decomposition and color correctionOptics & Laser Technology10.1016/j.optlastec.2024.111873181(111873)Online publication date: Feb-2025
https://doi.org/10.1016/j.optlastec.2024.111873
C.R. JS. V(2025)A multi-exposure image fusion using adaptive color dissimilarity and dynamic equalization techniquesJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104350107(104350)Online publication date: Mar-2025
https://doi.org/10.1016/j.jvcir.2024.104350
Park YKang S(2024)Effective Ghost Artifact Removal through Transformer-based HDR Image ReconstructionJOURNAL OF BROADCAST ENGINEERING10.5909/JBE.2024.29.3.23529:3(235-241)Online publication date: 31-May-2024
https://doi.org/10.5909/JBE.2024.29.3.235
Wang Xianfeng 王Liu Shiben 刘Tian Jiandong 田Zhao Juanping 赵Liu yajing 刘Hao Chunhui 郝(2024)基于双重注意力网络的高动态范围图像重建Laser & Optoelectronics Progress10.3788/LOP23177061:12(1237005)Online publication date: 2024
https://doi.org/10.3788/LOP231770
Han YYu XLuan HSuo J(2024)Event-Assisted Object Tracking on High-Speed Drones in Harsh Illumination EnvironmentDrones10.3390/drones80100228:1(22)Online publication date: 16-Jan-2024
https://doi.org/10.3390/drones8010022
Lin HLin YLin WChang C(2024)Reconstructing High Dynamic Range Image from a Single Low Dynamic Range Image Using Histogram LearningApplied Sciences10.3390/app1421984714:21(9847)Online publication date: 28-Oct-2024
https://doi.org/10.3390/app14219847
Mao RXu HCui R(2024)FPN-Retinex Enables Semi-Supervised Low-Light Image Enhancement Via Feature Pyramid Network With Flexible Backbones2024 43rd Chinese Control Conference (CCC)10.23919/CCC63176.2024.10662161(7325-7332)Online publication date: 28-Jul-2024
https://doi.org/10.23919/CCC63176.2024.10662161
Wu SVembar DSochenov APanneer SKim SKaplanyan AYan L(2024)GFFE: G-buffer Free Frame Extrapolation for Low-latency Real-time RenderingACM Transactions on Graphics10.1145/368792343:6(1-15)Online publication date: 19-Dec-2024
https://dl.acm.org/doi/10.1145/3687923
Cui JJiang WPeng ZPan ZCao ZCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)Exposure Completing for Temporally Consistent Neural High Dynamic Range Video RenderingProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3680935(10027-10035)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3680935
Zhu XZhang YFang YJiao JFu QLi PLi W(2024)Detail-preserving multi-scale exposure fusion via edge-preserving smoothing pyramidsInternational Conference on Image, Signal Processing, and Pattern Recognition (ISPP 2024)10.1117/12.3034003(206)Online publication date: 13-Jun-2024
https://doi.org/10.1117/12.3034003
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents