[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

End-to-end complex lens design with differentiate ray tracing

Published: 19 July 2021 Publication History

Abstract

Imaging systems have long been designed in separated steps: experience-driven optical design followed by sophisticated image processing. Although recent advances in computational imaging aim to bridge the gap in an end-to-end fashion, the image formation models used in these approaches have been quite simplistic, built either on simple wave optics models such as Fourier transform, or on similar paraxial models. Such models only support the optimization of a single lens surface, which limits the achievable image quality.
To overcome these challenges, we propose a general end-to-end complex lens design framework enabled by a differentiable ray tracing image formation model. Specifically, our model relies on the differentiable ray tracing rendering engine to render optical images in the full field by taking into account all on/off-axis aberrations governed by the theory of geometric optics. Our design pipeline can jointly optimize the lens module and the image reconstruction network for a specific imaging task. We demonstrate the effectiveness of the proposed method on two typical applications, including large field-of-view imaging and extended depth-of-field imaging. Both simulation and experimental results show superior image quality compared with conventional lens designs. Our framework offers a competitive alternative for the design of modern imaging systems.

Supplementary Material

VTT File (3450626.3459674.vtt)
ZIP File (a71-sun.zip)
a71-sun.zip
MP4 File (a71-sun.mp4)
MP4 File (3450626.3459674.mp4)
Presentation.

References

[1]
Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1--9.
[2]
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein Generative Adversarial Networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70 (Sydney, NSW, Australia) (ICML'17). JMLR.org, 214--223.
[3]
Seung-Hwan Baek, Hayato Ikoma, Daniel S Jeon, Yuqi Li, Wolfgang Heidrich, Gordon Wetzstein, and Min H Kim. 2020. End-to-end hyperspectral-depth imaging with learned diffractive optics. arXiv preprint arXiv:2009.00463 (2020).
[4]
Sai Bangaru, Tzu-Mao Li, and Frédo Durand. 2020. Unbiased Warped-Area Sampling for Differentiable Rendering. ACM Transactions on Graphics (TOG) 39, 6 (2020), 245:1--245:18.
[5]
V. Boominathan, J. K. Adams, J. T. Robinson, and A. Veeraraghavan. 2020. PhlatCam: Designed Phase-Mask Based Thin Lensless Camera. IEEE Transactions on Pattern Analysis andMachine Intelligence 42, 7 (2020), 1618--1629.
[6]
David J Brady, Michael E Gehm, Ronald A Stack, Daniel L Marks, David S Kittle, Dathon R Golish, EM Vera, and Steven D Feller. 2012. Multiscale gigapixel photography. Nature 486, 7403 (2012), 386.
[7]
W Thomas Cathey and Edward R Dowski. 2002. New paradigm for imaging systems. Applied Optics 41, 29 (2002), 6080--6092.
[8]
Ayan Chakrabarti. 2016. Learning Sensor Multiplexing Design through Back-propagation. In Advances in Neural Information Processing Systems, D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett (Eds.), Vol. 29. Curran Associates, Inc.
[9]
Julie Chang and Gordon Wetzstein. 2019a. Deep Optics for Monocular Depth Estimation and 3D Object Detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
[10]
Julie Chang and Gordon Wetzstein. 2019b. Deep optics for monocular depth estimation and 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 10193--10202.
[11]
Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. 2018. Learning to See in the Dark. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2018).
[12]
Shane Colburn, Alan Zhan, and Arka Majumdar. 2018. Metasurface optics for full-color computational imaging. Science Advances 4, 2 (2018).
[13]
Oliver Cossairt and Shree Nayar. 2010. Spectral focal sweep: Extended depth of field from chromatic aberrations. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1--8.
[14]
O. Cossairt, C. Zhou, and S.K. Nayar. 2010. Diffusion Coding Photography for Extended Depth of Field. ACM Transactions on Graphics (TOG) (Aug 2010).
[15]
O. S. Cossairt, D. Miau, and S. K. Nayar. 2011. Gigapixel Computational Imaging. In IEEE International Conference on Computational Photography (ICCP). 1--8.
[16]
Geoffroi Côté, Jean-François Lalonde, and Simon Thibault. 2019. Extrapolating from lens design databases using deep learning. Opt. Express 27, 20 (Sep 2019), 28279--28292.
[17]
Geoffroi Côté, Jean-François Lalonde, and Simon Thibault. 2021. Deep learning-enabled framework for automatic lens design starting point generation. Opt. Express 29, 3 (Feb 2021), 3841--3854.
[18]
Paul E. Debevec and Jitendra Malik. 1997. Recovering High Dynamic Range Radiance Maps from Photographs. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., USA, 369--378.
[19]
Edward R Dowski and W Thomas Cathey. 1995. Extended depth of field through wave-front coding. Applied optics 34, 11 (1995), 1859--1866.
[20]
Xiong Dun, Hayato Ikoma, Gordon Wetzstein, Zhanshan Wang, Xinbin Cheng, and Yifan Peng. 2020. Learned rotationally symmetric diffractive achromat for full-spectrum computational imaging. Optica 7, 8 (Aug 2020), 913--922.
[21]
FZ Fang, XD Zhang, A Weckenmann, GX Zhang, and C Evans. 2013. Manufacturing and measurement of freeform optics. CIRP Annals 62, 2 (2013), 823--846.
[22]
Angel Flores, Michael R. Wang, and Jame J. Yang. 2004. Achromatic hybrid refractive-diffractive lens with extended depth of focus. Applied Optics 43, 30 (Oct 2004), 5618--5630.
[23]
Grant R Fowles. 2012. Introduction to modern optics. Courier Dover Publications.
[24]
Qi Guo, Iuri Frosio, Orazio Gallo, Todd Zickler, and Jan Kautz. 2018. Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset. In The European Conference on Computer Vision (ECCV). Springer.
[25]
Harel Haim, Shay Elmalem, Raja Giryes, Alex Bronstein, and Emanuel Marom. 2018. Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask. IEEE Transactions on Computational Imaging 4 (2018), 298--310.
[26]
Samuel W Hasinoff and Kiriakos N Kutulakos. 2011. Light-efficient photography. IEEE Transactions on Pattern Analysis and Machine Intelligence 33, 11 (2011), 2203--2214.
[27]
Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016).
[28]
Roarke Horstmeyer, Richard Y. Chen, Barbara Kappes, and Benjamin Judkewitz. 2017. Convolutional neural networks that teach microscopes how to image. ArXiv abs/1709.07223 (2017).
[29]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2017).
[30]
Francis A Jenkins and Harvey E White. 2018. Fundamentals of optics. Tata McGraw-Hill Education.
[31]
Daniel S. Jeon, Seung-Hwan Baek, Shinyoung Yi, Qiang Fu, Xiong Dun, Wolfgang Heidrich, and Min H. Kim. 2019. Compact Snapshot Hyperspectral Imaging with Diffracted Rotation. ACM Transactions on Graphics (TOG) 38, 4 (2019), 117:1--13.
[32]
Michael Kellman, Emrah Bostan, Michael Chen, and Laura Waller. 2019. Data-Driven Design for Fourier Ptychographic Microscopy. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1--8.
[33]
Salman S. Khan, Adarsh V. R., Vivek Boominathan, Jasper Tan, Ashok Veeraraghavan, and Kaushik Mitra. 2019. Towards Photorealistic Reconstruction of Highly Multiplexed Lensless Images. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
[34]
Craig Kolb, Don Mitchell, and Pat Hanrahan. 1995. A realistic camera model for computer graphics. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques. 317--324.
[35]
Alankar Kotwal, Anat Levin, and Ioannis Gkioulekas. 2020. Interferometric Transmission Probing with Coded Mutual Intensity. 39, 4, Article 74 (July 2020), 16 pages.
[36]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2017. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064 (2017).
[37]
Orest Kupyn, Tetiana Martyniuk, Junru Wu, and Zhangyang Wang. 2019. DeblurGANv2: Deblurring (Orders-of-Magnitude) Faster and Better. In The IEEE International Conference on Computer Vision (ICCV).
[38]
Anat Levin. 2010. Analyzing Depth from Coded Aperture Sets. In Computer Vision - ECCV 2010, Kostas Daniilidis, Petros Maragos, and Nikos Paragios (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 214--227.
[39]
Anat Levin, Rob Fergus, Frédo Durand, and William T. Freeman. 2007. Image and Depth from a Conventional Camera with a Coded Aperture. ACM Transactions on Graphics (TOG) 26, 3 (July 2007), 70--es.
[40]
Anat Levin, Samuel W Hasinoff, Paul Green, Frédo Durand, and William T Freeman. 2009. 4D frequency analysis of computational cameras for depth of field extension. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 97.
[41]
Zhiqiang Liu, Angel Flores, Michael R. Wang, and Jianwen J. Yang. 2007. Diffractive infrared lens with extended depth of focus. Optical Engineering 46, 1 (2007), 1 -- 9.
[42]
Daniel Malacara-HernáUndez and Zacarías Malacara-Hernández. 2016. Handbook of optical design. CRC Press.
[43]
S. Mann and Rosalind W. Picard. 1994. Being 'undigital' with digital cameras: extending dynamic range by combining differently exposed pictures.
[44]
Christopher A Metzler, Hayato Ikoma, Yifan Peng, and Gordon Wetzstein. 2020. Deep optics for single-shot high-dynamic-range imaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1375--1385.
[45]
Mehjabin Monjur, Leonidas Spinoulas, Patrick R Gill, and David G Stork. 2015. Ultra-miniature, computationally efficient diffractive visual-bar-position sensor. In Proc. SensorComm. IEIFSA.
[46]
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 1, 2, 3.
[47]
S.K. Nayar, V. Branzoi, and T. Boult. 2004. Programmable Imaging using a Digital Micromirror Array. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) I, 436--443.
[48]
Elias Nehme, Daniel Freedman, Racheli Gordon, Boris Ferdman, Tomer Michaeli, and Yoav Shechtman. 2019. Dense three dimensional localization microscopy by deep learning.
[49]
Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. 2019. Mitsuba 2: A Retargetable Forward and Inverse Renderer. ACM Transactions on Graphics (TOG) 38, 6 (Dec. 2019).
[50]
Yifan Peng, Qilin Sun, Xiong Dun, Gordon Wetzstein, Wolfgang Heidrich, and Felix Heide. 2019. Learned Large Field-of-View Imaging with Thin-Plate Optics. ACM Transactions on Graphics (TOG) 38, 6, Article 219 (Nov. 2019), 14 pages.
[51]
E. Reinhard and K. Devlin. 2005. Dynamic range reduction inspired by photoreceptor physiology. IEEE Transactions on Visualization and Computer Graphics 11, 1 (2005), 13--24.
[52]
M. Rouf, R. Mantiuk, W. Heidrich, M. Trentacoste, and C. Lau. 2011. Glare Encoding of High Dynamic Range Images. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[53]
Christian J. Schuler, Harold Christopher Burger, Stefan Harmeling, and Bernhard Scholkopf. 2013. A Machine Learning Approach for Non-blind Image Deconvolution. In Proc. Computer Vision and Pattern Recognition.
[54]
Yoav Shechtman, Lucien E Weiss, Adam S. Backer, Maurice Y. Lee, and W E Moerner. 2016. Multicolour localization microscopy by point-spread-function engineering. Nature photonics 10 (2016), 590--594.
[55]
Yichang Shih, Brian Guenter, and Neel Joshi. 2012. Image enhancement using calibrated lens simulations. In European Conference on Computer Vision (ECCV). Springer, 42--56.
[56]
Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37, 4 (2018), 114.
[57]
Warren J. Smith. 2005. Modern lens design. McGraw-Hill.
[58]
David G Stork and Patrick R Gill. 2013. Lensless ultra-miniature CMOS computational imagers and sensors. Proc. SENSORCOMM (2013), 186--190.
[59]
David G Stork and Patrick R Gill. 2014. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements 7, 3 (2014), 4.
[60]
Qilin Sun, Xiong Dun, Yifan Peng, and Wolfgang Heidrich. 2018. Depth and Transient Imaging With Compressive SPAD Array Cameras. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[61]
Qilin Sun, Ethan Tseng, Qiang Fu, Wolfgang Heidrich, and Felix Heide. 2020a. Learning Rank-1 Diffractive Optics for Single-Shot High Dynamic Range Imaging. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[62]
Qilin Sun, Jian Zhang, Xiong Dun, Bernard Ghanem, Yifan Peng, and Wolfgang Heidrich. 2020b. End-to-End Learned, Optically Coded Super-Resolution SPAD Camera. ACM Transactions on Graphics (TOG) 39, 2, Article 9 (March 2020), 14 pages.
[63]
Sara C Tucker, W Thomas Cathey, and Edward R Dowski. 1999. Extended depth of field and aberration control for inexpensive digital microscope systems. Optics Express 4, 11 (1999), 467--474.
[64]
Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar. 2013. Picam:Anultra-thin high performance monolithic camera array. ACM Transactions on Graphics (TOG) 32, 6 (2013), 166.
[65]
Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. 2019a. PhaseCam3D - Learning Phase Masks for Passive Single View Depth Estimation. In IEEE International Conference on Computational Photography (ICCP).
[66]
Y. Wu, V. Boominathan, H. Chen, A. Sankaranarayanan, and A. Veeraraghavan. 2019b. PhaseCam3D â€" Learning Phase Masks for Passive Single View Depth Estimation. In IEEE International Conference on Computational Photography (ICCP). IEEE Computer Society, Los Alamitos, CA, USA, 1--12.
[67]
Y. Wu, F. Li, F. Willomitzer, A. Veeraraghavan, and O. Cossairt. 2020. WISHED: Wavefront imaging sensor with high resolution and depth ranging. In IEEE International Conference on Computational Photography (ICCP). 1--10.
[68]
Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. 2014. Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems. 1790--1798.
[69]
Xiaoyun Yuan, Lu Fang, Qionghai Dai, David J Brady, and Yebin Liu. 2017. Multiscale gigapixel video: A cross resolution image matching and warping approach. In IEEE International Conference on Computational Photography (ICCP). IEEE, 1--9.
[70]
Cheng Zhang, Bailey Miller, Kai Yan, Ioannis Gkioulekas, and Shuang Zhao. 2020. Path-Space Differentiable Rendering. ACM Transactions on Graphics (TOG) 39, 4 (2020), 143:1--143:19.
[71]
Cheng Zhang, Lifan Wu, Changxi Zheng, Ioannis Gkioulekas, Ravi Ramamoorthi, and Shuang Zhao. 2019. A Differential Theory of Radiative Transfer. ACM Transactions on Graphics (TOG) 38, 6 (2019), 227:1--227:16.
[72]
Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Rynson WH Lau, and Ming-Hsuan Yang. 2017. Learning fully convolutional networks for iterative non-blind deconvolution. (2017).
[73]
Xuaner Zhang, Ren Ng, and Qifeng Chen. 2018. Single Image Reflection Separation with PerceptualLosses. In IEEE Conference on Computer Vision and Pattern Recognition.

Cited By

View all
  • (2025)Computational optical imaging: on the convergence of physical and digital layersOptica10.1364/OPTICA.54494312:1(113)Online publication date: 17-Jan-2025
  • (2025)Diffraction modeling between arbitrary non-parallel planes using angular spectrum rearrangementOptica10.1364/OPTICA.54460412:1(39)Online publication date: 10-Jan-2025
  • (2025)Exoplanet Imaging via Differentiable RenderingIEEE Transactions on Computational Imaging10.1109/TCI.2025.352597111(36-51)Online publication date: 2025
  • Show More Cited By

Index Terms

  1. End-to-end complex lens design with differentiate ray tracing

      Recommendations

      Comments

      Please enable JavaScript to view thecomments powered by Disqus.

      Information & Contributors

      Information

      Published In

      cover image ACM Transactions on Graphics
      ACM Transactions on Graphics  Volume 40, Issue 4
      August 2021
      2170 pages
      ISSN:0730-0301
      EISSN:1557-7368
      DOI:10.1145/3450626
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 19 July 2021
      Published in TOG Volume 40, Issue 4

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. complex lens
      2. differentiable
      3. end-to-end
      4. raytracing

      Qualifiers

      • Research-article

      Funding Sources

      • KAUST

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)337
      • Downloads (Last 6 weeks)49
      Reflects downloads up to 17 Jan 2025

      Other Metrics

      Citations

      Cited By

      View all
      • (2025)Computational optical imaging: on the convergence of physical and digital layersOptica10.1364/OPTICA.54494312:1(113)Online publication date: 17-Jan-2025
      • (2025)Diffraction modeling between arbitrary non-parallel planes using angular spectrum rearrangementOptica10.1364/OPTICA.54460412:1(39)Online publication date: 10-Jan-2025
      • (2025)Exoplanet Imaging via Differentiable RenderingIEEE Transactions on Computational Imaging10.1109/TCI.2025.352597111(36-51)Online publication date: 2025
      • (2025)Monocular 3D Micro-PIV System Using Computational ImagingIEEE Photonics Journal10.1109/JPHOT.2024.352016317:1(1-9)Online publication date: Feb-2025
      • (2025)Representing domain-mixing optical degradation for real-world Computational Aberration Correction via vector quantizationOptics & Laser Technology10.1016/j.optlastec.2024.112080183(112080)Online publication date: May-2025
      • (2025)End-to-end aberration correction network for enhancing miniature microscope resolutionOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108558184(108558)Online publication date: Jan-2025
      • (2024)Compact freeform near-eye display system design enabled by optical-digital joint optimizationFrontiers in Physics10.3389/fphy.2024.144012912Online publication date: 19-Aug-2024
      • (2024)Differentiable design of a double-freeform lens with multi-level radial basis functions for extended source irradiance tailoringOptica10.1364/OPTICA.52048511:5(653)Online publication date: 6-May-2024
      • (2024)End-to-end automatic lens design with a differentiable diffraction modelOptics Express10.1364/OE.54059032:25(44328)Online publication date: 20-Nov-2024
      • (2024)Configurable multiple virtual lenses conjugated with singlet physical lens for achromatic extended depth-of-field imagingOptics Express10.1364/OE.53867032:23(40427)Online publication date: 22-Oct-2024
      • Show More Cited By

      View Options

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media