[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article
Open access

Compact snapshot hyperspectral imaging with diffracted rotation

Published: 12 July 2019 Publication History

Abstract

Traditional snapshot hyperspectral imaging systems include various optical elements: a dispersive optical element (prism), a coded aperture, several relay lenses, and an imaging lens, resulting in an impractically large form factor. We seek an alternative, minimal form factor of snapshot spectral imaging based on recent advances in diffractive optical technology. We thereupon present a compact, diffraction-based snapshot hyperspectral imaging method, using only a novel diffractive optical element (DOE) in front of a conventional, bare image sensor. Our diffractive imaging method replaces the common optical elements in hyperspectral imaging with a single optical element. To this end, we tackle two main challenges: First, the traditional diffractive lenses are not suitable for color imaging under incoherent illumination due to severe chromatic aberration because the size of the point spread function (PSF) changes depending on the wavelength. By leveraging this wavelength-dependent property alternatively for hyperspectral imaging, we introduce a novel DOE design that generates an anisotropic shape of the spectrally-varying PSF. The PSF size remains virtually unchanged, but instead the PSF shape rotates as the wavelength of light changes. Second, since there is no dispersive element and no coded aperture mask, the ill-posedness of spectral reconstruction increases significantly. Thus, we propose an end-to-end network solution based on the unrolled architecture of an optimization procedure with a spatial-spectral prior, specifically designed for deconvolution-based spectral reconstruction. Finally, we demonstrate hyperspectral imaging with a fabricated DOE attached to a conventional DSLR sensor. Results show that our method compares well with other state-of-the-art hyperspectral imaging methods in terms of spectral accuracy and spatial resolution, while our compact, diffraction-based spectral imaging method uses only a single optical element on a bare image sensor.

Supplementary Material

ZIP File (a117-jeon.zip)
Supplemental material

References

[1]
M. Abadi, P. Barham, J. Chen, Z. Chen, A. Davis, J. Dean, M. Devin, S. Ghemawat, G. Irving, M. Isard, M. Kudlur, J. Levenberg, R. Monga, S. Moore, D. G. Murray, B. Steiner, P. Tucker, V. Vasudevan, P. Warden, M. Wicke, Y. Yu, and X. Zheng. 2016.
[2]
TensorFlow: A System for Large-scale Machine Learning. In Proc. USENIX Conf. Operating Systems Design and Implementation (OSDI'16). 265--283.
[3]
Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1--9.
[4]
Nicholas Antipa, Sylvia Necula, Ren Ng, and Laura Waller. 2016. Single-shot diffuser-encoded light field imaging. In Proc. IEEE Int. Conf. Computational Photography (ICCP 2016). IEEE, 1--11.
[5]
Boaz Arad and Ohad Ben-Shahar. 2016. Sparse Recovery of Hyperspectral Signal from Natural RGB Images. In Proc. European Conference on Computer Vision (ECCV 2016). Springer, 19--34.
[6]
M Salman Asif, Ali Ayremlou, Aswin Sankaranarayanan, Ashok Veeraraghavan, and Richard G Baraniuk. 2017. FlatCam: Thin, lensless cameras using coded aperture and computation. IEEE Transactions on Computational Imaging (TCI) 3, 3 (2017), 384--397.
[7]
Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H. Kim. 2017. Compact Single-Shot Hyperspectral Imaging Using a Prism. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017).
[8]
Jose M. Bioucas-Dias and Mario A. T. Figueiredo. 2007. A new TwIST: two-step iterative shrinkage/thresholding for image restoration. IEEE Trans. Image Processing (TIP) 16, 12 (2007), 2992--3004.
[9]
Nicola Brusco, S Capeleto, M Fedel, Anna Paviotti, Luca Poletto, Guido Maria Cortelazzo, and G Tondello. 2006. A system for 3D modeling frescoed historical buildings with multispectral texture information. Machine Vision and Applications 17, 6 (2006), 373--393.
[10]
Ayan Chakrabarti and Todd Zickler. 2011. Statistics of real-world hyperspectral images. In Proc. Conference on Computer Vision and Pattern Recognition (CVPR 2011). IEEE, 193--200.
[11]
Inchang Choi, Daniel S. Jeon, Giljoo Nam, Diego Gutierrez, and Min H. Kim. 2017. High-Quality Hyperspectral Reconstruction Using a Spectral Prior. ACM Transactions on Graphics (Proc. SIGGRAPH Asia 2017) 36, 6 (2017).
[12]
B. Choudhury, R. Swanson, F. Heide, G. Wetzstein, and W. Heidrich. 2017. Consensus Convolutional Sparse Coding. In Proc. International Conference on Computer Vision (ICCV 2017). 4290--4298.
[13]
Weisheng Dong, Peiyao Wang, Wotao Yin, and Guangming Shi. 2018. Denoising Prior Driven Deep Neural Network for Image Restoration. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) (2018), 1--1.
[14]
M E Gehm, R John, D J Brady, R M Willett, and T J Schulz. 2007. Single-shot compressive spectral imaging with a dual-disperser architecture. OSA OE 15, 21 (2007), 14013--27.
[15]
Adam Greengard, Yoav Y Schechner, and Rafael Piestun. 2006. Depth from diffracted rotation. Optics letters 31, 2 (2006), 181--183.
[16]
Ralf Habel, Michael Kudenov, and Michael Wimmer. 2012. Practical spectral photography. In Computer graphics forum, Vol. 31. Wiley Online Library, 449--458.
[17]
Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016), 33543.
[18]
Daniel S Jeon, Inchang Choi, and Min H Kim. 2016. Multisampling Compressive Video Spectroscopy. Computer Graphics Forum 35, 2 (2016), 467--477.
[19]
William R Johnson, Daniel W Wilson, Wolfgang Fink, Mark Humayun, and Greg Bearman. 2007. Snapshot hyperspectral imaging in ophthalmology. Journal of biomedical optics 12, 1 (2007), 014036--014036.
[20]
Min H Kim. 2013. 3D Graphics Techniques for Capturing and Inspecting Hyperspectral Appearance. In Ubiquitous Virtual Reality (ISUVR), 2013 Int. Symp. on. IEEE, 15--18.
[21]
Min H Kim, Todd Alan Harvey, David S Kittle, Holly Rushmeier, Julie Dorsey, Richard O Prum, and David J Brady. 2012a. 3D imaging spectroscopy for measuring hyperspectral patterns on solid objects. ACM Transactions on Graphics 31, 4 (2012), 38.
[22]
Min H. Kim and Holly Rushmeier. 2011. Radiometric Characterization of Spectral Imaging for Textual Pigment Identification. In Proc. International Symposium on Virtual Reality, Archaeology and Cultural Heritage (VAST 2011). Eurographics, Tuscany, Italy, 57--64.
[23]
Min H Kim, Holly Rushmeier, John ffrench, and Irma Passeri. 2012b. Developing Open-Source Software for Art Conservators. In VAST12: The 13th International Symposium on Virtual Reality, Archaeology and Intelligent Cultural Heritage. Eurographics Association, Brighton, England, 97--104.
[24]
Min H Kim, Holly Rushmeier, John ffrench, Irma Passeri, and David Tidmarsh. 2014. Hyper3D: 3D Graphics Software for Examining Cultural Artifacts. ACM Journal on Computing and Cultural Heritage 7, 3 (2014), 1:1--19.
[25]
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. In The International Conference on Learning Representations (ICLR).
[26]
David Kittle, Kerkil Choi, Ashwin Wagadarikar, and David J Brady. 2010. Multiframe image estimation for coded aperture snapshot spectral imagers. Applied Optics 49, 36 (2010), 6824--6833.
[27]
Fred A Kruse, AB Lefkoff, JW Boardman, KB Heidebrecht, AT Shapiro, PJ Barloon, and AFH Goetz. 1993. The spectral image processing system (SIPS)---interactive visualization and analysis of imaging spectrometer data. Remote sensing of environment 44, 2--3 (1993), 145--163.
[28]
Haebom Lee and Min H. Kim. 2014. Building a Two-Way Hyperspectral Imaging System with Liquid Crystal Tunable Filters. In Proc. Int. Conf. Image and Signal Processing (ICISP 2014) (Lecture Notes in Computer Science (LNCS)), Vol. 8509. Springer, Normandy, France, 26--34.
[29]
Chengbo Li, Wotao Yin, and Yin Zhang. 2009. User's guide for TVAL3: TV minimization by augmented lagrangian and alternating direction algorithms. CAAM report 20, 46--47 (2009), 4.
[30]
Xing Lin, Yebin Liu, Jiamin Wu, and Qionghai Dai. 2014. Spatial-spectral encoded compressive hyperspectral imaging. ACM Transactions on Graphics 33, 6 (2014), 233.
[31]
Giljoo Nam and Min H. Kim. 2014. Multispectral Photometric Stereo for Acquiring High-Fidelity Surface Normals. IEEE Computer Graphics and Applications 34, 6 (2014), 57--68.
[32]
Takayuki Okamoto, Akinori Takahashi, and Ichirou Yamaguchi. 1993. Simultaneous Acquisition of Spectral and Spatial Intensity Distribution. Appl. Spectrosc. 47, 8 (Aug 1993), 1198--1202.
[33]
Donald C. O'Shea, Thomas J. Suleski, Alan D. Kathman, and Dennis W. Prather. 2003. Diffractive Optics: Design, Fabrication, and Test. SPIE Press.
[34]
Yifan Peng, Xiong Dun, Qilin Sun, Felix Heide, and Wolfgang Heidrich. 2018. Focal sweep imaging with multi-focal diffractive optics. In IEEE Proc. Int. Conf. Computational Photography (ICCP). IEEE, 1--8.
[35]
Yifan Peng, Qiang Fu, Hadi Amata, Shuochen Su, Felix Heide, and Wolfgang Heidrich. 2015. Computational imaging using lightweight diffractive-refractive optics. Optics Express 23, 24 (2015), 31393--31407.
[36]
Yifan Peng, Qiang Fu, Felix Heide, and Wolfgang Heidrich. 2016. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Transactions on Graphics (Proc. SIGGRAPH 2016) (2016), 1--11.
[37]
Wallace M Porter and Harry T Enmark. 1987. A system overview of the airborne visible/infrared imaging spectrometer (AVIRIS). In 31st Annual Technical Symposium. International Society for Optics and Photonics, 22--31.
[38]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[39]
K. Simonyan and A. Zisserman. 2015. Very Deep Convolutional Networks for Large-Scale Image Recognition. In Proc. Int. Conf. Learning Representation (ICLR).
[40]
Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (Proc. SIGGRAPH 2018) 37, 4 (2018), 114.
[41]
Gary J Swanson. 1991. Binary optics technology: theoretical limits on the diffraction efficiency of multilevel diffractive optical elements. Technical Report. MASSACHUSETTS INST OF TECH LEXINGTON LINCOLN LAB.
[42]
Kazuyuki Tajima, Takeshi Shimano, Yusuke Nakamura, Mayu Sao, and Taku Hoshizawa. 2017. Lensless light-field imaging with multi-phased Fresnel zone aperture. In Proc. IEEE Int. Conf. Computational Photography (ICCP). IEEE, 1--7.
[43]
Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. 2008. Single disperser design for coded aperture snapshot spectral imaging. Applied optics 47, 10 (2008), B44--B51.
[44]
Lizhi Wang, Chen Sun, Ying Fu, Min H. Kim, and Huang Hua. 2019. Hyperspectral Image Reconstruction Using a Deep Spatial-Spectral Prior. In Proc. of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2019). TBD.
[45]
Peng Wang and Rajesh Menon. 2015. Ultra-high-sensitivity color imaging via a transparent diffractive-filter array and computational optics. Optica 2, 11 (Nov 2015), 933--939.
[46]
Peng Wang and Rajesh Menon. 2018. Computational multispectral video imaging. J. Opt. Soc. Am. A 35, 1 (Jan 2018), 189--199.
[47]
Jian Zhang and Bernard Ghanem. 2018. ISTA-Net: Interpretable Optimization-Inspired Deep Network for Image Compressive Sensing. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR 2018). 1828--1837.
[48]
Kai Zhang, Wangmeng Zuo, Shuhang Gu, and Lei Zhang. 2017. Learning deep CNN denoiser prior for image restoration. In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2017), Vol. 2.

Cited By

View all
  • (2025)Computational optical imaging: on the convergence of physical and digital layersOptica10.1364/OPTICA.54494312:1(113)Online publication date: 17-Jan-2025
  • (2025)Compressive spectral imaging with color-coded illuminationOptics & Laser Technology10.1016/j.optlastec.2025.112446184(112446)Online publication date: Jun-2025
  • (2025)Exploring the functional characteristics of diffractive optical Element: A comprehensive reviewOptics & Laser Technology10.1016/j.optlastec.2024.112383183(112383)Online publication date: May-2025
  • Show More Cited By

Index Terms

  1. Compact snapshot hyperspectral imaging with diffracted rotation

    Recommendations

    Comments

    Please enable JavaScript to view thecomments powered by Disqus.

    Information & Contributors

    Information

    Published In

    cover image ACM Transactions on Graphics
    ACM Transactions on Graphics  Volume 38, Issue 4
    August 2019
    1480 pages
    ISSN:0730-0301
    EISSN:1557-7368
    DOI:10.1145/3306346
    Issue’s Table of Contents
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 12 July 2019
    Published in TOG Volume 38, Issue 4

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. diffraction
    2. hyperspectral imaging

    Qualifiers

    • Research-article

    Funding Sources

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)788
    • Downloads (Last 6 weeks)105
    Reflects downloads up to 14 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2025)Computational optical imaging: on the convergence of physical and digital layersOptica10.1364/OPTICA.54494312:1(113)Online publication date: 17-Jan-2025
    • (2025)Compressive spectral imaging with color-coded illuminationOptics & Laser Technology10.1016/j.optlastec.2025.112446184(112446)Online publication date: Jun-2025
    • (2025)Exploring the functional characteristics of diffractive optical Element: A comprehensive reviewOptics & Laser Technology10.1016/j.optlastec.2024.112383183(112383)Online publication date: May-2025
    • (2025)PSF-engineered snapshot full-Stokes polarizing-spectral-imaging via metasurface with reciprocally encoded anisotropic detour phaseOptics & Laser Technology10.1016/j.optlastec.2024.111645181(111645)Online publication date: Feb-2025
    • (2025)Two-stage framework for reconstructing spectral images from diffraction-blurred imagesOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108789186(108789)Online publication date: Mar-2025
    • (2024)计算光谱成像:光场编码与算法解码(特邀)Laser & Optoelectronics Progress10.3788/LOP24139761:16(1611003)Online publication date: 2024
    • (2024)快照式衍射计算光谱成像混合误差分析与抑制Acta Optica Sinica10.3788/AOS24088744:19(1911003)Online publication date: 2024
    • (2024)Hybrid Space Calibrated 3D Network of Diffractive Hyperspectral Optical Imaging SensorSensors10.3390/s2421690324:21(6903)Online publication date: 28-Oct-2024
    • (2024)Snapshot spectral imaging: from spatial-spectral mapping to metasurface-based imagingNanophotonics10.1515/nanoph-2023-086713:8(1303-1330)Online publication date: 22-Mar-2024
    • (2024)Direct object detection with snapshot multispectral compressed imaging in a short-wave infrared bandOptics Letters10.1364/OL.51728449:8(1941)Online publication date: 4-Apr-2024
    • Show More Cited By

    View Options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Login options

    Full Access

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media