More Web Proxy on the site http://driver.im/

research-article

Open access

Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging

Authors:

Clara Callenberg,

Matthias B. HullinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 40, Issue 4

Article No.: 61, Pages 1 - 12

https://doi.org/10.1145/3450626.3459824

Published: 19 July 2021 Publication History

Abstract

Time-correlated imaging is an emerging sensing modality that has been shown to enable promising application scenarios, including lidar ranging, fluorescence lifetime imaging, and even non-line-of-sight sensing. A leading technology for obtaining time-correlated light measurements are single-photon avalanche diodes (SPADs), which are extremely sensitive and capable of temporal resolution on the order of tens of picoseconds. However, the rare and expensive optical setups used by researchers have so far prohibited these novel sensing techniques from entering the mass market. Fortunately, SPADs also exist in a radically cheaper and more power-efficient version that has been widely deployed as proximity sensors in mobile devices for almost a decade. These commodity SPAD sensors can be obtained at a mere few cents per detector pixel. However, their inferior data quality and severe technical drawbacks compared to their high-end counterparts necessitate the use of additional optics and suitable processing algorithms. In this paper, we adopt an existing evaluation platform for commodity SPAD sensors, and modify it to unlock time-of-flight (ToF) histogramming and hence computational imaging. Based on this platform, we develop and demonstrate a family of hardware/software systems that, for the first time, implement applications that had so far been limited to significantly more advanced, higher-priced setups: direct ToF depth imaging, non-line-of-sight object tracking, and material classification.

Supplementary Material

VTT File (3450626.3459824.vtt)

Download
19.07 KB

ZIP File (a61-callenberg.zip)

a61-callenberg.zip

Download
22.47 MB

MP4 File (a61-callenberg.mp4)

Download
143.20 MB

MP4 File (3450626.3459824.mp4)

Presentation.

Download
333.68 MB

References

[1]

Nils Abramson. 1978. Light-in-flight recording by holography. Optics letters 3, 4 (1978), 121--123.

[2]

Yoann Altmann, Stephen McLaughlin, Miles J Padgett, Vivek K Goyal, Alfred O Hero, and Daniele Faccio. 2018. Quantum-inspired computational imaging. Science 361, 6403 (2018).

[3]

Victor Arellano, Diego Gutierrez, and Adrian Jarabo. 2017. Fast back-projection for non-line of sight reconstruction. Opt. Express 25, 10 (2017), 11574--11583.

[4]

David Patrick Baxter. 2015. Application using a single photon avalanche diode (SPAD). (June 16 2015). US Patent 9,058,081.

[5]

Katherine L Bouman, Vickie Ye, Adam B Yedidia, Frédo Durand, Gregory W Wornell, Antonio Torralba, and William T Freeman. 2017. Turning corners into cameras: Principles and methods. In IEEE International Conference on Computer Vision (ICCV). 2289--2297.

[6]

Samuel Burri, Harald Homulle, Claudio Bruschini, and Edoardo Charbon. 2016. LinoSPAD: a time-resolved 256x1 CMOS SPAD line sensor system featuring 64 FPGA-based TDC channels running at up to 8.5 giga-events per second. Proc. SPIE 9899 (2016), 98990D-10.

[7]

Mauro Buttafava, Jessica Zeman, Alberto Tosi, Kevin Eliceiri, and Andreas Velten. 2015. Non-line-of-sight imaging using a time-gated single photon avalanche diode. Optics express 23, 16 (2015), 20997--21011.

[8]

Richard H Byrd, Peihuang Lu, Jorge Nocedal, and Ciyou Zhu. 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on scientific computing 16, 5 (1995), 1190--1208.

[9]

C. Callenberg, A. Lyons, D. den Brok, A. Fatima, A. Turpin, V. Zickus, L. Machesky, J. Whitelaw, D. Faccio, and M. B. Hullin. 2021. Super-resolution time-resolved imaging using computational sensor fusion. Scientific Reports 11, 1 (Jan. 2021), 1689.

[10]

Barbara Caputo, Eric Hayman, and P Mallikarjuna. 2005. Class-specific material categorisation. In Tenth IEEE International Conference on Computer Vision (ICCV'05) Volume 1, Vol. 2. IEEE, 1597--1604.

Digital Library

[11]

Edoardo Charbon. 2008. Towards large scale CMOS single-photon detector arrays for lab-on-chip applications. Journal of Physics D: Applied Physics 41, 9 (2008), 094010.

[12]

Wenzheng Chen, Simon Daneau, Fahim Mannan, and Felix Heide. 2019. Steady-state non-line-of-sight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6790--6799.

[13]

Wenzheng Chen, Fangyin Wei, Kiriakos N Kutulakos, Szymon Rusinkiewicz, and Felix Heide. 2020. Learned feature embeddings for non-line-of-sight imaging and recognition. ACM Trans. Graph. 39, 6 (2020), 1--18.

Digital Library

[14]

Liang Gao, Jinyang Liang, Chiye Li, and Lihong V Wang. 2014. Single-shot compressed ultrafast photography at one hundred billion frames per second. Nature 516, 7529 (2014), 74.

[15]

Genevieve Gariepy, Nikola Krstajić, Robert Henderson, Chunyong Li, Robert R Thomson, Gerald S Buller, Barmak Heshmat, Ramesh Raskar, Jonathan Leach, and Daniele Faccio. 2015. Single-photon sensitive light-in-fight imaging. Nature communications 6 (2015), 6021.

[16]

Javier Grau Chopite, Matthias B. Hullin, Michael Wand, and Julian Iseringhausen. 2020. Deep Non-Line-of-Sight Reconstruction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[17]

Felix Heide, Matthias B. Hullin, James Gregson, and Wolfgang Heidrich. 2013. Low-Budget Transient Imaging using Photonic Mixer Devices. ACM Trans. Graph. 32, 4 (2013), 45:1--45:10.

Digital Library

[18]

Felix Heide, Matthew O'Toole, Kai Zang, David B Lindell, Steven Diamond, and Gordon Wetzstein. 2019. Non-line-of-sight imaging with partial occluders and surface normals. ACM Trans. Graph. 38, 3 (2019), 1--10.

Digital Library

[19]

Felix Heide, Lei Xiao, Wolfgang Heidrich, and Matthias B Hullin. 2014. Diffuse mirrors: 3D reconstruction from diffuse indirect illumination using inexpensive time-of-flight sensors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3222--3229.

Digital Library

[20]

Robert K. Henderson, Nick Johnston, Haochang Chen, David Day Uei Li, Graham Hungerford, Richard Hirsch, David McLoskey, Philip Yip, and David J.S. Birch. 2018. A 192X128 Time Correlated Single Photon Counting Imager in 40nm CMOS Technology. ESSCIRC 2018 - IEEE 44th European Solid State Circuits Conference (2018), 54--57.

[21]

Quercus Hernandez, Diego Gutierrez, and Adrian Jarabo. 2017. A Computational Model of a Single-Photon Avalanche Diode Sensor for Transient Imaging. (2017). arXiv:physics.ins-det/1703.02635

[22]

Atul Ingle, Andreas Velten, and Mohit Gupta. 2019. High Flux Passive Imaging With Single-Photon Sensors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).

[23]

Julian Iseringhausen and Matthias Hullin. 2020. Non-line-of-sight reconstruction using efficient transient rendering. ACM Trans. Graph. 39, 1 (2020), 1--14.

Digital Library

[24]

Adrian Jarabo, Belen Masia, Julio Marco, and Diego Gutierrez. 2017. Recent advances in transient imaging: A computer graphics and vision perspective. Visual Informatics 1, 1 (2017), 65--79.

[25]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. 2013. Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. 32, 6 (2013).

Digital Library

[26]

Jonathan Klein, Christoph Peters, Jaime Martín, Martin Laurenzis, and Matthias B Hullin. 2016. Tracking objects outside the line of sight using 2D intensity images. Scientific reports 6 (2016), 32491.

[27]

Dilip Krishnan and Rob Fergus. 2009. Fast Image Deconvolution using Hyper-Laplacian Priors. In Advances in Neural Information Processing Systems 22, Y. Bengio, D. Schuurmans, J. D. Lafferty, C. K. I. Williams, and A. Culotta (Eds.). Curran Associates, Inc., 1033--1041.

[28]

Jinyang Liang, Cheng Ma, Liren Zhu, Yujia Chen, Liang Gao, and Lihong V Wang. 2017. Single-shot real-time video recording of a photonic Mach cone induced by a scattered light pulse. Science advances 3, 1 (2017), e1601814.

[29]

David B Lindell, Matthew O'Toole, and Gordon Wetzstein. 2018. Single-photon 3D imaging with deep sensor fusion. ACM Trans. Graph. 37, 4 (2018), 113.

Digital Library

[30]

David B. Lindell, Gordon Wetzstein, and Matthew O'Toole. 2019. Wave-based non-line-of-sight imaging using fast f-k migration. ACM Trans. Graph. 38, 4 (2019), 116.

Digital Library

[31]

Ce Liu, Lavanya Sharan, Edward H Adelson, and Ruth Rosenholtz. 2010. Exploring features in a bayesian framework for material recognition. In 2010 ieee computer society conference on computer vision and pattern recognition. IEEE, 239--246.

[32]

Xiaochun Liu, Ibón Guillén, Marco La Manna, Ji Hyun Nam, Syed Azer Reza, Toan Huu Le, Adrian Jarabo, Diego Gutierrez, and Andreas Velten. 2019. Non-line-of-sight imaging using phasor-field virtual wave optics. Nature (2019), 1--4.

[33]

Andreas Meuleman, Seung-Hwan Baek, Felix Heide, and Min H Kim. 2020. Single-Shot Monocular RGB-D Imaging Using Uneven Double Refraction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2465--2474.

[34]

Nikhil Naik, Shuang Zhao, Andreas Velten, Ramesh Raskar, and Kavita Bala. 2011. Single View Reflectance Capture Using Multiplexed Scattering and Time-of-flight Imaging. ACM Trans. Graph. 30, 6 (2011), 171:1--171:10.

Digital Library

[35]

Desmond O'Connor. 2012. Time-correlated single photon counting. Academic Press.

[36]

Matthew O'Toole, Felix Heide, Lei Xiao, Matthias B. Hullin, Wolfgang Heidrich, and Kiriakos N. Kutulakos. 2014. Temporal Frequency Probing for 5D Transient Analysis of Global Light Transport. ACM Trans. Graph. 33, 4 (Aug. 2014).

Digital Library

[37]

Matthew O'Toole, David B. Lindell, and Gordon Wetzstein. 2018. Confocal Non-line-of-sight imaging based on the light cone transform. Nature (2018), 338--341. Issue 555.

[38]

Christoph Peters, Jonathan Klein, Matthias B. Hullin, and Reinhard Klein. 2015. Solving Trigonometric Moment Problems for Fast Transient Imaging. ACM Trans. Graph. 34, 6 (Nov. 2015).

Digital Library

[39]

Corneliu Rablau. 2019. LIDAR-A new (self-driving) vehicle for introducing optics to broader engineering and non-engineering audiences. In Education and Training in Optics and Photonics. Optical Society of America, 11143_138.

[40]

Sabbir Rangwala. 2020. The iPhone 12 - LiDAR At Your Fingertips. Forbes (12 November 2020). https://www.forbes.com/sites/sabbirrangwala/2020/11/12/the-iphone-12lidar-at-your-fingertips/

[41]

Justin A. Richardson, Lindsay A. Grant, and Robert K. Henderson. 2009. Low dark count single-photon avalanche diode structure compatible with standard nanometer scale CMOS technology. IEEE Photonics Technology Letters 21, 14 (2009), 1020--1022.

[42]

N. Scheiner, F. Kraus, F. Wei, B. Phan, F. Mannan, N. Appenrodt, W. Ritter, J. Dickmann, K. Dietmayer, B. Sick, et al. 2020. Seeing Around Street Corners: Non-Line-of-Sight Detection and Tracking In-the-Wild Using Doppler Radar. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2068--2077.

[43]

Brent Schwarz. 2010. LIDAR: Mapping the world in 3D. Nat. Photonics 4, 7 (2010), 1749--4885.

[44]

Nikolai Smolyanskiy, Alexey Kamenev, and Stan Birchfield. 2018. On the importance of stereo for accurate depth estimation: An efficient semi-supervised deep neural network approach. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 1007--1015.

[45]

STMicroelectronics. 2019. STMicroelectronics Ships 1 Billionth Time-of-Flight Module. (26 November 2019). https://www.st.com/content/st_com/en/about/media-center/press-item.html/t4210.html Press release.

[46]

Shuochen Su, Felix Heide, Robin Swanson, Jonathan Klein, Clara Callenberg, Matthias Hullin, and Wolfgang Heidrich. 2016. Material Classification Using Raw Time-Of-Flight Measurements. In Proc. IEEE CVPR.

[47]

Kenichiro Tanaka, Yasuhiro Mukaigawa, Takuya Funatomi, Hiroyuki Kubo, Yasuyuki Matsushita, and Yasushi Yagi. 2017. Material classification using frequency-and depth-dependent time-of-flight distortion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 79--88.

[48]

Chia-Yin Tsai, Aswin C Sankaranarayanan, and Ioannis Gkioulekas. 2019. Beyond Volumetric Albedo-A Surface Optimization Framework for Non-Line-Of-Sight Imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1545--1555.

[49]

Manik Varma and Andrew Zisserman. 2008. A statistical approach to material classification using image patch exemplars. IEEE transactions on pattern analysis and machine intelligence 31, 11 (2008), 2032--2047.

[50]

Andreas Velten, Thomas Willwacher, Otkrist Gupta, Ashok Veeraraghavan, Moungi G. Bawendi, and Ramesh Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature Communications 3 (2012), 745.

[51]

Andreas Velten, Di Wu, Adrian Jarabo, Belen Masia, Christopher Barsi, Chinmaya Joshi, Everett Lawson, Moungi Bawendi, Diego Gutierrez, and Ramesh Raskar. 2013. Femto-photography: capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4 (2013), 44.

Digital Library

[52]

Junko Yoshida. 2018. ST & Apple, through Thick and Thin. EETimes (22 August 2018). https://www.eetimes.com/st-apple-through-thick-and-thin/

[53]

Franco Zappa, Simone Tisa, Alberto Tosi, and Sergio Cova. 2007. Principles and features of single-photon avalanche diode arrays. Sensors and Actuators A: Physical 140, 1 (2007), 103--112.

[54]

Vytautas Zickus, Ming-Lo Wu, Kazuhiro Morimoto, Valentin Kapitany, Areeba Fatima, Alex Turpin, Robert Insall, Jamie Whitelaw, Laura Machesky, Claudio Bruschini, et al. 2020. Fluorescence lifetime imaging with a megapixel SPAD camera and neural network lifetime estimation. Scientific Reports 10, 1 (2020), 1--10.

Cited By

Scholes SLeach J(2024)Path Tracing-Inspired Modeling of Non-Line-of-Sight SPAD DataSensors10.3390/s2420652224:20(6522)Online publication date: 10-Oct-2024
https://doi.org/10.3390/s24206522
Zang ZLi D(2024)Object Classification through Heterogeneous Fog with a Fast Data-driven Algorithm Using a Low-Cost Single-Photon Avalanche Diode ArrayOptics Express10.1364/OE.527244Online publication date: 26-Jul-2024
https://doi.org/10.1364/OE.527244
Folden JIngle AKoppal S(2024)FoveaSPAD: Exploiting Depth Priors for Adaptive and Efficient Single-Photon 3D ImagingIEEE Transactions on Computational Imaging10.1109/TCI.2024.350336010(1722-1735)Online publication date: 2024
https://doi.org/10.1109/TCI.2024.3503360
Show More Cited By

Index Terms

Low-cost SPAD sensing for non-line-of-sight tracking, material classification and depth imaging
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Computational photography
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks
2. Hardware
  1. Communication hardware, interfaces and storage
    1. Sensor applications and deployments

Recommendations

Confocal non-line-of-sight imaging
SIGGRAPH '18: ACM SIGGRAPH 2018 Talks

Non-line-of-sight (NLOS) imaging aims at recovering the shape of objects hidden outside the direct line of sight of a camera. In this work, we report on a new approach for acquiring time-resolved measurements that are suitable for NLOS imaging. The ...
Efficient Non-Line-of-Sight Imaging from Transient Sinograms
Computer Vision – ECCV 2020
Abstract
Non-line-of-sight (NLOS) imaging techniques use light that diffusely reflects off of visible surfaces (e.g., walls) to see around corners. One approach involves using pulsed lasers and ultrafast sensors to measure the travel time of multiply ... $^{}$ $^{}$ $^{}$ $^{}$
Computational time-resolved imaging, single-photon sensing, and non-line-of-sight imaging
SIGGRAPH '20: ACM SIGGRAPH 2020 Courses

Emerging detector technologies are capable of ultrafast capture of single photons, enabling imaging at the speed of light. Not only can these detectors be used for imaging at essentially trillion frame-per-second rates, but combining them with ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 40, Issue 4

August 2021

2170 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/3450626

Editor:
Sylvain Paris
Adobe Inc.

Issue’s Table of Contents

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 19 July 2021

Published in TOG Volume 40, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

27
Total Citations
View Citations
1,780
Total Downloads

Downloads (Last 12 months)459
Downloads (Last 6 weeks)76

Reflects downloads up to 21 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Scholes SLeach J(2024)Path Tracing-Inspired Modeling of Non-Line-of-Sight SPAD DataSensors10.3390/s2420652224:20(6522)Online publication date: 10-Oct-2024
https://doi.org/10.3390/s24206522
Zang ZLi D(2024)Object Classification through Heterogeneous Fog with a Fast Data-driven Algorithm Using a Low-Cost Single-Photon Avalanche Diode ArrayOptics Express10.1364/OE.527244Online publication date: 26-Jul-2024
https://doi.org/10.1364/OE.527244
Folden JIngle AKoppal S(2024)FoveaSPAD: Exploiting Depth Priors for Adaptive and Efficient Single-Photon 3D ImagingIEEE Transactions on Computational Imaging10.1109/TCI.2024.350336010(1722-1735)Online publication date: 2024
https://doi.org/10.1109/TCI.2024.3503360
Berta RDabbous ALazzaroni LPau DBellotti F(2024)Developing a TinyML Image Classifier in an HourIEEE Open Journal of the Industrial Electronics Society10.1109/OJIES.2024.34519595(946-960)Online publication date: 2024
https://doi.org/10.1109/OJIES.2024.3451959
Zhu SWang CZhang PLam E(2024)Harnessing Noise for Materials Differentiation in Computational Neuromorphic Imaging2024 9th Optoelectronics Global Conference (OGC)10.1109/OGC62429.2024.10738780(47-50)Online publication date: 10-Sep-2024
https://doi.org/10.1109/OGC62429.2024.10738780
Sifferman CSun WGupta MGleicher M(2024)Using a Distance Sensor to Detect Deviations in a Planar SurfaceIEEE Robotics and Automation Letters10.1109/LRA.2024.34456659:10(8515-8522)Online publication date: Oct-2024
https://doi.org/10.1109/LRA.2024.3445665
Kitichotkul RRapp JGoyal V(2024)The Role of Detection Times in Reflectivity Estimation With Single-Photon LidarIEEE Journal of Selected Topics in Quantum Electronics10.1109/JSTQE.2023.333383430:1: Single-Photon Technologies(1-14)Online publication date: Jan-2024
https://doi.org/10.1109/JSTQE.2023.3333834
Zheng XLi TDing KLuo Y(2024)Non-Line-of-Sight Target Tracking With a Single Time Multiplexed ChannelIEEE Photonics Journal10.1109/JPHOT.2024.347107016:6(1-6)Online publication date: Dec-2024
https://doi.org/10.1109/JPHOT.2024.3471070
Klinghoffer TXiang XSomasundaram SFan YRichardt CRaskar RRanjan R(2024)PlatoNeRF: 3D Reconstruction in Plato's Cave via Single-View Two-Bounce Lidar2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.01380(14565-14574)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.01380
Mu FSifferman CJungerman SLi YHan MGleicher MGupta MLi Y(2024)Towards 3D Vision with Low-Cost Single-Photon Cameras2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00507(5302-5311)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00507
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents