Open AccessReview

Three-Dimensional Photon Counting Imaging with Axially Distributed Sensing

Myungjin Cho

^1,* and

Bahram Javidi

Department of Electrical, Electronic, and Control Engineering, Institute of Information Telecommunication Convergence (IITC), Hankyong National University, 327 Chungang-ro, Anseong-si, Kyonggi-do 456-749, Korea

Electrical and Computer Engineering Department, University of Connecticut, Unit 4157, Storrs, CT 06269, USA

Author to whom correspondence should be addressed.

Sensors 2016, 16(8), 1184; https://doi.org/10.3390/s16081184

Submission received: 25 March 2016 / Revised: 7 July 2016 / Accepted: 25 July 2016 / Published: 28 July 2016

(This article belongs to the Special Issue Photon-Counting Image Sensors)

Download

Browse Figures

Graphical abstract
"> Figure 1
Mathematical model of photon counting detector. "> Figure 2
Photon counting imaging with ADS. (a) Image sensing; (b) Computational reconstruction, k is the number of recorded images by ADS. "> Figure 3
(a) Experimental setup for ADS; (b) Non-occluded 3D object; (c) Partially occluded 3D object. "> Figure 4
Conventional elemental images with slightly different perspectives by ADS, for: (a,b) non-occluded object; (c,d) occluded object, respectively. "> Figure 5
Photon-limited elemental images with Np = 10,000 of non-occluded object and occluded object at (a,c) farthest location from camera and (b,d) closest location from camera. "> Figure 6
Reconstruction results of photon counting ADS with Np = 10,000 for (a) non-occluded object and (b) occluded object. Enlarged part of the reconstructed 3D images for (c) non-occluded object and (d) occluded object. "> Figure 7
Peak Signal to Noise Ratio (PSNR). ">

Versions Notes

Abstract

In this paper, we review three-dimensional (3D) photon counting imaging with axially distributed sensing. Under severely photon-starved conditions, we have proposed various imaging and algorithmic approaches to reconstruct a scene in 3D, which are not possible by using conventional imaging system due to lack of sufficient number of photons. In this paper, we present an overview of optical sensing and imaging system along with dedicated algorithms for reconstructing 3D scenes by photon counting axially distributed sensing, which may be implemented by moving a single image sensor along its optical axis. To visualize the 3D image, statistical estimation methods and computational reconstruction of axially distributed sensing is applied.

Keywords:

axially distributed sensing; photon counting imaging; statistical estimation; Poisson distribution

Graphical Abstract

1. Introduction

Under severely photon-starved conditions, scenes recorded optically may not be properly reconstructed by conventional imaging systems. In many fields, such as noninvasive microscopy, night vision, astronomy, military applications, etc., image acquisition or visualization may be carried out in a low light level environment. Recently, many approaches for three-dimensional (3D) photon counting imaging have been reported [1,2,3,4,5,6]. For example, 3D information can be recorded and reconstructed under photon-starved conditions with photon counting integral imaging [7,8,9,10,11,12,13]. In 3D photon counting imaging, 3D images can be visualized by statistical estimations, such as maximum likelihood estimation [2] and Bayesian approaches [4]. Photon counting detection under such conditions can be modeled using a Poisson distribution since photon events may occur rarely in unit time and space [14]. Using this mathematical photon counting imaging model, 3D visualization and object recognition can be performed under photon-starved conditions. Additionally, optical encryption with improved security level has been accomplished photon counting imaging properties [15,16,17].

In photon counting imaging, the visual quality of the recorded image or the reconstructed image depends on the number of photons from the scene. To enhance its visual quality, some techniques have been proposed [2,3,4]. Three-dimensional photon counting imaging captures multiple 2D images from the scenes using a lenslet array or moving camera. The statistical properties of the optical rays, as well as photon counting, are used and the visual quality of the reconstructed image can be enhanced.

To obtain the 3D information with high resolution from the scenes and remove the requirement of lateral parallax of image sensor by integral imaging [11], axially distributed sensing (ADS) may be used [12]. In order to obtain the 3D information, ADS uses only a single camera moving along its optical axis. We show that photon counting with ADS [5] can be used to obtain the 3D information of the scenes under photon-starved conditions. We can obtain the depth map of the 3D objects. Thus, we can create the 3D profile of the object and regenerate elemental images for multiple viewing points for 3D display [18,19,20,21].

In this paper, we present an overview of the basic concept of photon counting imaging, and our work on 3D reconstruction using 3D photon counting ADS along with some experiments to illustrate 3D photon counting imaging with ADS.

2. 3D Photon Counting Imaging

2.1. Mathematical Model of Photon Counting Detection

Photon counting detection may be modeled by Poisson distribution because the photon events occur rarely in unit time and space under photon-starved conditions [14]. The photon counting detection fundamental steps are illustrated in Figure 1. For computational simplicity, the image has only one-dimension. Using the following equation, the photon counting image can be constructed [2].

λ_{x} = \frac{I_{x}}{\sum_{x = 1}^{N_{x}} I_{x}}

(1)

C_{x} | λ_{x} ~ P o i s s o n (N_{p} λ_{x})

(2)

where I_x is the light intensity of the image at pixel x, N_x is the total number of pixels in the image, λ_x is the normalized irradiance at pixel x, N_p is the extracted number of photons from the image, C_x is the number of photons at pixel x, respectively. Here, the total energy of λ_x is unity.

Now, we have 2D photon-limited images, which are generated from Equations (1) and (2). When N_p is very small or the scenes are under severely photon-starved conditions, the image cannot be visualized or recognized. We have shown that passive 3D imaging technique such as integral imaging [1,2,3,4] can enhance the visual quality of these photon-limited images and obtain the 3D information. In integral imaging, 3D information can be recorded through a lenslet array or a camera array. Here, multiple 2D images with different perspectives can be acquired. These images are referred to as elemental images. To obtain high lateral and depth resolutions, synthetic aperture integral imaging (SAII) [11], which uses multiple cameras can be used. Then, using computational integral imaging reconstruction (CIIR) [13], 3D images with enhanced visual quality can be reconstructed. In 3D photon counting integral imaging, to estimate 3D information from multiple photon-limited images, statistical estimations such as maximum likelihood estimation (MLE) [2] may be used, as well as computational reconstruction [13].

2.2. Photon Counting Axially Distributed Sensing (ADS)

To remove lateral movement of image sensor required by SAII, axially distributed sensing has been reported [12]. It can capture multiple 2D images with slightly different perspectives by moving single image sensor along its optical axis. Thus, we can adopt this technique to 3D photon counting imaging. Figure 2 shows the basic concept of 3D photon counting imaging with ADS for pickup and reconstruction.

In the pickup process, by moving single photon counting camera along its optical axis, multiple 2D images with slightly different perspectives can be recorded as shown in Figure 2a. Computational reconstruction of ADS can be implemented by considering slightly different magnification ratio for each photon-limited image as the following equation [5]:

M_{k} (z_{r}) = \frac{z_{r} - k Δ z}{z_{r}}

(3)

where Δz is the moving step for single camera along its optical axis, z_r is the reconstruction depth, and k is the index of the recorded images by ADS, respectively. Since magnification causes the degradation of the reconstructed 3D image quality by the image interpolation method, in this paper, demagnification is used. Then, using MLE process [2], computational reconstruction of photon counting ADS as shown in Figure 2b can be implemented by follows [5]:

L (N_{p} λ_{k} | C_{k}) = \prod_{k = 1}^{K} \frac{{(N_{p} λ_{k})}^{C_{k}} e^{- N_{p} λ_{k}}}{C_{k}!}

(4)

l (N_{p} λ_{k} | C_{k}) \propto \sum_{k = 1}^{K} [C_{k} \log (N_{p} λ_{k}) - N_{p} λ_{k}]

(5)

\frac{\partial (N_{p} λ_{k} | C_{k})}{\partial λ_{k}} = 0, {\hat{λ}}_{k} = \frac{C_{k}}{N_{p}}

(6)

\hat{I} {(x, z_{r})}_{A D S} = \frac{1}{N_{p} K} \sum_{k = 1}^{K} C_{k} (\frac{x}{M_{k} (z_{r})})

(7)

where L(·|·) and l(·|·) are the likelihood function and log-likelihood function, C_k is the kth photon-limited elemental image by ADS and K is the total number of the captured photon-limited elemental images, respectively.

3. Experimental Results

3.1. Photon Counting Imaging with Axially Distributed Sensing

3.1.1. Experimental Setup

Experimental setup for photon counting ADS is illustrated in Figure 3a. In this setup, the focal length of the camera lens is 50 mm. The camera has 1000 (H) × 1000 (V) pixels and axial separation between moving image sensor, Δz, is 2 mm. We used a 3D car model in Figure 3b as the 3D object. Its location is z_r = 320 mm. Finally, we recorded 50 multiple images using ADS. In this experiment, we use both non-occluded and occluded 3D objects as shown in Figure 3b,c. The photon counting images are obtained digitally by applying the Poisson model (Equations (1) and (2)) to the digitally captured elemental images.

Figure 4 shows conventional elemental images by ADS for non-occluded 3D object and occluded 3D object. It is noticed that slightly different perspectives between farthest and closest images exist. Thus, using these perspectives and computational reconstruction of ADS, we can reconstruct 3D images.

3.1.2. Results

Figure 5 shows photon-limited elemental images for photon counting ADS of non-occluded and occluded objects at farthest and closest positions, which are obtained by using Equations (1) and (2) applied to the digitally captured images. Since the number of photons is low, that is N_p = 10,000 (0.01 photons/pixel), its visual quality is low and the objects are not well recognized. However, using computational reconstruction of ADS, as depicted in Equations (3)–(7), the visual quality of the reconstructed image can be improved, as shown in Figure 6. To evaluate the visual quality of the reconstructed 3D images, we calculate the peak signal to noise ratio (PSNR) between the original reconstructed Three-dimensional images using conventional ADS and the experimental results as shown in Figure 7. The plot has some fluctuations because photons are generated by Poisson random process.

Using multiple images obtained by ADS, we can regenerate the elemental images for 3D multi-view display [18]. Then, the depth map of the 3D objects can be extracted by 3D profilometry [19]. In [19], the extracted depth with contours of equal depth is shown. The errors of the estimated depth may occur due to the specular reflection off of the glossy surface, which departs from the Lambertian assumption. In addition, in [19], the computational reconstruction results of the 3D objects by slicing them in a certain depth range are shown.

4. Conclusions

In this paper, we have presented an overview of 3D photon counting imaging system under photon-starved conditions using ADS. Photon counting ADS uses camera movement along the optical axis unlike photon counting integral imaging which requires lateral parallax. It can reconstruct 3D images under photon starved conditions, including occluded objects. For real-time photon counting ADS, faster reconstruction algorithm with a Graphic Processing Unit (GPU) may be required.

Acknowledgments

This research was supported in part by Basic Science Research Program through the National Research Foundation of Korea (NRF) funded by the Ministry of Education (NRF-2015R1A2A1A16074936). B. Javidi acknowledges support from NSF under grant # NSF/IIS-1422179.

Author Contributions

M. Cho and B. Javidi conceived and designed the experiments; M. Cho performed the experiments; M. Cho and B. Javidi analyzed the data; M. Cho and B. Javidi wrote the paper.

Conflicts of Interest

The authors declare no conflict of interest.

Abbreviations

The following abbreviations are used in this manuscript:

ADS	Axially distributed sensing
CIIR	Computational integral imaging reconstruction
MLE	Maximum likelihood estimation
SAII	Synthetic aperture integral imaging

References

Yeom, S.; Javidi, B.; Watson, E. Photon counting passive 3D image sensing for automatic target recognition. Opt. Exp. 2005, 13, 9310–9330. [Google Scholar] [CrossRef]
Tavakoli, B.; Javidi, B.; Watson, E. Three dimensional visualization by photon counting computational integral imaging. Opt. Exp. 2008, 16, 4426–4436. [Google Scholar] [CrossRef]
DaneshPanah, M.; Javidi, B.; Watson, E. Three dimensional object recognition with photon counting imagery in the presence of noise. Opt. Exp. 2010, 18, 26450–26460. [Google Scholar] [CrossRef] [PubMed]
Jung, J.; Cho, M.; Dey, D.K.; Javidi, B. Three-dimensional photon counting integral imaging using Bayesian estimation. Opt. Lett. 2010, 35, 1825–1827. [Google Scholar] [CrossRef] [PubMed]
Cho, M.; Javidi, B. Three-dimensional photon counting axially distributed image sensing. IEEE/OSA J. Disp. Tech. 2013, 9, 56–62. [Google Scholar] [CrossRef]
Cho, M.; Mahalanobis, A.; Javidi, B. 3D passive photon counting automatic target recognition using advanced correlation filters. Opt. Lett. 2011, 36, 861–863. [Google Scholar] [CrossRef] [PubMed]
Lippmann, G. La Photographie Integrale. Comp. Rend. Acad. Sci. 1908, 146, 446–451. (In French) [Google Scholar]
Okoshi, T. Three-Dimensional Imaging Technique; Academic: New York, NY, USA, 1976. [Google Scholar]
Okano, F.; Arai, J.; Mitani, K.; Okui, M. Real-time integral imaging based on extremely high resolution video system. IEEE Proc. 2006, 94, 490–501. [Google Scholar] [CrossRef]
Javidi, B.; Okano, F.; Son, J.Y. Three-Dimensional Imaging, Visualization, Display; Springer: New York, NY, USA, 2009. [Google Scholar]
Jang, J.-S.; Javidi, B. Three-dimensional synthetic aperture integral imaging. Opt. Lett. 2002, 27, 1144–1146. [Google Scholar] [CrossRef] [PubMed]
Schulein, R.; DaneshPanah, M.; Javidi, B. 3D imaging with axially distributed sensing. Opt. Lett. 2009, 34, 2012–2014. [Google Scholar] [CrossRef] [PubMed]
Hong, S.-H.; Jang, J.-S.; Javidi, B. Three-dimensional volumetric object reconstruction using computational integral imaging. Opt. Exp. 2004, 12, 483–491. [Google Scholar] [CrossRef]
Goodman, J.W. Statistical Optics; Wiley: New York, NY, USA, 1985. [Google Scholar]
Perez-Cabre, E.; Abril, H.C.; Millan, M.; Javidi, B. Photon-counting double-random-phase encoding for secure image verification and retrieval. J. Opt. 2012, 14. [Google Scholar] [CrossRef]
Pérez-Cabré, E.; Cho, M.; Javidi, B. Information authentication using photon counting double random phase encrypted images. J. Opt. Lett. 2010, 35, 22–24. [Google Scholar] [CrossRef] [PubMed]
Matoba, O.; Nomura, T.; Perez-Cabre, E.; Millan, M.S.; Javidi, B. Optical techniques for information security. IEEE J. Proc. 2009, 97, 1128–1148. [Google Scholar] [CrossRef]
Cho, M.; Shin, D. 3D integral imaging display using axially recorded multiple images. J. Opt. Soc. Korea 2013, 17, 410–414. [Google Scholar] [CrossRef]
Daneshpanah, M.; Javidi, B. Profilometry and optical slicing by passive three dimensional imaging. Opt. Lett. 2009, 34, 1105–1107. [Google Scholar] [CrossRef] [PubMed]
Pollefeys, M.; Koch, R.; Vergauwen, M.; Deknuydt, B.; Gool, L.V. Three-dimensional scene reconstruction from images. Proc. SPIE 2000, 3958, 215–226. [Google Scholar]
Saavedra, G.; Martinez-Cuenca, R.; Martinez-Corral, M.; Navarro, H.; Daneshpanah, M.; Javidi, B. Digital slicing of 3D scenes by Fourier filtering of integral images. Opt. Express 2008, 16, 17154–17160. [Google Scholar] [CrossRef] [PubMed]

Figure 1. Mathematical model of photon counting detector.

Figure 2. Photon counting imaging with ADS. (a) Image sensing; (b) Computational reconstruction, k is the number of recorded images by ADS.

Figure 3. (a) Experimental setup for ADS; (b) Non-occluded 3D object; (c) Partially occluded 3D object.

Figure 4. Conventional elemental images with slightly different perspectives by ADS, for: (a,b) non-occluded object; (c,d) occluded object, respectively.

Figure 5. Photon-limited elemental images with N_p = 10,000 of non-occluded object and occluded object at (a,c) farthest location from camera and (b,d) closest location from camera.

Figure 6. Reconstruction results of photon counting ADS with N_p = 10,000 for (a) non-occluded object and (b) occluded object. Enlarged part of the reconstructed 3D images for (c) non-occluded object and (d) occluded object.

Figure 7. Peak Signal to Noise Ratio (PSNR).

© 2016 by the authors; licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC-BY) license (http://creativecommons.org/licenses/by/4.0/).

Share and Cite

MDPI and ACS Style

Cho, M.; Javidi, B. Three-Dimensional Photon Counting Imaging with Axially Distributed Sensing. Sensors 2016, 16, 1184. https://doi.org/10.3390/s16081184

AMA Style

Cho M, Javidi B. Three-Dimensional Photon Counting Imaging with Axially Distributed Sensing. Sensors. 2016; 16(8):1184. https://doi.org/10.3390/s16081184

Chicago/Turabian Style

Cho, Myungjin, and Bahram Javidi. 2016. "Three-Dimensional Photon Counting Imaging with Axially Distributed Sensing" Sensors 16, no. 8: 1184. https://doi.org/10.3390/s16081184

APA Style

Cho, M., & Javidi, B. (2016). Three-Dimensional Photon Counting Imaging with Axially Distributed Sensing. Sensors, 16(8), 1184. https://doi.org/10.3390/s16081184

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Menu