ChromaFlash: Snapshot Hyperspectral Imaging Using Rolling Shutter Cameras
Authors: 
Dhruv Verma, 
Ian Ruffolo, David B. Lindell, Kiriakos N. Kutulakos, 
Alex MariakakisAuthors Info & Claims
Article No.: 132, Pages 1 - 31
Abstract
Hyperspectral imaging captures scene information across narrow, contiguous bands of the electromagnetic spectrum. Despite its proven utility in industrial and biomedical applications, its ubiquity has been limited by bulky form factors, slow capture times, and prohibitive costs. In this work, we propose a generalized approach to snapshot hyperspectral imaging that only requires a standard rolling shutter camera and wavelength-adjustable lighting. The crux of this approach entails using the rolling shutter as a spatiotemporal mask, varying incoming light quicker than the camera's frame rate in order for the captured image to contain rows of pixels illuminated at different wavelengths. An image reconstruction pipeline then converts this coded image into a complete hyperspectral image using sparse optimization. We demonstrate the feasibility of this approach by deploying a low-cost system called ChromaFlash, which uses a smartphone's camera for image acquisition and a series of LEDs to change the scene's illumination. We evaluated ChromaFlash through simulations on two public hyperspectral datasets and assessed its spatial and spectral accuracy across various system parameters. We also tested the real-world performance of our prototype by capturing diverse scenes under varied ambient lighting conditions. In both experiments, ChromaFlash outperformed state-of-the-art techniques that use deep learning to convert RGB images into hyperspectral ones, achieving snapshot performance not demonstrated by prior attempts at accessible hyperspectral imaging.
References
[1]
Karan Ahuja, Chris Harrison, Mayank Goel, and Robert Xiao. 2019. MeCap: Whole-Body Digitization for Low-Cost VR/AR Headsets. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 453--462. https://doi.org/10.1145/3332165.3347889
[2]
Naveed Akhtar and Ajmal Mian. 2018. Hyperspectral recovery from RGB images using Gaussian processes. IEEE transactions on pattern analysis and machine intelligence 42, 1 (2018), 100--113.
[3]
Aitor Alvarez-Gila, Joost Van De Weijer, and Estibaliz Garrote. 2017. Adversarial networks for spatial context-aware spectral image reconstruction from RGB. In Proceedings of the IEEE international conference on computer vision workshops. 480--490.
[4]
Nick Antipa, Patrick Oare, Emrah Bostan, Ren Ng, and Laura Waller. 2019. Video from stills: Lensless imaging with rolling shutter. In 2019 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--8.
[5]
Boaz Arad, Radu Timofte, Rony Yahel, Nimrod Morag, Amir Bernat, Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Luc Van Gool, Shuai Liu, Yongqiang Li, Chaoyu Feng, Lei Lei, Jiaojiao Li, Songcheng Du, Chaoxiong Wu, Yihong Leng, Rui Song, Mingwei Zhang, Chongxing Song, Shuyi Zhao, Zhiqiang Lang, Wei Wei, Lei Zhang, Renwei Dian, Tianci Shan, Anjing Guo, Chengguo Feng, Jinyang Liu, Mirko Agarla, Simone Bianco, Marco Buzzelli, Luigi Celona, Raimondo Schettini, Jiang He, Yi Xiao, Jiajun Xiao, Qiangqiang Yuan, Jie Li, Liangpei Zhang, Taesung Kwon, Dohoon Ryu, Hyokyoung Bae, Hao-Hsiang Yang, Hua-En Chang, Zhi-Kai Huang, Wei-Ting Chen, Sy-Yen Kuo, Junyu Chen, Haiwei Li, Song Liu, Sabarinathan, K Uma, B Sathya Bama, and S. Mohamed Mansoor Roomi. 2022. NTIRE 2022 Spectral Recovery Challenge and Data Set. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops. 863--881.
[6]
Gonzalo R Arce, David J Brady, Lawrence Carin, Henry Arguello, and David S Kittle. 2013. Compressive coded aperture spectral imaging: An introduction. IEEE Signal Processing Magazine 31, 1 (2013), 105--115.
[7]
Seung-Hwan Baek, Incheol Kim, Diego Gutierrez, and Min H Kim. 2017. Compact single-shot hyperspectral imaging using a prism. ACM Transactions on Graphics (TOG) 36, 6 (2017), 1--12.
[8]
Richard L Baer, William D Holland, Jack M Holm, and Poorvi L Vora. 1999. Comparison of primary and complementary color filters for CCD-based digital photography. In Sensors, Cameras, and Applications for Digital Photography, Vol. 3650. SPIE, 16--25.
[9]
Ananta Narayanan Balaji, Clayton Kimber, David Li, Shengzhi Wu, Ruofei Du, and David Kim. 2023. RetroSphere: Self-Contained Passive 3D Controller Tracking for Augmented Reality. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 4, Article 157 (jan 2023), 36 pages. https://doi.org/10.1145/3569479
[10]
Stephen Boyd, Neal Parikh, Eric Chu, Borja Peleato, Jonathan Eckstein, et al. 2011. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends® in Machine learning 3, 1 (2011), 1--122.
[11]
Nam Bui, Anh Nguyen, Phuc Nguyen, Hoang Truong, Ashwin Ashok, Thang Dinh, Robin Deterding, and Tam Vu. 2017. Pho2: Smartphone based blood oxygen level measurement systems using near-ir and red wave-guided light. In Proceedings of the 15th ACM conference on embedded network sensor systems. 1--14.
[12]
Nam Bui, Anh Nguyen, Phuc Nguyen, Hoang Truong, Ashwin Ashok, Thang Dinh, Robin Deterding, and Tam Vu. 2020. Smartphone-based spo2 measurement by exploiting wavelengths separation and chromophore compensation. ACM Transactions on Sensor Networks (TOSN) 16, 1 (2020), 1--30.
[13]
Yuanhao Cai, Jing Lin, Zudi Lin, Haoqian Wang, Yulun Zhang, Hanspeter Pfister, Radu Timofte, and Luc Van Gool. 2022. Mst++: Multi-stage spectral-wise transformer for efficient spectral reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 745--755.
[14]
Emmanuel J Candes and Terence Tao. 2005. Decoding by linear programming. IEEE transactions on information theory 51, 12 (2005), 4203--4215.
[15]
Xun Cao, Hao Du, Xin Tong, Qionghai Dai, and Stephen Lin. 2011. A prism-mask system for multispectral video acquisition. IEEE transactions on pattern analysis and machine intelligence 33, 12 (2011), 2423--2435.
[16]
Pravin V Dhole, Vijay D Dhangar, Sulochana D Shejul, and Bharti W Gawali. 2023. A Survey on Hyperspectral Sensing Techniques for Identification of Fake Pharmaceuticals Medicines. AITC-2023 and CSSP-2023 (2023), 244.
[17]
David L Donoho, Xiaoming Huo, et al. 2001. Uncertainty principles and ideal atomic decomposition. IEEE transactions on information theory 47, 7 (2001), 2845--2862.
[18]
Jason M Eichenholz, Nick Barnett, and Dave Fish. 2010. Sequential filter wheel multispectral imaging systems. In Applied industrial optics: Spectroscopy, imaging and metrology. Optica Publishing Group, ATuB2.
[19]
James E Fowler. 2014. Compressive pushbroom and whiskbroom sensing for hyperspectral remote-sensing imaging. In 2014 IEEE international conference on image processing (ICIP). IEEE, 684--688.
[20]
Biebele Joslyn Fubara, Mohamed Sedky, and David Dyke. 2020. RGB to spectral reconstruction via learned basis functions and weights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 480--481.
[21]
Mayank Goel, Eric Whitmire, Alex Mariakakis, T Scott Saponas, Neel Joshi, Dan Morris, Brian Guenter, Marcel Gavriliu, Gaetano Borriello, and Shwetak N Patel. 2015. HyperCam: hyperspectral imaging for ubiquitous computing applications. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 145--156.
[22]
Hans Grahn and Paul Geladi. 2007. Techniques and applications of hyperspectral image analysis. John Wiley & Sons.
[23]
Xavier Hadoux, Flora Hui, Jeremiah KH Lim, Colin L Masters, Alice Pébay, Sophie Chevalier, Jason Ha, Samantha Loi, Christopher J Fowler, Christopher Rowe, et al. 2019. Non-invasive in vivo hyperspectral imaging of the retina for potential biomarker use in Alzheimer's disease. Nature communications 10, 1 (2019), 4227.
[24]
Kaiming He, Jian Sun, and Xiaoou Tang. 2012. Guided image filtering. IEEE transactions on pattern analysis and machine intelligence 35, 6 (2012), 1397--1409.
[25]
Martin L Henriksen, Celine B Karlsen, Pernille Klarskov, and Mogens Hinge. 2022. Plastic classification via in-line hyperspectral camera analysis and unsupervised machine learning. Vibrational Spectroscopy 118 (2022), 103329.
[26]
Haiyan Hu, Qianyi Huang, and Qian Zhang. 2023. BabyNutri: A Cost-Effective Baby Food Macronutrients Analyzer Based on Spectral Reconstruction. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 7, 1 (2023), 1--30.
[27]
Haiquan Hu, Hao Zhou, Zhihai Xu, Qi Li, Huajun Feng, Yueting Chen, Tingting Jiang, and Wenbin Xu. 2022. Practical snapshot hyperspectral imaging with DOE. Optics and Lasers in Engineering 156 (2022), 107098.
[28]
Daniel S Jeon, Seung-Hwan Baek, Shinyoung Yi, Qiang Fu, Xiong Dun, Wolfgang Heidrich, and Min H Kim. 2019. Compact snapshot hyperspectral imaging with diffracted rotation. (2019).
[29]
Weiwei Jiang, Gabriele Marini, Niels van Berkel, Zhanna Sarsenbayeva, Zheyu Tan, Chu Luo, Xin He, Tilman Dingler, Jorge Goncalves, Yoshihiro Kawahara, et al. 2019. Probing sucrose contents in everyday drinks using miniaturized near-infrared spectroscopy scanners. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 3, 4 (2019), 1--25.
[30]
William R Johnson, Daniel W Wilson, Wolfgang Fink, Mark Humayun, and Greg Bearman. 2007. Snapshot hyperspectral imaging in ophthalmology. Journal of biomedical optics 12, 1 (2007), 014036--014036.
[31]
Sriharsha Koundinya, Himanshu Sharma, Manoj Sharma, Avinash Upadhyay, Raunak Manekar, Rudrabha Mukhopadhyay, Abhijit Karmakar, and Santanu Chaudhury. 2018. 2D-3D CNN based architectures for spectral reconstruction from RGB images. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 844--851.
[32]
Haebom Lee and Min H Kim. 2014. Building a two-way hyperspectral imaging system with liquid crystal tunable filters. In Image and Signal Processing: 6th International Conference, ICISP 2014, Cherbourg, France, June 30-July 2, 2014. Proceedings 6. Springer, 26--34.
[33]
Kareth M León-López, Laura V Galvis Carreno, and Henry Arguello Fuentes. 2018. Temporal colored coded aperture design in compressive spectral video sensing. IEEE Transactions on Image Processing 28, 1 (2018), 253--264.
[34]
Yuqi Li, Qiang Fu, and Wolfgang Heidrich. 2021. Multispectral illumination estimation using deep unrolling network. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2672--2681.
[35]
Pengfei Liu and Huaici Zhao. 2020. Adversarial networks for scale feature-attention spectral image reconstruction from a single RGB. Sensors 20, 8 (2020), 2426.
[36]
S Mahesh, DS Jayas, J Paliwal, and NDG White. 2015. Hyperspectral imaging to classify and monitor quality of agricultural materials. Journal of Stored Products Research 61 (2015), 17--26.
[37]
Hidenori Matsui, Takahiro Hashizume, and Koji Yatani. 2018. Al-light: An alcohol-sensing smart ice cube. Proceedings of the ACM on interactive, mobile, wearable and ubiquitous technologies 2, 3 (2018), 1--20.
[38]
Binu Melit Devassy and Sony George. 2021. Forensic analysis of beverage stains using hyperspectral imaging. Scientific reports 11, 1 (2021), 6512.
[39]
Kristina Monakhova, Kyrollos Yanny, Neerja Aggarwal, and Laura Waller. 2020. Spectral DiffuserCam: lensless snapshot hyperspectral imaging with a spectral filter array. Optica 7, 10 (2020), 1298--1307.
[40]
Muyao Niu, Zhuoxiao Li, Zhihang Zhong, and Yinqiang Zheng. 2023. Visibility constrained wide-band illumination spectrum design for seeing-in-the-dark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 13976--13985.
[41]
Seoung Wug Oh, Michael S Brown, Marc Pollefeys, and Seon Joo Kim. 2016. Do it yourself hyperspectral imaging with everyday digital cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2461--2469.
[42]
Ryosuke Okuta, Yuya Unno, Daisuke Nishino, Shohei Hido, and Crissman Loomis. 2017. CuPy: A NumPy-Compatible Library for NVIDIA GPU Calculations. In Proceedings of Workshop on Machine Learning Systems (LearningSys) in The Thirty-first Annual Conference on Neural Information Processing Systems (NIPS). http://learningsys.org/nips17/assets/papers/paper_16.pdf
[43]
Michael Oren and Shree K Nayar. 1994. Generalization of Lambert's reflectance model. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques. 239--246.
[44]
Jong-Il Park, Moon-Hyun Lee, Michael D. Grossberg, and Shree K. Nayar. 2007. Multispectral Imaging Using Multiplexed Illumination. In 2007 IEEE 11th International Conference on Computer Vision. 1--8. https://doi.org/10.1109/ICCV.2007.4409090
[45]
Matteo Ravasi and Ivan Vasconcelos. 2020. PyLops---A linear-operator Python library for scalable algebra and optimization. SoftwareX 11 (2020), 100361.
[46]
Kyle Rector, Cynthia L. Bennett, and Julie A. Kientz. 2013. Eyes-Free Yoga: An Exergame Using Depth Cameras for Blind & Low Vision Exercise. In Proceedings of the 15th International ACM SIGACCESS Conference on Computers and Accessibility (Bellevue, Washington) (ASSETS '13). Association for Computing Machinery, New York, NY, USA, Article 12, 8 pages. https://doi.org/10.1145/2513383.2513392
[47]
Katherine Salesin, Dario Seyb, Sarah Friday, and Wojciech Jarosz. 2022. DIY hyperspectral imaging via polarization-induced spectral filters. In 2022 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--12.
[48]
Soumyadip Sengupta, Brian Curless, Ira Kemelmacher-Shlizerman, and Steven M Seitz. 2021. A light stage on every desk. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2420--2429.
[49]
Aamir Shahzad, Mohamad Naufal Saad, Nicolas Walter, Aamir Saeed Malik, and Fabrice Meriaudeau. 2014. Hyperspectral venous image quality assessment for optimum illumination range selection based on skin tone characteristics. Biomedical engineering online 13, 1 (2014), 1--13.
[50]
Neha Sharma, Muhammad Shahzaib Waseem, Shahrzad Mirzaei, and Mohamed Hefeeda. 2023. MobiSpectral: Hyperspectral Imaging on Mobile Devices. In Proceedings of the 29th Annual International Conference on Mobile Computing and Networking. 1--15.
[51]
Mark Sheinin, Yoav Y. Scheduler, and Kiriakos N. Kutulakos. 2018. Rolling shutter imaging on the electric grid. In 2018 IEEE International Conference on Computational Photography (ICCP). 1--12. https://doi.org/10.1109/ICCPHOT.2018.8368472
[52]
Mary B Stuart, Andrew JS McGonigle, Matthew Davies, Matthew J Hobbs, Nicholas A Boone, Leigh R Stanger, Chengxi Zhu, Tom D Pering, and Jon R Willmott. 2021. Low-cost hyperspectral imaging with a smartphone. Journal of Imaging 7, 8 (2021), 136.
[53]
Muhammad Usman, Munaza Munsif, Urooj Mushtaq, Abdur-Rehman Anwar, and Nazeer Muhammad. 2021. Green gap in GaN-based light-emitting diodes: in perspective. Critical Reviews in Solid State and Materials Sciences 46, 5 (2021), 450--467.
[54]
M Vaughan. 2017. The Fabry-Perot interferometer: history, theory, practice and applications. Routledge.
[55]
Singanallur V Venkatakrishnan, Charles A Bouman, and Brendt Wohlberg. 2013. Plug-and-play priors for model based reconstruction. In 2013 IEEE global conference on signal and information processing. IEEE, 945--948.
[56]
Ashwin Wagadarikar, Renu John, Rebecca Willett, and David Brady. 2008. Single disperser design for coded aperture snapshot spectral imaging. Applied optics 47, 10 (2008), B44-B51.
[57]
Edward Jay Wang, William Li, Doug Hawkins, Terry Gernsheimer, Colette Norby-Slycord, and Shwetak N Patel. 2016. HemaApp: noninvasive blood screening of hemoglobin using smartphone cameras. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 593--604.
[58]
Edward Jay Wang, Junyi Zhu, William Li, Rajneil Rana, and Shwetak Patel. 2017. HemaApp IR: noninvasive hemoglobin measurement using unmodified smartphone cameras and built-in LEDs. In Proceedings of the 2017 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2017 ACM International Symposium on Wearable Computers. 305--308.
[59]
Xingbo Wang, Jean-Baptiste Thomas, Jon Yngve Hardeberg, and Pierre Gouton. 2013. Discrete wavelet transform based multispectral filter array demosaicking. In 2013 Colour and Visual Computing Symposium (CVCS). IEEE, 1--6.
[60]
Amanda Watson, Claire Kendell, Anush Lingamoorthy, Insup Lee, and James Weimer. 2023. Lumos: An Open-Source Device for Wearable Spectroscopy Research. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (2023), 1--24.
[61]
Yuzhen Wei, Feiyue Wu, Jie Xu, Junjing Sha, Zhangfeng Zhao, Yong He, and Xiaoli Li. 2019. Visual detection of the moisture content of tea leaves with hyperspectral imaging technology. Journal of Food Engineering 248 (2019), 89--96.
[62]
Zhiwei Xiong, Zhan Shi, Huiqun Li, Lizhi Wang, Dong Liu, and Feng Wu. 2017. Hscnn: Cnn-based hyperspectral image recovery from spectrally undersampled projections. In Proceedings of the IEEE International Conference on Computer Vision Workshops. 518--525.
[63]
F. Yasuma, T. Mitsunaga, D. Iso, and S.K. Nayar. 2008. Generalized Assorted Pixel Camera: Post-Capture Control of Resolution, Dynamic Range and Spectrum. Technical Report.
[64]
Hui-Shyong Yeo, Juyoung Lee, Andrea Bianchi, David Harris-Birtill, and Aaron Quigley. 2017. Specam: Sensing surface color and material with the front-facing camera of a mobile device. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1--9.
[65]
Zitong Yu, Xiaobai Li, and Guoying Zhao. 2021. Facial-video-based physiological signal measurement: Recent advances and affective applications. IEEE Signal Processing Magazine 38, 6 (2021), 50--58.
[66]
Xin Yuan, David J Brady, and Aggelos K Katsaggelos. 2021. Snapshot compressive imaging: Theory, algorithms, and applications. IEEE Signal Processing Magazine 38, 2 (2021), 65--88.
[67]
Yuanyuan Zhao, Hui Guo, Zhan Ma, Xun Cao, Tao Yue, and Xuemei Hu. 2019. Hyperspectral imaging with random printed mask. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 10149--10157.
[68]
Tiancheng Zhi, Bernardo R Pires, Martial Hebert, and Srinivasa G Narasimhan. 2019. Multispectral imaging for fine-grained recognition of powders on complex backgrounds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 8699--8708.
Index Terms
- ChromaFlash: Snapshot Hyperspectral Imaging Using Rolling Shutter Cameras
Recommendations
Self-calibration of omnidirectional multi-cameras including synchronization and rolling shutter
Deal with consumer 360 cameras and spherical cameras without a privileged direction.Initialize the time offsets and intrinsic parameters using monocular structure-from-motion.Start multi-camera structure-from-motion with central and global shutter ...
Rolling Shutter Stereo
ICCV '13: Proceedings of the 2013 IEEE International Conference on Computer VisionA huge fraction of cameras used nowadays is based on CMOS sensors with a rolling shutter that exposes the image line by line. For dynamic scenes/cameras this introduces undesired effects like stretch, shear and wobble. It has been shown earlier that ...
Motion and velocity estimation of rolling shutter cameras
CVMP '12: Proceedings of the 9th European Conference on Visual Media ProductionMost modern camera designs based on CMOS sensors do not have a global, but a rolling shutter that exposes the lines of the sensor at different times. This leads to distortions under camera or object motion and affects computer vision techniques. This ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 September 2024
Published in IMWUT Volume 8, Issue 3
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Funding Sources
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 291Total Downloads
- Downloads (Last 12 months)291
- Downloads (Last 6 weeks)44
Reflects downloads up to 23 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in