Abstract
Hyperspectral reconstruction from RGB images have been proposed as an alternative approach to overcome the limited scope of conventional hyperspectral imaging. With the advancement in convolutional neural networks (CNNs), this approach has recently gained attention. However, most of the existing state-of-the-art frameworks focuses on hyperspectral reconstruction from clean RGB images i.e. with no noise and degradation. This limits the applicability of the current deep learning frameworks on real-world RGB images. Thus, in this paper we propose a novel deep learning framework for robust real RGB (with noise and degradation) for hyperspectral reconstruction. The proposed framework is motivated towards the extraction of noise-free features from real RGB images crucial for hyperspectral reconstruction. This is achieved with the use of a deep convolutional auto-encoder (DCAE) module and subsequent utilization of these noise-free features in an attention-based deep spectral back-projection network (DSBPN) for hyperspectral reconstruction. The proposed framework (DCAE-DSBPN) is trained in an end-to-end manner with joint optimization of denoising loss and hyperspectral reconstruction loss. Experimental results demonstrates that the proposed framework outperforms the existing state-of-the-art methods.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Alvarez-Gila, A., Van De Weijer, J., Garrote, E.: Adversarial networks for spatial context-aware spectral image reconstruction from RGB. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 480–490 (2017)
Arad, B., Ben-Shahar, O.: Sparse recovery of hyperspectral signal from natural RGB images. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV 2016. LNCS, vol. 9911, pp. 19–34. Springer, Cham (2016). https://doi.org/10.1007/978-3-319-46478-7_2
Arad, B., Ben-Shahar, O., Timofte, R.: NTIRE 2018 challenge on spectral reconstruction from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 929–938 (2018)
Arad, B., Timofte, R., Ben-Shahar, O., Lin, Y.T., Finlayson, G.D.: NTIRE 2020 challenge on spectral reconstruction from an RGB image. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 446–447 (2020)
Can, Y.B., Timofte, R.: An efficient CNN for spectral reconstruction from RGB images. arXiv preprint arXiv:1804.04647 (2018)
Chakrabarti, A., Zickler, T.: Statistics of real-world hyperspectral images. In: CVPR 2011, pp. 193–200. IEEE (2011)
Chen, S., Wang, G., Cui, X., Liu, Q.: Stepwise method based on wiener estimation for spectral reconstruction in spectroscopic raman imaging. Opt. Express 25(2), 1005–1018 (2017)
Cui, J., et al.: Pet image denoising using unsupervised deep learning. Eur. J. Nucl. Med. Mol. Imaging 46(13), 2780–2789 (2019)
Fubara, B.J., Sedky, M., Dyke, D.: RGB to spectral reconstruction via learned basis functions and weights. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 480–481 (2020)
Funamizu, H., Shimoma, S., Yuasa, T., Aizu, Y.: Effects of spatiotemporal averaging processes on the estimation of spectral reflectance in color digital holography using speckle illuminations. Appl. Opt. 53(30), 7072–7080 (2014)
Girshick, R., Donahue, J., Darrell, T., Malik, J.: Rich feature hierarchies for accurate object detection and semantic segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 580–587 (2014)
Godard, C., Matzen, K., Uyttendaele, M.: Deep burst denoising. In: Proceedings of the European Conference on Computer Vision (ECCV) (2018)
Goel, M., et al.: Hypercam: hyperspectral imaging for ubiquitous computing applications. In: Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 145–156 (2015)
Goetz, A.F., Vane, G., Solomon, J.E., Rock, B.N.: Imaging spectrometry for earth remote sensing. Science 228(4704), 1147–1153 (1985)
Han, Y., Ye, J.C.: Framing U-Net via deep convolutional framelets: application to sparse-view CT. IEEE Trans. Med. Imaging 37(6), 1418–1429 (2018)
Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018)
Jifara, W., Jiang, F., Rho, S., Cheng, M., Liu, S.: Medical image denoising using convolutional neural network: a residual learning approach. J. Supercomput. 75(2), 704–718 (2019)
Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980
Koundinya, S., et al.: 2D-3D CNN based architectures for spectral reconstruction from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 844–851 (2018)
Li, J., Wu, C., Song, R., Li, Y., Liu, F.: Adaptive weighted attention network with camera spectral sensitivity prior for spectral reconstruction from RGB images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 462–463 (2020)
Liu, P., Zhao, H.: Adversarial networks for scale feature-attention spectral image reconstruction from a single RGB. Sensors 20(8), 2426 (2020)
Nie, S., Gu, L., Zheng, Y., Lam, A., Ono, N., Sato, I.: Deeply learned filter response functions for hyperspectral reconstruction. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4767–4776 (2018)
Oh, S.W., Brown, M.S., Pollefeys, M., Kim, S.J.: Do it yourself hyperspectral imaging with everyday digital cameras. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2461–2469 (2016)
Peng, H., Chen, X., Zhao, J.: Residual pixel attention network for spectral reconstruction from RGB images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 486–487 (2020)
Shi, Z., Chen, C., Xiong, Z., Liu, D., Wu, F.: HSCNN+: advanced CNN-based hyperspectral recovery from RGB images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 939–947 (2018)
Takatani, T., Aoto, T., Mukaigawa, Y.: One-shot hyperspectral imaging using faced reflectors. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4039–4047 (2017)
Vaswani, A., et al.: Attention is all you need. CoRR abs/1706.03762 (2017). http://arxiv.org/abs/1706.03762
Wang, W., Wang, J.: Double ghost convolution attention mechanism network: a framework for hyperspectral reconstruction of a single RGB image. Sensors 21(2), 666 (2021)
Xiong, Z., Shi, Z., Li, H., Wang, L., Liu, D., Wu, F.: HSCNN: CNN-based hyperspectral image recovery from spectrally undersampled projections. In: Proceedings of the IEEE International Conference on Computer Vision Workshops, pp. 518–525 (2017)
Yan, Y., Zhang, L., Li, J., Wei, W., Zhang, Y.: Accurate spectral super-resolution from single RGB image using multi-scale CNN. In: Lai, J.-H., et al. (eds.) PRCV 2018. LNCS, vol. 11257, pp. 206–217. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-03335-4_18
Yasuma, F., Mitsunaga, T., Iso, D., Nayar, S.K.: Generalized assorted pixel camera: postcapture control of resolution, dynamic range, and spectrum. IEEE Trans. Image Process. 19(9), 2241–2253 (2010)
Zhao, J., et al.: Deep learning in hyperspectral image reconstruction from single RGB images-a case study on tomato quality parameters. Remote Sens. 12(19), 3258 (2020)
Zhao, Y., Po, L.M., Yan, Q., Liu, W., Lin, T.: Hierarchical regression network for spectral reconstruction from RGB images. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 422–423 (2020)
Zhu, Y., Li, B., Xu, X.: Spectral reconstruction and accuracy appraisal based on pseudo inverse method. In: 2012 Symposium on Photonics and Optoelectronics, pp. 1–3. IEEE (2012)
Zou, C., Wei, M.: Cluster-based deep convolutional networks for spectral reconstruction from RGB images. Neurocomputing 464, 342–351 (2021)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2024 Springer Nature Switzerland AG
About this paper
Cite this paper
Shukla, A. et al. (2024). Auto-Encoder Guided Attention Based Network for Hyperspectral Recovery from Real RGB Images. In: Ghosh, A., King, I., Bhattacharyya, M., Sankar Ray, S., K. Pal, S. (eds) Pattern Recognition and Machine Intelligence. PReMI 2021. Lecture Notes in Computer Science, vol 13102. Springer, Cham. https://doi.org/10.1007/978-3-031-12700-7_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-12700-7_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-12699-4
Online ISBN: 978-3-031-12700-7
eBook Packages: Computer ScienceComputer Science (R0)