More Web Proxy on the site http://driver.im/

research-article

Deep Self-Supervised Hyperspectral Image Reconstruction

Authors:

Xian-Hua HanAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications and Applications, Volume 18, Issue 3s

Article No.: 149, Pages 1 - 20

https://doi.org/10.1145/3510373

Published: 01 November 2022 Publication History

Abstract

Reconstructing a high-resolution hyperspectral (HR-HS) image via merging a low-resolution hyperspectral (LR-HS) image and a high-resolution RGB (HR-RGB) image has become a hot research topic, and can greatly benefit for different subsequent high-level vision tasks. Recently, deep learning–based approaches have evolved for HS image reconstruction and validated impressive performance. However, to learn a good reconstruction model in the deep learning–based methods, it is mandatory to previously collect large-scale training triplets consisting of the LR-HS, HR-RGB, and HR-HS images, which is difficult to be collected in real applications. This study proposes a deep self-supervised HS image reconstruction framework (DSSH), which does not have to depend on any handcrafted prior and previously collected training triplets at all. The proposed DSSH method leverages the designed network architecture itself for capturing the prior of the underlying structure in the latent HR-HS image and employs the observed LR-HS and HR-RGB images only for network parameter learning. Experiments on two benchmark HS image datasets validated that the proposed DSSH method manifests very impressive reconstruction performance, and is even better than some state-of-the-art supervised learning approaches.

References

[1]

Naveed Akhtar, Faisal Shafait, and Ajmal Mian. 2014. Sparse spatio-spectral representation for hyperspectral image super-resolution. In European Conference on Computer Vision. Springer, 63–78.

[2]

Naveed Akhtar, Faisal Shafait, and Ajmal Mian. 2015. Bayesian sparse representation for hyperspectral image super resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3631–3640.

[3]

Naveed Akhtar, Faisal Shafait, and Ajmal Mian. 2016. Hierarchical beta process with Gaussian process prior for hyperspectral image super resolution. In European Conference on Computer Vision. Springer, 103–120.

[4]

Michael Barnes, Zhihong Pan, and Sizhong Zhang. 2018. Systems and methods for hyperspectral medical imaging using real-time projection of spectral information. (Feb. 2018). US Patent No. 9,883,833. Filed May 13, 2009. Issued Feb. 6, 2018.

[5]

José M. Bioucas-Dias, Antonio Plaza, Gustavo Camps-Valls, Paul Scheunders, Nasser Nasrabadi, and Jocelyn Chanussot. 2013. Hyperspectral remote sensing data analysis and future challenges. IEEE Geoscience and Remote Sensing Magazine 1, 2 (2013), 6–36.

[6]

Charlotte A. Bishop, Jian Guo Liu, and Philippa J. Mason. 2011. Hyperspectral remote sensing for mineral exploration in Pulang, Yunnan Province, China. International Journal of Remote Sensing 32, 9 (2011), 2409–2426.

Digital Library

[7]

Ayan Chakrabarti and Todd Zickler. 2011. Statistics of real-world hyperspectral images. In CVPR 2011. IEEE, 193–200.

Digital Library

[8]

Hendrik Dahlkamp, Adrian Kaehler, David Stavens, Sebastian Thrun, and Gary R. Bradski. 2006. Self-supervised monocular road detection in desert terrain. In Robotics: Science and Systems, Vol. 38.

[9]

Renwei Dian, Shutao Li, Anjing Guo, and Leyuan Fang. 2018. Deep hyperspectral image sharpening. IEEE Transactions on Neural Networks and Learning Systems99 (2018), 1–11.

[10]

Weisheng Dong, Fazuo Fu, Guangming Shi, Xun Cao, Jinjian Wu, Guangyu Li, and Xin Li. 2016. Hyperspectral image super-resolution via non-negative structured sparse representation. IEEE Transactions on Image Processing 25, 5 (2016), 2337–2352.

Digital Library

[11]

Ying Fu, Tao Zhang, Yinqiang Zheng, Debing Zhang, and Hua Huang. 2019. Hyperspectral image super-resolution with optimized RGB guidance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 11661–11670.

[12]

Xian-Hua Han and Yen-Wei Chen. 2019. Deep residual network of spectral and spatial fusion for hyperspectral image super-resolution. In 2019 IEEE 5th International Conference on Multimedia Big Data (BigMM’19). IEEE, 266–270.

[13]

Xian-Hua Han, Boxin Shi, and Yinqiang Zheng. 2018. Self-similarity constrained sparse representation for hyperspectral image super-resolution. IEEE Transactions on Image Processing 27, 11 (2018), 5625–5637.

Digital Library

[14]

Xian-Hua Han, Boxin Shi, and Yinqiang Zheng. 2018. SSF-CNN: Spatial and spectral fusion with CNN for hyperspectral image super-resolution. In 2018 25th IEEE International Conference on Image Processing (ICIP’18). IEEE, 2506–2510.

[15]

Xian-Hua Han, YongQing Sun, and Yen-Wei Chen. 2019. Residual component estimating CNN for image super-resolution. In 2019 IEEE 5th International Conference on Multimedia Big Data (BigMM’19). IEEE, 443–447.

[16]

Xian-Hua Han, YinQiang Zheng, and Yen-Wei Chen. 2019. Multi-level and multi-scale spatial and spectral fusion CNN for hyperspectral image super-resolution. In Proceedings of the IEEE International Conference on Computer Vision Workshops.

[17]

Wei He, Hongyan Zhang, Liangpei Zhang, and Huanfeng Shen. 2015. Total-variation-regularized low-rank matrix factorization for hyperspectral image restoration. IEEE Transactions on Geoscience and Remote Sensing 54, 1 (2015), 178–188.

[18]

Zhi He, Han Liu, Yiwen Wang, and Jie Hu. 2017. Generative adversarial networks-based semi-supervised learning for hyperspectral image classification. Remote Sensing 9, 10 (2017), 1042.

[19]

Ryuji Imamura, Tatsuki Itasaka, and Masahiro Okuda. 2019. Self-supervised hyperspectral image restoration using separable image prior. arXiv:1907.00651.

[20]

Ryuji Imamura, Tatsuki Itasaka, and Masahiro Okuda. 2019. Zero-shot hyperspectral image denoising with separable image prior. In Proceedings of the IEEE International Conference on Computer Vision Workshops.

[21]

Rei Kawakami, Yasuyuki Matsushita, John Wright, Moshe Ben-Ezra, Yu-Wing Tai, and Katsushi Ikeuchi. 2011. High-resolution hyperspectral imaging via matrix factorization. In CVPR 2011. IEEE, 2329–2336.

Digital Library

[22]

Hyunsoo Kim and Haesun Park. 2007. Sparse non-negative matrix factorizations via alternating non-negativity-constrained least squares for microarray data analysis. Bioinformatics 23, 12 (2007), 1495–1502.

[23]

Chiman Kwan, Joon Hee Choi, Stanley Chan, Jin Zhou, and Bence Budavari. 2017. Resolution enhancement for hyperspectral images: A super-resolution and fusion approach. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’17). IEEE, 6180–6184.

Digital Library

[24]

Craig A. Laben and Bernard V. Brower. 2000. Process for enhancing the spatial resolution of multispectral imagery using pan-sharpening. (Jan. 2000). US Patent 6,011,875. Filed Apr. 29, 1998. Issued Jan. 4, 2000.

[25]

Charis Lanaras, Emmanuel Baltsavias, and Konrad Schindler. 2015. Hyperspectral super-resolution by coupled spectral unmixing. In Proceedings of the IEEE International Conference on Computer Vision. 3586–3594.

Digital Library

[26]

Yunsong Li, Jing Hu, Xi Zhao, Weiying Xie, and JiaoJiao Li. 2017. Hyperspectral image super-resolution using deep convolutional neural network. Neurocomputing 266 (2017), 29–41.

[27]

Zhe Liu, Yinqiang Zheng, and Xian-Hua Han. 2020. Unsupervised multispectral and hyperspectral image fusion with deep spatial and spectral priors. In Proceedings of the Asian Conference on Computer Vision.

[28]

Farid Melgani and Lorenzo Bruzzone. 2004. Classification of hyperspectral remote sensing images with support vector machines. IEEE Transactions on Geoscience and Remote Sensing 42, 8 (2004), 1778–1790.

[29]

Ying Qu, Hairong Qi, and Chiman Kwan. 2018. Unsupervised sparse Dirichlet-Net for hyperspectral image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2511–2520.

[30]

Oleksii Sidorov and Jon Yngve Hardeberg. 2019. Deep hyperspectral prior: Denoising, inpainting, super-resolution. arXiv:1902.00301.

[31]

Oleksii Sidorov and Jon Yngve Hardeberg. 2019. Deep hyperspectral prior: Denoising, inpainting, super-resolution. CoRR abs/1902.00301 (2019). arXiv:1902.00301. http://arxiv.org/abs/1902.00301.

[32]

Murat Şımşek and Ediz Polat. 2015. The effect of dictionary learning algorithms on super-resolution hyperspectral reconstruction. In 2015 XXV International Conference on Information, Communication and Automation Technologies (ICAT’15). IEEE, 1–5.

Digital Library

[33]

Tatsumi Uezato, Danfeng Hong, Naoto Yokoya, and Wei He. 2020. Guided deep decoder: Unsupervised image pair fusion. In European Conference on Computer Vision. Springer, 87–102.

Digital Library

[34]

Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. 2018. Deep image prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9446–9454.

[35]

Tuan Vo-Dinh. 2004. A hyperspectral imaging system for in vivo optical diagnostics. IEEE Engineering in Medicine and Biology Magazine 23, 5 (2004), 40–49.

[36]

Lizhi Wang, Chen Sun, Ying Fu, Min H. Kim, and Hua Huang. 2019. Hyperspectral image reconstruction using a deep spatial-spectral prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8032–8041.

[37]

Eliot Wycoff, Tsung-Han Chan, Kui Jia, Wing-Kin Ma, and Yi Ma. 2013. A non-negative sparse promoting algorithm for high resolution hyperspectral imaging. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 1409–1413.

[38]

Qi Xie, Minghao Zhou, Qian Zhao, Deyu Meng, Wangmeng Zuo, and Zongben Xu. 2019. Multispectral and hyperspectral image fusion by MS/HS fusion net. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 1585–1594.

[39]

Jun-Li Xu, Cecilia Riccioli, and Da-Wen Sun. 2017. Comparison of hyperspectral imaging and computer vision for automatic differentiation of organically and conventionally farmed salmon. Journal of Food Engineering 196 (2017), 170–182.

[40]

Fumihito Yasuma, Tomoo Mitsunaga, Daisuke Iso, and Shree K. Nayar. 2010. Generalized assorted pixel camera: Postcapture control of resolution, dynamic range, and spectrum. IEEE Transactions on Image Processing 19, 9 (2010), 2241–2253.

Digital Library

[41]

Naoto Yokoya, Takehisa Yairi, and Akira Iwasaki. 2011. Coupled non-negative matrix factorization (CNMF) for hyperspectral and multispectral data fusion: Application to pasture classification. In 2011 IEEE International Geoscience and Remote Sensing Symposium. IEEE, 1779–1782.

[42]

Naoto Yokoya, Xiao Xiang Zhu, and Antonio Plaza. 2017. Multisensor coupled spectral unmixing for time-series analysis. IEEE Transactions on Geoscience and Remote Sensing 55, 5 (2017), 2842–2857.

[43]

Hongyan Zhang, Liangpei Zhang, and Huanfeng Shen. 2012. A super-resolution reconstruction algorithm for hyperspectral images. Signal Processing 92, 9 (2012), 2082–2096.

Digital Library

[44]

Lei Zhang, Jiangtao Nie, Wei Wei, Yanning Zhang, Shengcai Liao, and Ling Shao. 2020. Unsupervised adaptation learning for hyperspectral imagery super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3073–3082.

[45]

Yongqiang Zhao, Jinxiang Yang, Qingyong Zhang, Lin Song, Yongmei Cheng, and Quan Pan. 2011. Hyperspectral imagery super-resolution by sparse representation and spectral regularization. EURASIP Journal on Advances in Signal Processing 2011, 1 (2011), 1–10.

[46]

Zhiyu Zhu, Junhui Hou, Jie Chen, Huanqiang Zeng, and Jiantao Zhou. 2020. Residual component estimating CNN for image super-resolution. IEEE Transactions on Image Processing 30 (2020), 1423–1428.

[47]

Changzhong Zou and Xusheng Huang. 2020. Hyperspectral image super-resolution combining with deep learning and spectral unmixing. Signal Processing: Image Communication 84 (2020), 115833.

Cited By

Rabari DMeghrajani YDesai L(2024)A Study of Secret Image Sharing Schemes Using Visual and Quantum CryptographyHarnessing Quantum Cryptography for Next-Generation Security Solutions10.4018/979-8-3693-9220-1.ch015(439-466)Online publication date: 27-Sep-2024
https://doi.org/10.4018/979-8-3693-9220-1.ch015
Demetriou DLockhat ZBrzozowski LSaini KDlamini ZHull R(2024)The Convergence of Radiology and Genomics: Advancing Breast Cancer Diagnosis with RadiogenomicsCancers10.3390/cancers1605107616:5(1076)Online publication date: 6-Mar-2024
https://doi.org/10.3390/cancers16051076
Liu CChen YZhu MHao CLi HWang X(2024)DEGAN: Detail-Enhanced Generative Adversarial Network for Monocular Depth-Based 3D ReconstructionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369082620:12(1-17)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1145/3690826
Show More Cited By

Index Terms

Deep Self-Supervised Hyperspectral Image Reconstruction
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Learning latent representations
        Deep belief networks

Recommendations

Self-supervised deep learning for joint 3D low-dose PET/CT image denoising
Abstract
Deep learning (DL)-based denoising of low-dose positron emission tomography (LDPET) and low-dose computed tomography (LDCT) has been widely explored. However, previous methods have focused only on single modality denoising, neglecting the ...
Graphical abstract

Display Omitted
Highlights
- A two-stage self-supervised framework named MAsk-then-Cycle (MAC) is proposed to enable self-supervised denoising of LDPET/LDCT images. MAC includes a masked autoencoder-based pre-training stage and a self-supervised denoising training ...
Lightweight Hyperspectral Image Reconstruction Network with Deep Feature Hallucination
Computer Vision – ACCV 2022 Workshops
Abstract
Hyperspectral image reconstruction from a compressive snapshot is an dispensable step in the advanced hyperspectral imaging systems to solve the low spatial and/or temporal resolution issue. Most existing methods extensively exploit various hand-...
Hyperspectral Image Reconstruction for SD-CASSI Systems Based on Residual Attention Network
Algorithmic Aspects in Information and Management
Abstract
Hyperspectral images contain both spatial and spectral information, which can be utilized to material identification. Therefore, they find significant advantages in object detection. Hyperspectral images are also believed to play an important part ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 18, Issue 3s

October 2022

381 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3567476

Editor:
Abdulmotaleb El Saddik
Mohamed Bin Zayed University of Artificial Intelligence, UAE and University of Ottawa, Canada

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2022

Online AM: 23 August 2022

Accepted: 01 January 2022

Revised: 12 November 2021

Received: 05 July 2021

Published in TOMM Volume 18, Issue 3s

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
431
Total Downloads

Downloads (Last 12 months)180
Downloads (Last 6 weeks)20

Reflects downloads up to 12 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rabari DMeghrajani YDesai L(2024)A Study of Secret Image Sharing Schemes Using Visual and Quantum CryptographyHarnessing Quantum Cryptography for Next-Generation Security Solutions10.4018/979-8-3693-9220-1.ch015(439-466)Online publication date: 27-Sep-2024
https://doi.org/10.4018/979-8-3693-9220-1.ch015
Demetriou DLockhat ZBrzozowski LSaini KDlamini ZHull R(2024)The Convergence of Radiology and Genomics: Advancing Breast Cancer Diagnosis with RadiogenomicsCancers10.3390/cancers1605107616:5(1076)Online publication date: 6-Mar-2024
https://doi.org/10.3390/cancers16051076
Liu CChen YZhu MHao CLi HWang X(2024)DEGAN: Detail-Enhanced Generative Adversarial Network for Monocular Depth-Based 3D ReconstructionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/369082620:12(1-17)Online publication date: 30-Aug-2024
https://dl.acm.org/doi/10.1145/3690826
Rosh GB H PBoregowda LMitra KCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)R2SFD: Improving Single Image Reflection Removal using Semantic Feature DictionaryProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681450(10277-10286)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681450
Peng BSun LLei JLiu BShen HLi WHuang Q(2024)Self-Supervised Monocular Depth Estimation via Binocular Geometric Correlation LearningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366357020:8(1-19)Online publication date: 13-Jun-2024
https://dl.acm.org/doi/10.1145/3663570
Huang QLi PHuang YShuang FCai Y(2024)Region-Focused Network for Dense CaptioningACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364837020:6(1-20)Online publication date: 26-Mar-2024
https://dl.acm.org/doi/10.1145/3648370
He WLi ZWang HXu TWang ZHuai BYuan NChen E(2024)Multimodal Dialogue Systems via Capturing Context-aware Dependencies and Ordinal Information of Semantic ElementsACM Transactions on Intelligent Systems and Technology10.1145/364509915:3(1-25)Online publication date: 15-Apr-2024
https://dl.acm.org/doi/10.1145/3645099
Li YXiao FLiang WGui L(2024)Multiply Complementary Priors for Image Compressive Sensing Reconstruction in Impulsive NoiseACM Transactions on Multimedia Computing, Communications, and Applications10.1145/364303220:6(1-22)Online publication date: 8-Mar-2024
https://dl.acm.org/doi/10.1145/3643032
Qiu HLi HWu QShi HWang LMeng FXu L(2024)Learning Offset Probability Distribution for Accurate Object DetectionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/363721420:5(1-24)Online publication date: 22-Jan-2024
https://dl.acm.org/doi/10.1145/3637214
Ma JWang PKong DWang ZLiu JPei HZhao J(2024)Robust Visual Question Answering: Datasets, Methods, and Future ChallengesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336615446:8(5575-5594)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3366154
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents