More Web Proxy on the site http://driver.im/

research-article

Window-based transformer generative adversarial network for autonomous underwater image enhancement

Authors:

Fayaz Ali Dharejo,

Md. Jalil Piran,

Sajid JavedAuthors Info & Claims

Volume 126, Issue PD

https://doi.org/10.1016/j.engappai.2023.107069

Published: 27 February 2024 Publication History

Abstract

The estimation of high-quality underwater images is an important step towards the development of computer vision systems in marine environments. This fundamental step contains numerous computer vision and robotics applications including marine exploration, robotics manipulation, navigation, object detection, tracking, and sea life monitoring. However, this pre-processing step becomes more challenging in the presence of back-scattering of underwater particles and attenuation issues, which lead to the formation of hazy underwater images. Vision transformers have recently demonstrated outstanding performance in many computer vision applications. Window-based Transformers (WT) show promising enhancement performance by computing self-attention within non-overlapping local windows. WT has been identified as an essential component in improving representation capabilities; however, it has received less attention in improving the performance of Underwater Image Enhancement (UIE). Therefore, we propose a novel end-to-end Underwater window-based Transformer Generative Adversarial Network (UwTGAN). Our proposed algorithm consists of two main components, including a transformer generator that generates a restored underwater image and a transformer discriminator that classifies the generated underwater image. Both components are equipped with Window-based Self-Attention Blocks (WSABs), which maximize efficiency by limiting self-attention computation to non-overlapping local windows and provide relatively low computational costs. WSAB-based transformer generators and discriminators are trained end-to-end. We formulated an efficient loss function to ensure that the variables are closely integrated. Extensive experimental evaluations are performed on four independent underwater image datasets. Our results demonstrate that the proposed UwTGAN algorithm has outperformed several state-of-the-art UIE methods in terms of both quantitative and qualitative metrics by a significant margin.

References

[1]

Akkaynak Derya, Treibitz Tali, Sea-Thru: A method for removing water from underwater images, in: CVPR, 2019, pp. 1682–1691.

[2]

Akkaynak Derya, Treibitz Tali, Shlesinger Tom, Loya Yossi, Tamir Raz, Iluz David, What is the space of attenuation coefficients in underwater computer vision?, in: CVPR, 2017, pp. 568–577.

[3]

Alawode Basit, Guo Yuhang, Ummar Mehnaz, Werghi Naoufel, Dias Jorge, Mian Ajmal, Javed Sajid, UTB180: A high-quality benchmark for underwater tracking, in: ACCV, 2022.

[4]

Albitar Houssam, Dandan Kinan, Ananiev Anani, Kalaykov Ivan, Underwater robotics: surface cleaning technics, adhesion and locomotion systems, IJARS 13 (1) (2016) 7.

[5]

Aldhaheri Sara, De Masi Giulia, Pairet Èric, Ardón Paola, Underwater robot manipulation: advances, challenges and prospective ventures, in: IEEE OCEANS, 2022.

[6]

Ancuti Codruta O., Ancuti Cosmin, De Vleeschouwer Christophe, Garcia Rafael, Locally adaptive color correction for underwater image dehazing and matching, in: CVPRW, 2017, pp. 997–1005.

[7]

Ancuti Cosmin, Ancuti Codruta Orniana, Haber Tom, Bekaert Philippe, Enhancing underwater images and videos by fusion, in: IEEE CVPR, 2012, pp. 81–88.

[8]

Anwar Saeed, Li Chongyi, Diving deeper into underwater image enhancement: A survey, SPIC 89 (2020).

[9]

Anwar Saeed, Li Chongyi, Porikli Fatih, Deep underwater image enhancement, arXiv (2018).

[10]

Aydin, Tunç Ozan, Mantiuk, Rafał, Myszkowski, Karol, Seidel, Hans-Peter, 2008. Dynamic Range Independent Image Quality Assessment, 27, (3), 1–10.

[11]

Blue robotics - underwater ROVs, thrusters, sonars, and cameras, 2020, https://www.bluerobotics.com/, Accessed: 2023-3-20.

[12]

Boudiaf Abderrahmene, Guo Yuhang, Ghimire Adarsh, Werghi Naoufel, De Masi Giulia, Javed Sajid, Dias Jorge, Underwater image enhancement using pre-trained transformer, in: ICIAP, 2022.

[13]

Buongiorno Nardelli Bruno, Cavaliere Davide, Charles Elodie, Ciani Daniele, Super-resolving ocean dynamics from space with computer vision algorithms, Remote Sens. 14 (5) (2022) 1159.

[14]

Chen Ting, Kornblith Simon, Norouzi Mohammad, Hinton Geoffrey, A simple framework for contrastive learning of visual representations, in: ICML, JMLR, 2020.

[15]

Deng, Zhuo, Cai, Yuanhao, Chen, Lu, Gong, Zheng, Bao, Qiqi, Yao, Xue, Fang, Dong, Yang, Wenming, Zhang, Shaochong, Ma, Lan, Rformer: Transformer-based generative adversarial network for real fundus image restoration on a new clinical benchmark.

[16]

Deng Jia, Dong Wei, Socher Richard, Li Li-Jia, Li Kai, Fei-Fei Li, ImageNet: A large-scale hierarchical image database, in: IEEE CVPR, 2009, pp. 248–255.

[17]

Desai Chaitra, Reddy Badduri Sai Sudheer, Tabib Ramesh Ashok, Patil Ujwala, Mudenagudi Uma, AquaGAN: Restoration of underwater images, in: IEEE CVPR, 2022, pp. 296–304.

[18]

Dharejo Fayaz Ali, Zhou Yuanchun, Deeba Farah, Du Yi, A color enhancement scene estimation approach for single image haze removal, IEEE GRSL 17 (9) (2019) 1613–1617.

[19]

Dosovitskiy Alexey, Beyer Lucas, Kolesnikov Alexander, Weissenborn Dirk, Zhai Xiaohua, Unterthiner Thomas, Dehghani Mostafa, Minderer Matthias, Heigold Georg, Gelly Sylvain, Uszkoreit Jakob, Houlsby Neil, An image is worth 16x16 words: Transformers for image recognition at scale, in: ICLR, 2021.

[20]

Dosovitskiy Alexey, Beyer Lucas, Kolesnikov Alexander, Weissenborn Dirk, Zhai Xiaohua, Unterthiner Thomas, Dehghani Mostafa, Minderer Matthias, Heigold Georg, Gelly Sylvain, et al., An image is worth 16x16 words: Transformers for image recognition at scale, ICLR (2021).

[21]

Dudek Gregory, Giguere Philippe, Prahacs Chris, Saunderson Shane, Sattar Junaed, Torres-Mendez Luz-Abril, Jenkin Michael, German Andrew, Hogue Andrew, Ripsman Arlene, Zacher Jim, Milios Evangelos, Liu Hui, Zhang Pifu, Buehler Marti, Georgiades Christina, AQUA: An amphibious autonomous robot, Computer (Long Beach Calif.) 40 (1) (2007) 46–53.

[22]

Fabbri Cameron, Islam Md Jahidul, Sattar Junaed, Enhancing underwater imagery using generative adversarial networks, in: ICRA, IEEE, 2018, pp. 7159–7165.

[23]

Fayaz Sheezan, Parah Shabir A., Qureshi G.J., Underwater object detection: architectures and algorithms–a comprehensive review, MIA 81 (15) (2022) 20871–20916.

[24]

Fu Xueyang, Fan Zhiwen, Ling Mei, Huang Yue, Ding Xinghao, Two-step approach for single underwater image enhancement, in: ISPACS, 2017, pp. 789–794.

[25]

Fu Xueyang, Zhuang Peixian, Huang Yue, Liao Yinghao, Zhang Xiao-Ping, Ding Xinghao, A retinex-based enhancing approach for single underwater image, in: IEEE ICIP, 2014, pp. 4572–4576.

[26]

Galdran Adrian, Pardo David, Picón Artzai, Alvarez-Gila Aitor, Automatic red-channel underwater image restoration, JVCIR 26 (2015) 132–145.

[27]

Goodfellow Ian, Pouget-Abadie Jean, Mirza Mehdi, Xu Bing, Warde-Farley David, Ozair Sherjil, Courville Aaron, Bengio Yoshua, Generative adversarial networks, Commun. ACM 63 (11) (2020).

[28]

https://gopro.com/, Accessed: 2023-3-20.

[29]

Guo Yecai, Li Hanyu, Zhuang Peixian, Underwater image enhancement using a multiscale dense generative adversarial network, IEEE JCE 45 (3) (2019) 862–870.

[30]

Han Junlin, Shoeiby Mehrdad, Malthus Tim, Botha Elizabeth, Anstee Janet, Anwar Saeed, Wei Ran, Armin Mohammad Ali, Li Honngdong, Petersson Lars, Underwater image restoration via contrastive learning and a real-world dataset, Remote Sens. (2022).

[31]

He Kaiming, Fan Haoqi, Wu Yuxin, Xie Saining, Girshick Ross, Momentum contrast for unsupervised visual representation learning, in: CVPR, IEEE, 2020.

[32]

Islam, Md Jahidul, Luo, Peigen, Sattar, Junaed, 2020a. Simultaneous Enhancement and Super-Resolution of Underwater Imagery for Improved Visual Perception. In: RSS. Corvalis, Oregon, USA.

[33]

Islam Md Jahidul, Xia Youya, Sattar Junaed, Fast underwater image enhancement for improved visual perception, IEEE RA-L 5 (2) (2020) 3227–3234.

[34]

Isola Phillip, Zhu Jun-Yan, Zhou Tinghui, Efros Alexei, Image-to-image translation with conditional adversarial networks, in: CVPR, 2017.

[35]

Jiang Yifan, Chang Shiyu, Wang Zhangyang, Transgan: Two pure transformers can make one strong gan, and that can scale up, NIPS 34 (2021).

[36]

Johnson Justin, Alahi Alexandre, Fei-Fei Li, Perceptual losses for real-time style transfer and super-resolution, in: ECCV, 2016.

[37]

Kingma Diederik P., Ba Jimmy, Adam: A method for stochastic optimization, in: ICLR, 2015, URL http://arxiv.org/abs/1412.6980.

[38]

Li Chongyi, Anwar Saeed, Porikli Fatih, Underwater scene prior inspired deep underwater image and video enhancement, Pattern Recognit. 98 (2020).

Digital Library

[39]

Li Chongyi, Guo Chunle, Ren Wenqi, Cong Runmin, Hou Junhui, Kwong Sam, Tao Dacheng, An underwater image enhancement benchmark dataset and beyond, IEEE TIP 29 (2019) 4376–4389.

[40]

Li Chongyi, Guo Chunle, Ren Wenqi, Cong Runmin, Hou Junhui, Kwong Sam, Tao Dacheng, An underwater image enhancement benchmark dataset and beyond, IEEE TIP 29 (2020) 4376–4389.

[41]

Li Jie, Skinner Katherine A., Eustice Ryan, Johnson-Roberson M., WaterGAN: Unsupervised generative network to enable real-time color correction of monocular underwater images, RA-L (2017).

[42]

Liu Ze, Lin Yutong, Cao Yue, Hu Han, Wei Yixuan, Zhang Zheng, Lin Stephen, Guo Baining, Swin transformer: Hierarchical vision transformer using shifted windows, in: CVF, 2021.

[43]

Loshchilov Ilya, Hutter Frank, SGDR: Stochastic gradient descent with warm restarts, in: ICLR, 2017, URL https://openreview.net/forum?id=Skq89Scxx.

[44]

Lyu Zhangkai, Peng Andrew, Wang Qingwei, Ding Dandan, An efficient learning-based method for underwater image enhancement, Displays 74 (2022).

[45]

Mallet Delphine, Pelletier Dominique, Underwater video techniques for observing coastal marine biodiversity: a review of sixty years of publications (1952–2012), Fish. Res. 154 (2014) 44–62.

[46]

Maniyath Shima Ramesh, Vijayakumar K, Singh Laxman, Sharma Sudhir Kumar, Olabiyisi Tunde, Learning-based approach to underwater image dehazing using CycleGAN, Arab. J. Geosci. 14 (18) (2021).

[47]

https://www.openrov.com/, Accessed: 2023-3-20.

[48]

Panetta Karen, Gao Chen, Agaian Sos, Human-visual-system-inspired underwater image quality measures, JOE 41 (3) (2016).

[49]

Panetta Karen, Kezebou Landry, Oludare Victor, Agaian Sos, Comprehensive underwater object tracking benchmark dataset and underwater image enhancement with GAN, JOE 47 (1) (2021) 59–75.

[50]

Peng Yan-Tsung, Cosman Pamela C., Underwater image restoration based on image blurriness and light absorption, TIP 26 (4) (2017).

[51]

Peng Lintao, Zhu Chunli, Bian Liheng, U-shape transformer for underwater image enhancement, in: ECCV, 2023, pp. 290–307.

[52]

Risholm Petter, Ivarsen Peter Ørnulf, Haugholt Karl Henrik, Mohammed Ahmed, Underwater marker-based pose-estimation with associated uncertainty, in: ICCV, 2021.

[53]

Salmond J, Passenger J, Kovacs E, Roelfsema C, Stetner D, Reef Check Australia 2018 Heron Island Reef Health Report, Reef Check Foundation Ltd, 2018.

[54]

Treibitz Tali, Schechner Yoav Y., Active polarization descattering, IEEE T-PAMI 31 (3) (2009) 385–399.

[55]

Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N, Kaiser Łukasz, Polosukhin Illia, Attention is all you need, NIPS 30 (2017).

[56]

Wang Zhou, Bovik A.C., Sheikh H.R., Simoncelli E.P., Image quality assessment: from error visibility to structural similarity, IEEE TIP 13 (4) (2004) 600–612.

Digital Library

[57]

Wang Yudong, Guo Jichang, Gao Huan, Yue Huihui, Uieĉ 2-Net: CNN-based underwater image enhancement using two color space, SPIC 96 (2021).

[58]

Wang Yang, Zhang Jing, Cao Yang, Wang Zengfu, A deep CNN method for underwater image enhancement, in: IEEE ICIP, 2017.

[59]

Wang Nan, Zhou Yabin, Han Fenglei, Zhu Haitao, Yao Jingzheng, UWGAN: underwater GAN for real-world underwater color restoration and dehazing, arXiv (2019).

[60]

Wu Junjun, Liu Xilin, Lu Qinghua, Lin Zeqin, Qin Ningwei, Shi Qingwu, FW-GAN: Underwater image enhancement using generative adversarial network with multi-scale fusion, Signal Process., Image Commun. 109 (2022).

[61]

Wu Yinghao, Ta Xuxiang, Xiao Ruichao, Wei Yaoguang, An Dong, Li Daoliang, Survey of underwater robot positioning navigation, Appl. Ocean Res. 90 (2019).

[62]

Yang Miao, Hu Ke, Du Yixiang, Wei Zhiqiang, Sheng Zhibin, Hu Jintong, Underwater image enhancement based on conditional generative adversarial network, SPIC 81 (2020).

[63]

Yang Miao, Sowmya Arcot, An underwater color image quality evaluation metric, IEEE TIP 24 (12) (2015) 6062–6071.

[64]

Zamir Syed Waqas, Arora Aditya, Khan Salman, Hayat Munawar, Khan Fahad Shahbaz, Yang Ming-Hsuan, Restormer: Efficient transformer for high-resolution image restoration, in: CVPR, 2022.

[65]

Zhang Yinghui, Liu Tingshuai, Zhao Bo, Ge Fengxiang, SFA-GAN: structure–frequency-aware generative adversarial network for underwater image enhancement, Signal, Image and Video Processing (2023).

[66]

Zhao, Hengshuang, Jiang, Li, Jia, Jiaya, Torr, Philip HS, Koltun, Vladlen, Point transformer. IEEE CVF.

[67]

Zhu Jun-Yan, Park Taesung, Isola Phillip, Efros Alexei A, Unpaired image-to-image translation using cycle-consistent adversarial networks, in: IEEE ICCV, 2017.

Cited By

Teng SLiu AChen BWang JWu ZFu J(2024)Unsupervised learning method for underwater concrete crack image enhancement and augmentation based on cross domain translation strategyEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108884136:PAOnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108884

Index Terms

Window-based transformer generative adversarial network for autonomous underwater image enhancement
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Underwater image enhancement based on conditional generative adversarial network
Abstract
Underwater images play an essential role in acquiring and understanding underwater information. High-quality underwater images can guarantee the reliability of underwater intelligent systems. Unfortunately, underwater images are ...
U-shape Transformer for Underwater Image Enhancement
Computer Vision – ECCV 2022 Workshops
Abstract
The light absorption and scattering of underwater impurities lead to poor underwater imaging quality. The existing data-driven based underwater image enhancement (UIE) techniques suffer from the lack of a large-scale dataset containing various ...
A two-stage and two-branch generative adversarial network-based underwater image enhancement
Abstract
Due to the absorption and scattering of light under the water, various distortions are formed in the underwater images, which seriously limit the underwater visual task. To address this problem, a two-stage and two-branch underwater image ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Engineering Applications of Artificial Intelligence

Engineering Applications of Artificial Intelligence Volume 126, Issue PD

Nov 2023

1653 pages

ISSN:0952-1976

Issue’s Table of Contents

Elsevier Ltd.

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 27 February 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Teng SLiu AChen BWang JWu ZFu J(2024)Unsupervised learning method for underwater concrete crack image enhancement and augmentation based on cross domain translation strategyEngineering Applications of Artificial Intelligence10.1016/j.engappai.2024.108884136:PAOnline publication date: 1-Oct-2024
https://dl.acm.org/doi/10.1016/j.engappai.2024.108884

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents