[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ skip to main content
research-article

ResTransUnet: : An effective network combined with Transformer and U-Net for liver segmentation in CT scans

Published: 24 July 2024 Publication History

Abstract

Liver segmentation is a fundamental prerequisite for the diagnosis and surgical planning of hepatocellular carcinoma. Traditionally, the liver contour is drawn manually by radiologists using a slice-by-slice method. However, this process is time-consuming and error-prone, depending on the radiologist’s experience. In this paper, we propose a new end-to-end automatic liver segmentation framework, named ResTransUNet, which exploits the transformer’s ability to capture global context for remote interactions and spatial relationships, as well as the excellent performance of the original U-Net architecture. The main contribution of this paper lies in proposing a novel fusion network that combines Unet and Transformer architectures. In the encoding structure, a dual-path approach is utilized, where features are extracted separately using both convolutional neural networks (CNNs) and Transformer networks. Additionally, an effective feature enhancement unit is designed to transfer the global features extracted by the Transformer network to the CNN for feature enhancement. This model aims to address the drawbacks of traditional Unet-based methods, such as feature loss during encoding and poor capture of global features. Moreover, it avoids the disadvantages of pure Transformer models, which suffer from large parameter sizes and high computational complexity. The experimental results on the LiTS2017 dataset demonstrate remarkable performance for our proposed model, with Dice coefficients, volumetric overlap error (VOE), and relative volume difference (RVD) values for liver segmentation reaching 0.9535, 0.0804, and −0.0007, respectively. Furthermore, to further validate the model’s generalization capability, we conducted tests on the 3Dircadb, Chaos, and Sliver07 datasets. The experimental results demonstrate that the proposed method outperforms other closely related models with higher liver segmentation accuracy. In addition, significant improvements can be achieved by applying our method when handling liver segmentation with small and discontinuous liver regions, as well as blurred liver boundaries. The code is available at the website: https://github.com/Jouiry/ResTransUNet.

Highlights

ResTransUnet merges U-Net’s rapid learning with Transformer’s broad generalization.
An attention mechanism is proposed to fuse local features with global features.
Higher accuracy compared to the U-Net and Transformer-based methods.
Extensive experiments on datasets demonstrate ResTransUnet’s superior performance.

References

[1]
Altini Nicola, Prencipe Berardino, Cascarano Giacomo Donato, Brunetti Antonio, Brunetti Gioacchino, Triggiani Vito, Carnimeo Leonarda, Marino Francescomaria, Guerriero Andrea, Villani Laura, et al., Liver, kidney and spleen segmentation from CT scans and MRI with deep learning: A survey, Neurocomputing 490 (2022) 30–53.
[2]
Khan Rayyan Azam, Luo Yigang, Wu Fang-Xiang, Machine learning based liver disease diagnosis: A systematic review, Neurocomputing 468 (2022) 492–509.
[3]
Soler Luc, Delingette Hervé, Malandain Grégoire, Montagnat Johan, Ayache Nicholas, Koehl Christophe, Dourthe Olivier, Malassagne Benoit, Smith Michelle, Mutter Didier, et al., Fully automatic anatomical, pathological, and functional segmentation from CT scans for hepatic surgery, Comput. Aided Surg. 6 (3) (2001) 131–142.
[4]
Jan Hendrik Moltz, Lars Bornemann, Volker Dicken, H Peitgen, Segmentation of liver metastases in CT scans by adaptive thresholding and morphological processing, in: MICCAI Workshop, Vol. 41, (43) 2008, p. 195.
[5]
Lu Xiaoqi, Wu Jianshuai, Ren Xiaoying, Zhang Baohua, Li Yinhui, The study and application of the improved region growing algorithm for liver segmentation, Optik 125 (9) (2014) 2142–2147.
[6]
Al-Shaikhli Saif Dawood Salman, Yang Michael Ying, Rosenhahn Bodo, Automatic 3D liver segmentation using sparse representation of global and local image information via level set formulation, 2015, arXiv preprint arXiv:1508.01521.
[7]
Massoptier Laurent, Casciaro Sergio, A new fully automatic and robust algorithm for fast segmentation of liver tissue and tumors from CT scans, Eur. Radiol. 18 (8) (2008) 1658–1665.
[8]
Häme Yrjö, Liver tumor segmentation using implicit surface evolution, Midas J. (2008) 1–10.
[9]
Luo Suhuai, Jin Jesse S., Chalup Stephan K., Qian Guoyu, A liver segmentation algorithm based on wavelets and machine learning, in: 2009 International Conference on Computational Intelligence and Natural Computing, Vol. 2, IEEE, 2009, pp. 122–125.
[10]
Li Xuhui, Huang Cheng, Jia Fucang, Li Zongmin, Fang Chihua, Fan Yingfang, Automatic liver segmentation using statistical prior models and free-form deformation, in: International MICCAI Workshop on Medical Computer Vision, Springer, 2014, pp. 181–188.
[11]
Jonathan Long, Evan Shelhamer, Trevor Darrell, Fully convolutional networks for semantic segmentation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 3431–3440.
[12]
Ben-Cohen Avi, Diamant Idit, Klang Eyal, Amitai Michal, Greenspan Hayit, Fully convolutional network for liver segmentation and lesions detection, in: Deep Learning and Data Labeling for Medical Applications, Springer, 2016, pp. 77–85.
[13]
Ronneberger Olaf, Fischer Philipp, Brox Thomas, U-net: Convolutional networks for biomedical image segmentation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2015, pp. 234–241.
[14]
Zhou Zongwei, Rahman Siddiquee Md Mahfuzur, Tajbakhsh Nima, Liang Jianming, Unet++: A nested u-net architecture for medical image segmentation, in: Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, Springer, 2018, pp. 3–11.
[15]
Çiçek Özgün, Abdulkadir Ahmed, Lienkamp Soeren S, Brox Thomas, Ronneberger Olaf, 3D U-Net: learning dense volumetric segmentation from sparse annotation, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 424–432.
[16]
Milletari Fausto, Navab Nassir, Ahmadi Seyed-Ahmad, V-net: Fully convolutional neural networks for volumetric medical image segmentation, in: 2016 Fourth International Conference on 3D Vision, 3DV, IEEE, 2016, pp. 565–571.
[17]
Liu Huaxiang, Fu Youyao, Zhang Shiqing, Liu Jun, Wang Yong, Wang Guoyu, Fang Jiangxiong, GCHA-Net: Global context and hybrid attention network for automatic liver segmentation, Comput. Biol. Med. 152 (2023).
[18]
Jiang Linfeng, Ou Jiajie, Liu Ruihua, Zou Yangyang, Xie Ting, Xiao Hanguang, Bai Ting, RMAU-Net: Residual multi-scale attention U-Net for liver and tumor segmentation in CT images, Comput. Biol. Med. 158 (2023).
[19]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, Jian Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 770–778.
[20]
Christ Patrick Ferdinand, Elshaer Mohamed Ezzeldin A, Ettlinger Florian, Tatavarty Sunil, Bickel Marc, Bilic Patrick, Rempfler Markus, Armbruster Marco, Hofmann Felix, D’Anastasi Melvin, et al., Automatic liver and lesion segmentation in CT using cascaded fully convolutional neural networks and 3D conditional random fields, in: International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, 2016, pp. 415–423.
[21]
Kaluva Krishna Chaitanya, Khened Mahendra, Kori Avinash, Krishnamurthi Ganapathy, 2D-densely connected convolution neural networks for automatic liver and tumor segmentation, 2018, arXiv preprint arXiv:1802.02182.
[22]
Li Xiaomeng, Chen Hao, Qi Xiaojuan, Dou Qi, Fu Chi-Wing, Heng Pheng-Ann, H-DenseUNet: hybrid densely connected UNet for liver and tumor segmentation from CT volumes, IEEE Trans. Med. Imaging 37 (12) (2018) 2663–2674.
[23]
Han Xiao, Automatic liver lesion segmentation using a deep convolutional neural network method, 2017, arXiv preprint arXiv:1704.07239.
[24]
Vaswani Ashish, Shazeer Noam, Parmar Niki, Uszkoreit Jakob, Jones Llion, Gomez Aidan N, Kaiser Łukasz, Polosukhin Illia, Attention is all you need, Advances in Neural Information Processing Systems, vol. 30, 2017.
[25]
Chen Jieneng, Lu Yongyi, Yu Qihang, Luo Xiangde, Adeli Ehsan, Wang Yan, Lu Le, Yuille Alan L, Zhou Yuyin, Transunet: Transformers make strong encoders for medical image segmentation, 2021, arXiv preprint arXiv:2102.04306.
[26]
Cao Hu, Wang Yueyue, Chen Joy, Jiang Dongsheng, Zhang Xiaopeng, Tian Qi, Wang Manning, Swin-unet: Unet-like pure transformer for medical image segmentation, 2021, arXiv preprint arXiv:2105.05537.
[27]
Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, Daguang Xu, Unetr: Transformers for 3d medical image segmentation, in: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 574–584.
[28]
Xiao Xiao, Lian Shen, Luo Zhiming, Li Shaozi, Weighted res-unet for high-quality retina vessel segmentation, in: 2018 9th International Conference on Information Technology in Medicine and Education, ITME, IEEE, 2018, pp. 327–331.
[29]
Huang Huimin, Lin Lanfen, Tong Ruofeng, Hu Hongjie, Zhang Qiaowei, Iwamoto Yutaro, Han Xianhua, Chen Yen-Wei, Wu Jian, Unet 3+: A full-scale connected unet for medical image segmentation, in: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, IEEE, 2020, pp. 1055–1059.
[30]
Wang Jinke, Lv Peiqing, Wang Haiying, Shi Changfa, SAR-U-Net: squeeze-and-excitation block and atrous spatial pyramid pooling based residual U-Net for automatic liver segmentation in computed tomography, Comput. Methods Programs Biomed. 208 (2021).
[31]
Li Jian, Liu Kongyu, Hu Yating, Zhang Hongchen, Heidari Ali Asghar, Chen Huiling, Zhang Weijiang, Algarni Abeer D, Elmannai Hela, Eres-UNet++: Liver CT image segmentation based on high-efficiency channel attention and Res-UNet++, Comput. Biol. Med. 158 (2023).
[32]
Devlin Jacob, Chang Ming-Wei, Lee Kenton, Toutanova Kristina, Bert: Pre-training of deep bidirectional transformers for language understanding, 2018, arXiv preprint arXiv:1810.04805.
[33]
Dosovitskiy Alexey, Beyer Lucas, Kolesnikov Alexander, Weissenborn Dirk, Zhai Xiaohua, Unterthiner Thomas, Dehghani Mostafa, Minderer Matthias, Heigold Georg, Gelly Sylvain, et al., An image is worth 16x16 words: Transformers for image recognition at scale, 2020, arXiv preprint arXiv:2010.11929.
[34]
Touvron Hugo, Cord Matthieu, Douze Matthijs, Massa Francisco, Sablayrolles Alexandre, Jégou Hervé, Training data-efficient image transformers & distillation through attention, in: International Conference on Machine Learning, PMLR, 2021, pp. 10347–10357.
[35]
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao, Pyramid vision transformer: A versatile backbone for dense prediction without convolutions, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 568–578.
[36]
Han Kai, Xiao An, Wu Enhua, Guo Jianyuan, Xu Chunjing, Wang Yunhe, Transformer in transformer, Adv. Neural Inf. Process. Syst. 34 (2021) 15908–15919.
[37]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, Baining Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
[38]
Jie Hu, Li Shen, Gang Sun, Squeeze-and-excitation networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7132–7141.
[39]
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, Tat-Seng Chua, Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 5659–5667.
[40]
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang, Cheng Li, Honggang Zhang, Xiaogang Wang, Xiaoou Tang, Residual attention network for image classification, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 3156–3164.
[41]
Bilic Patrick, Christ Patrick, Li Hongwei Bran, Vorontsov Eugene, et al., The liver tumor segmentation benchmark (LiTS), Med. Image Anal. 84 (2023).
[42]
Kavur A. Emre, Gezer N. Sinem, ş Mustafa Barı, Aslan Sinem, Conze Pierre-Henri, et al., CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation, Med. Image Anal. 69 (2021).
[43]
Heimann Tobias, Van Ginneken Bram, Styner Martin A, Arzhaeva Yulia, Aurich Volker, Bauer Christian, Beck Andreas, Becker Christoph, Beichel Reinhard, Bekes György, et al., Comparison and evaluation of methods for liver segmentation from CT datasets, IEEE Trans. Med. Imaging 28 (8) (2009) 1251–1265.
[44]
Ma Jun, Zhang Yao, Gu Song, Zhu Cheng, Ge Cheng, Zhang Yichi, An Xingle, Wang Congcong, Wang Qiyuan, Liu Xin, et al., Abdomenct-1k: Is abdominal organ segmentation a solved problem?, IEEE Trans. Pattern Anal. Mach. Intell. 44 (10) (2021) 6695–6714.

Recommendations

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Computers in Biology and Medicine
Computers in Biology and Medicine  Volume 177, Issue C
Jul 2024
720 pages

Publisher

Pergamon Press, Inc.

United States

Publication History

Published: 24 July 2024

Author Tags

  1. Liver segmentation
  2. Transformers
  3. Deep learning
  4. Medical imaging processing

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • 0
    Total Citations
  • 0
    Total Downloads
  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 06 Jan 2025

Other Metrics

Citations

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media