MTSC2: Multi-task Survival Prediction on Colorectal CT Images with Clinical Information
Abstract
Colorectal cancer (CRC) is a highly malignant tumor, and accurate survival prediction is crucial in determining the most suitable treatment options. Recent studies have demonstrated that deep learning-based survival models have the potential to outperform traditional radiomics-based models. However, the complexity and variation of the colorectal focal area make it challenging for current deep-learning models to effectively capture the survival prediction-related information. To tackle this challenge, we propose an end-to-end multi-task survival prediction model that incorporates clinical information and segments the region of interest (ROI) of preoperative computed tomography (CT) scans as an auxiliary task. Our model achieves a concordance index of 0.7275 that outperforms other existing methods for CRC survival prediction on our in-house dataset and can help improve the accuracy and efficiency of survival prediction for patients with colorectal cancer.
References
[1]
Amine Amyar, Romain Modzelewski, Hua Li, and Su Ruan. 2020. Multi-task deep learning based CT imaging analysis for COVID-19 pneumonia: Classification and segmentation. Computers in Biology and Medicine 126 (Nov. 2020), 104037.
[2]
Vincent Andrearczyk, Pierre Fontaine, Valentin Oreiller, Joel Castelli, Mario Jreige, John O. Prior, and Adrien Depeursinge. 2021. Multi-task Deep Segmentation and Radiomics for Automatic Prognosis in Head and Neck Cancer. In Predictive Intelligence in Medicine(Lecture Notes in Computer Science), Islem Rekik, Ehsan Adeli, Sang Hyun Park, and Julia Schnabel (Eds.). Springer International Publishing, Cham, 147–156.
[3]
Du Cai, Xin Duan, Wei Wang, Ze-Ping Huang, Qiqi Zhu, Min-Er Zhong, Min-Yi Lv, Cheng-Hang Li, Wei-Bin Kou, Xiao-Jian Wu, and Feng Gao. 2021. A Metabolism-Related Radiomics Signature for Predicting the Prognosis of Colorectal Cancer. Frontiers in Molecular Biosciences 7 (Jan. 2021), 613918.
[4]
Travers Ching, Xun Zhu, and Lana X. Garmire. 2018. Cox-nnet: An artificial neural network method for prognosis prediction of high-throughput omics data. PLOS Computational Biology 14, 4 (April 2018), e1006076.
[5]
Robert J. Gillies, Paul E. Kinahan, and Hedvig Hricak. 2016. Radiomics: Images Are More than Pictures, They Are Data. Radiology 278, 2 (Feb. 2016), 563–577. Publisher: Radiological Society of North America.
[6]
Bingxin Gu, Mingyuan Meng, Mingzhen Xu, David Dagan Feng, Lei Bi, Jinman Kim, and Shaoli Song. 2023. Multi-task deep learning-based radiomic nomogram for prognostic prediction in locoregionally advanced nasopharyngeal carcinoma. European Journal of Nuclear Medicine and Molecular Imaging (Aug. 2023).
[7]
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep Residual Learning for Image Recognition. arXiv:1512.03385 [cs].
[8]
Natally Horvat. 2019. Novel imaging techniques of rectal cancer: what do radiomics and radiogenomics have to offer? A literature review. Abdominal Radiology (2019), 11.
[9]
Jie Hu, Li Shen, and Gang Sun. 2018. Squeeze-and-Excitation Networks. 7132–7141.
[10]
Shih-Cheng Huang, Tanay Kothari, Imon Banerjee, Chris Chute, Robyn L. Ball, Norah Borus, Andrew Huang, Bhavik N. Patel, Pranav Rajpurkar, Jeremy Irvin, Jared Dunnmon, Joseph Bledsoe, Katie Shpanskaya, Abhay Dhaliwal, Roham Zamanian, Andrew Y. Ng, and Matthew P. Lungren. 2020. PENet—a scalable deep-learning model for automated diagnosis of pulmonary embolism using volumetric CT imaging. npj Digital Medicine 3, 1 (Dec. 2020), 61.
[11]
Cheng Jin, Heng Yu, Jia Ke, Peirong Ding, Yongju Yi, Xiaofeng Jiang, Xin Duan, Jinghua Tang, Daniel T. Chang, Xiaojian Wu, Feng Gao, and Ruijiang Li. 2021. Predicting treatment response from longitudinal images using multi-task deep learning. Nature Communications 12, 1 (Dec. 2021), 1851.
[12]
Jared L. Katzman, Uri Shaham, Alexander Cloninger, Jonathan Bates, Tingting Jiang, and Yuval Kluger. 2018. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Medical Research Methodology 18 (Feb. 2018), 24.
[13]
Hyungjin Kim, Jin Mo Goo, Kyung Hee Lee, Young Tae Kim, and Chang Min Park. 2020. Preoperative CT-based Deep Learning Model for Predicting Disease-Free Survival in Patients with Lung Adenocarcinomas. Radiology 296, 1 (July 2020), 216–224.
[14]
Xuehua Li, Dong Liang, Jixin Meng, Jie Zhou, Zhao Chen, Siyun Huang, Baolan Lu, Yun Qiu, Mark E. Baker, Ziyin Ye, Qinghua Cao, Mingyu Wang, Chenglang Yuan, Zhihui Chen, Shengyu Feng, Yuxuan Zhang, Marietta Iacucci, Subrata Ghosh, Florian Rieder, Canhui Sun, Minhu Chen, Ziping Li, Ren Mao, Bingsheng Huang, and Shi-Ting Feng. 2021. Development and Validation of a Novel Computed-Tomography Enterography Radiomic Approach for Characterization of Intestinal Fibrosis in Crohn’s Disease. Gastroenterology 160, 7 (June 2021), 2303–2316.e11.
[15]
Ting Lin, Jinhai Mai, Meng Yan, Zhenhui Li, Xianyue Quan, and Xin Chen. 2021. A Nomogram Based on CT Deep Learning Signature: A Potential Tool for the Prediction of Overall Survival in Resected Non-Small Cell Lung Cancer Patients. Cancer Management and Research Volume 13 (March 2021), 2897–2906.
[16]
Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, Furu Wei, and Baining Guo. 2022. Swin Transformer V2: Scaling Up Capacity and Resolution. arXiv:2111.09883 [cs].
[17]
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. arXiv:2103.14030 [cs].
[18]
Bin Lou, Semihcan Doken, Tingliang Zhuang, Danielle Wingerter, Mishka Gidwani, Nilesh Mistry, Lance Ladic, Ali Kamen, and Mohamed E Abazeed. 2019. An image-based deep learning framework for individualising radiotherapy dose: a retrospective analysis of outcome prediction. The Lancet Digital Health 1, 3 (July 2019), e136–e147.
[19]
Chubin Ou, Sitong Zhou, Ronghua Yang, Weili Jiang, Haoyang He, Wenjun Gan, Wentao Chen, Xinchi Qin, Wei Luo, Xiaobing Pi, and Jiehua Li. 2022. A deep learning based multimodal fusion model for skin lesion diagnosis using smartphone collected clinical images and metadata. Frontiers in Surgery 9 (2022).
[20]
Ishak Pacal, Dervis Karaboga, Alper Basturk, Bahriye Akay, and Ufuk Nalbantoglu. 2020. A comprehensive review of deep learning in colon cancer. Computers in Biology and Medicine 126 (Nov. 2020), 104003.
[21]
Andre G. C. Pacheco and Renato A. Krohling. 2021. An Attention-Based Mechanism to Combine Images and Metadata in Deep Learning Models Applied to Skin Cancer Classification. IEEE Journal of Biomedical and Health Informatics 25, 9 (Sept. 2021), 3554–3563.
[22]
Nahian Siddique, Sidike Paheding, Colin P. Elkin, and Vijay Devabhaktuni. 2021. U-Net and Its Variants for Medical Image Segmentation: A Review of Theory and Applications. IEEE Access 9 (2021), 82031–82057. Conference Name: IEEE Access.
[23]
Rebecca L. Siegel, Kimberly D. Miller, Ann Goding Sauer, Stacey A. Fedewa, Lynn F. Butterly, Joseph C. Anderson, Andrea Cercek, Robert A. Smith, and Ahmedin Jemal. 2020. Colorectal cancer statistics, 2020. CA: A Cancer Journal for Clinicians 70, 3 (2020), 145–164. _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.3322/caac.21601.
[24]
Carole H. Sudre, Wenqi Li, Tom Vercauteren, Sébastien Ourselin, and M. Jorge Cardoso. 2017. Generalised Dice overlap as a deep learning loss function for highly unbalanced segmentations. Vol. 10553. 240–248. arXiv:1707.03237 [cs].
[25]
Maribel Torres-Velázquez, Wei-Jie Chen, Xue Li, and Alan B. McMillan. 2021. Application and Construction of Deep Learning Networks in Medical Imaging. IEEE Transactions on Radiation and Plasma Medical Sciences 5, 2 (March 2021), 137–159. Conference Name: IEEE Transactions on Radiation and Plasma Medical Sciences.
[26]
Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. 2023. ConvNeXt V2: Co-Designing and Scaling ConvNets With Masked Autoencoders. 16133–16142.
[27]
Peng Xu, Xiatian Zhu, and David A. Clifton. 2023. Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023), 1–20. Conference Name: IEEE Transactions on Pattern Analysis and Machine Intelligence.
[28]
Yu Zhang and Qiang Yang. 2022. A Survey on Multi-Task Learning. IEEE Transactions on Knowledge and Data Engineering 34, 12 (Dec. 2022), 5586–5609. Conference Name: IEEE Transactions on Knowledge and Data Engineering.
[29]
Yan Zhao, Xiuying Wang, Tongtong Che, Guoqing Bao, and Shuyu Li. 2023. Multi-task deep learning for medical image computing and analysis: A review. Computers in Biology and Medicine 153 (Feb. 2023), 106496.
[30]
Hanci Zheng, Zongying Lin, Qizheng Zhou, Xingchen Peng, Jianghong Xiao, Chen Zu, Zhengyang Jiao, and Yan Wang. 2022. Multi-transSP: Multimodal Transformer for Survival Prediction of Nasopharyngeal Carcinoma Patients. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2022, Linwei Wang, Qi Dou, P. Thomas Fletcher, Stefanie Speidel, and Shuo Li (Eds.). Vol. 13437. Springer Nature Switzerland, Cham, 234–243. Series Title: Lecture Notes in Computer Science.
Index Terms
- MTSC2: Multi-task Survival Prediction on Colorectal CT Images with Clinical Information
Recommendations
3-D Segmentation Algorithm of Small Lung Nodules in Spiral CT Images
Computed tomography (CT) is the most sensitive imaging technique for detecting lung nodules, and is now being evaluated as a screening tool for lung cancer in several large samples studies all over the world. In this report, we describe a semiautomatic ...
Pancreatic Cancer Survival Prediction Using CT Scans and Clinical Variables
Simulation, Image Processing, and Ultrasound Systems for Assisted Diagnosis and NavigationAbstractPancreatic cancer resulted in 411,600 deaths globally in 2015. Pancreatic ductal adenocarcinoma (PDAC) is the most common type of pancreatic cancer and it is highly lethal. Survival for patients with PDAC is dismal due to its aggressive nature, ...
A Multi-Scale and Multi-Resolution Approach for Liver Tumor Segmentation in CT Scans
ICBIP '21: Proceedings of the 6th International Conference on Biomedical Signal and Image ProcessingHepatocellular carcinoma currently causes over 800 000 fatalities per year worldwide – and the number of cases is increasing. An early diagnosis and treatment play a crucial role in saving patients’ lives. The purpose of this study is the exploration of ...
Comments
Please enable JavaScript to view thecomments powered by Disqus.Information & Contributors
Information
Published In
March 2024
478 pages
ISBN:9798400716416
DOI:10.1145/3654823
Copyright © 2024 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 May 2024
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
CACML 2024
CACML 2024: 2024 3rd Asia Conference on Algorithms, Computing and Machine Learning
March 22 - 24, 2024
Shanghai, China
Acceptance Rates
Overall Acceptance Rate 93 of 241 submissions, 39%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 21Total Downloads
- Downloads (Last 12 months)21
- Downloads (Last 6 weeks)8
Reflects downloads up to 01 Mar 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format