Abstract
Lung cancer is the deadliest cancer worldwide. This fact has led to increased development of medical and computational methods to improve early diagnosis, aiming at reducing its fatality rate. Radiologists conduct lung cancer screening and diagnosis by localizing and characterizing pathologies. Therefore, there is an inherent relationship between visual clinical findings and spatial location in the images. However, in previous work, this spatial relationship between multimodal data has not been exploited. In this work, we propose a Spatially-Aware Multimodal Network with Attention (SAMA) for early lung cancer diagnosis. Our approach takes advantage of the spatial relationship between visual and clinical information, emulating the diagnostic process of the specialist. Specifically, we propose a multimodal fusion module composed of dynamic filtering of visual features with clinical data followed by a channel attention mechanism. We provide empirical evidence of the potential of SAMA to integrate spatially visual and clinical information. Our method outperforms by 14.3% the state-of-the-art method in the LUng CAncer Screening with Multimodal Biomarkers Dataset.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Ardila, D., et al.: End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nat. Med. 25(6), 954–961 (2019)
Astaraki, M., Toma-Dasu, I., Smedby, Ö., Wang, C.: Normal appearance autoencoder for lung cancer detection and segmentation. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., Khan, A. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 249–256. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_28
Daza, L., Castillo, A., Escobar, M., Valencia, S., Pinzón, B., Arbeláez, P.: LUCAS: lung cancer screening with multimodal biomarkers. In: Syeda-Mahmood, T., Drechsler, K., Greenspan, H., Madabhushi, A., Karargyris, A., Linguraru, M.G., Oyarzun Laura, C., Shekhar, R., Wesarg, S., González Ballester, M.Á., Erdt, M. (eds.) CLIP/ML-CDS -2020. LNCS, vol. 12445, pp. 115–124. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-60946-7_12
Gould, M.K., et al.: Evaluation of individuals with pulmonary nodules: when is it lung cancer?: diagnosis and management of lung cancer: American college of chest physicians evidence-based clinical practice guidelines. Chest 143(5), e93S-e120S (2013)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141 (2018)
Huang, H., Xu, H., Wang, X., Silamu, W.: Maximum f1-score discriminative training criterion for automatic mispronunciation detection. IEEE/ACM Trans. Audio, Speech, Lang. Process. 23(4), 787–797 (2015)
Huang, P., et al.: Prediction of lung cancer risk at follow-up screening with low-dose CT: a training and validation study of a deep learning method. Lancet Digit. Health 1(7), e353–e362 (2019)
de Koning, H.J., et al.: Reduced lung-cancer mortality with volume CT screening in a randomized trial. N. Engl. J. Med. 382(6), 503–513 (2020)
Lara, J.S., Contreras O., V.H., Otálora, S., Müller, H., González, F.A.: Multimodal latent semantic alignment for automated prostate tissue classification and retrieval. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) MICCAI 2020. LNCS, vol. 12265, pp. 572–581. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59722-1_55
Li, Y., et al.: Learning tumor growth via follow-up volume prediction for lung nodules. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) MICCAI 2020. LNCS, vol. 12266, pp. 508–517. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59725-2_49
Li, Z., Tao, R., Gavves, E., Snoek, C.G., Smeulders, A.W.: Tracking by natural language specification. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6495–6503 (2017)
Liu, C., Lin, Z., Shen, X., Yang, J., Lu, X., Yuille, A.: Recurrent multimodal interaction for referring image segmentation. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1271–1280 (2017)
Margffoy-Tuay, E., Pérez, J.C., Botero, E., Arbeláez, P.: Dynamic multimodal instance segmentation guided by natural language queries. In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 630–645 (2018)
Moriya, T., et al.: Unsupervised segmentation of Micro-CT images of lung cancer specimen using deep generative models. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., Khan, A. (eds.) MICCAI 2019. LNCS, vol. 11769, pp. 240–248. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32226-7_27
Rossi, S.E., Franquet, T., Volpacchio, M., Giménez, A., Aguilar, G.: Tree-in-bud pattern at thin-section CT of the lungs: radiologic-pathologic overview. Radiographics 25(3), 789–801 (2005)
Shaffie, A., et al.: A novel technology to integrate imaging and clinical markers for non-invasive diagnosis of lung cancer. Sci. Rep. 11(1), 1–10 (2021)
Society, A.C.: Lung cancer statistics: how common is lung cancer (2020). https://www.cancer.org/cancer/lung-cancer/about/key-statistics.html
Svoboda, E.: Artificial intelligence is improving the detection of lung cancer. Nature 587(7834), S20–S22 (2020)
Tam, L.K., Wang, X., Turkbey, E., Lu, K., Wen, Y., Xu, D.: Weakly supervised one-stage vision and language disease detection using large scale pneumonia and pneumothorax studies. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) MICCAI 2020. LNCS, vol. 12264, pp. 45–55. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59719-1_5
Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: the missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
Xia, C., et al.: A Multi-modality Network for Cardiomyopathy Death Risk Prediction with CMR Images and Clinical Information. In: Shen, D., Liu, T., Peters, T.M., Staib, L.H., Essert, C., Zhou, S., Yap, P.-T., Khan, A. (eds.) MICCAI 2019. LNCS, vol. 11765, pp. 577–585. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-32245-8_64
Yang, Z., Gong, B., Wang, L., Huang, W., Yu, D., Luo, J.: A fast and accurate one-stage approach to visual grounding. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4683–4693 (2019)
Yu, M., et al.: Toward rapid stroke diagnosis with multimodal deep learning. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) MICCAI 2020. LNCS, vol. 12263, pp. 616–626. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59716-0_59
Zhang, H., Gu, Y., Qin, Y., Yao, F., Yang, G.-Z.: Learning with sure data for nodule-level lung cancer prediction. In: Martel, A.L., Abolmaesumi, P., Stoyanov, D., Mateus, D., Zuluaga, M.A., Zhou, S.K., Racoceanu, D., Joskowicz, L. (eds.) MICCAI 2020. LNCS, vol. 12266, pp. 570–578. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59725-2_55
Acknowledgments
We thank AstraZeneca Colombia and the Lung Ambition Alliance for a research grant used for the development of this project.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2021 Springer Nature Switzerland AG
About this paper
Cite this paper
Roa, M., Daza, L., Escobar, M., Castillo, A., Arbelaez, P. (2021). SAMA: Spatially-Aware Multimodal Network with Attention For Early Lung Cancer Diagnosis. In: Syeda-Mahmood, T., et al. Multimodal Learning for Clinical Decision Support. ML-CDS 2021. Lecture Notes in Computer Science(), vol 13050. Springer, Cham. https://doi.org/10.1007/978-3-030-89847-2_5
Download citation
DOI: https://doi.org/10.1007/978-3-030-89847-2_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-030-89846-5
Online ISBN: 978-3-030-89847-2
eBook Packages: Computer ScienceComputer Science (R0)