Abstract
Multimodality eye disease screening is crucial in ophthalmology as it integrates information from diverse sources to complement their respective performances. However, the existing methods are weak in assessing the reliability of each unimodality, and directly fusing an unreliable modality may cause screening errors. To address this issue, we introduce a novel multimodality evidential fusion pipeline for eye disease screening, EyeMoSt, which provides a measure of confidence for unimodality and elegantly integrates the multimodality information from a multi-distribution fusion perspective. Specifically, our model estimates both local uncertainty for unimodality and global uncertainty for the fusion modality to produce reliable classification results. More importantly, the proposed mixture of Student’s t distributions adaptively integrates different modalities to endow the model with heavy-tailed properties, increasing robustness and reliability. Our experimental findings on both public and in-house datasets show that our model is more reliable than current methods. Additionally, EyeMost has the potential ability to serve as a data quality discriminator, enabling reliable decision-making for multimodality eye disease screening.
K. Zou and T. Lin—Equal contribution.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Our code has been released in https://github.com/Cocofeat/EyeMoSt.
- 2.
The ethical approval of this dataset was obtained from the Ethical Committee.
References
Amini, A., Schwarting, W., Soleimany, A., Rus, D.: Deep evidential regression. Adv. Neural. Inf. Process. Syst. 33, 14927–14937 (2020)
Baltrušaitis, T., Ahuja, C., Morency, L.P.: Multimodal machine learning: a survey and taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41(2), 423–443 (2018)
Cai, Z., Lin, L., He, H., Tang, X.: COROLLA: an efficient multi-modality fusion framework with supervised contrastive learning for glaucoma grading. In: IEEE International Symposium on Biomedical Imaging (ISBI), pp. 1–4. IEEE (2022)
Cai, Z., Lin, L., He, H., Tang, X.: Uni4Eye: unified 2D and 3D self-supervised pre-training via masked image modeling transformer for ophthalmic image classification. In: Wang, L., Dou, Q., Fletcher, P.T., Speidel, S., Li, S. (eds.) Medical Image Computing and Computer Assisted Intervention – MICCAI 2022. MICCAI 2022. LNCS, vol. 13438. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16452-1_9
Chen, S., Ma, K., Zheng, Y.: Med3D: transfer learning for 3D medical image analysis. arXiv preprint arXiv:1904.00625 (2019)
Gal, Y., Ghahramani, Z.: Dropout as a Bayesian approximation: representing model uncertainty in deep learning. In: International Conference on Machine Learning, pp. 1050–1059. PMLR (2016)
Gao, S.H., Cheng, M.M., Zhao, K., Zhang, X.Y., Yang, M.H., Torr, P.: Res2net: a new multi-scale backbone architecture. IEEE Trans. Pattern Anal. Mach. Intell. 43(2), 652–662 (2021)
Han, Z., Zhang, C., Fu, H., Zhou, J.T.: Trusted multi-view classification. In: International Conference on Learning Representations (2020)
He, X., Deng, Y., Fang, L., Peng, Q.: Multi-modal retinal image classification with modality-specific attention network. IEEE Trans. Med. Imaging 40(6), 1591–1602 (2021)
Hua, C.H., et al.: Convolutional network with twofold feature augmentation for diabetic retinopathy recognition from multi-modal images. IEEE J. Biomed. Health Inform. 25(7), 2686–2697 (2020)
Jung, M.C., Zhao, H., Dipnall, J., Gabbe, B., Du, L.: Uncertainty estimation for multi-view data: the power of seeing the whole picture. In: Advances in Neural Information Processing Systems (2022)
Kendall, A., Gal, Y.: What uncertainties do we need in Bayesian deep learning for computer vision? In: Advances in Neural Information Processing Systems (2017)
Kingma, D., Ba, J.: Adam: a method for stochastic optimization. Computer Science (2014)
Lakshminarayanan, B., Pritzel, A., Blundell, C.: Simple and scalable predictive uncertainty estimation using deep ensembles. In: Advances in Neural Information Processing Systems 30 (2017)
Li, X., Jia, M., Islam, M.T., Yu, L., Xing, L.: Self-supervised feature learning via exploiting multi-modal data for retinal disease diagnosis. IEEE Trans. Med. Imaging 39(12), 4023–4033 (2020)
Li, Y., El Habib Daho, M., Conze, P.H., et al.: Multimodal information fusion for glaucoma and diabetic retinopathy classification. In: Antony, B., Fu, H., Lee, C.S., MacGillivray, T., Xu, Y., Zheng, Y. (eds.) Ophthalmic Medical Image Analysis. OMIA 2022. LNCS, vol. 13576. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-16525-2_6
Liu, J.Z., Lin, Z., Padhy, S., Tran, D., Bedrax-Weiss, T., Lakshminarayanan, B.: Simple and principled uncertainty estimation with deterministic deep learning via distance awareness. In: Proceedings of the 34th International Conference on Neural Information Processing Systems (2020)
MacKay, D.J.: A practical Bayesian framework for backpropagation networks. Neural Comput. 4(3), 448–472 (1992)
Malinin, A., Gales, M.: Predictive uncertainty estimation via prior networks. In: Advances in Neural Information Processing Systems 31 (2018)
Maronas, J., Paredes, R., Ramos, D.: Calibration of deep probabilistic models with decoupled Bayesian neural networks. Neurocomputing 407, 194–205 (2020)
Neal, R.M.: Bayesian learning for neural networks, vol. 118. Springer New York, NY (2012). https://doi.org/10.1007/978-1-4612-0745-0
Ranganath, R., Gerrish, S., Blei, D.: Black box variational inference. In: Artificial Intelligence and Statistics, pp. 814–822. PMLR (2014)
Rodrigues, E.O., Conci, A., Liatsis, P.: Element: multi-modal retinal vessel segmentation based on a coupled region growing and machine learning approach. IEEE J. Biomed. Health Inform. 24(12), 3507–3519 (2020)
Roth, M., Özkan, E., Gustafsson, F.: A student’s t filter for heavy tailed process and measurement noise. In: 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 5770–5774. IEEE (2013)
Sensoy, M., Kaplan, L., Kandemir, M.: Evidential deep learning to quantify classification uncertainty. In: Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 3183–3193 (2018)
Van Amersfoort, J., Smith, L., Teh, Y.W., Gal, Y.: Uncertainty estimation using a single deep deterministic neural network. In: International Conference on Machine Learning, pp. 9690–9700. PMLR (2020)
Wang, W., et al.: Learning two-stream CNN for multi-modal age-related macular degeneration categorization. IEEE J. Biomed. Health Inform. 26(8), 4111–4122 (2022)
Woo, S., Park, J., Lee, J.-Y., Kweon, I.S.: CBAM: convolutional block attention module. In: Ferrari, V., Hebert, M., Sminchisescu, C., Weiss, Y. (eds.) ECCV 2018. LNCS, vol. 11211, pp. 3–19. Springer, Cham (2018). https://doi.org/10.1007/978-3-030-01234-2_1
Wu, J., Fang, H., Li, F., et al.: Gamma challenge: glaucoma grading from multi-modality images. arXiv preprint arXiv:2202.06511 (2022)
Zadeh, L.A.: Review of a mathematical theory of evidence. AI Mag. 5(3), 81 (1984)
Zhou, T., Ruan, S., Canu, S.: A review: deep learning for medical image segmentation using multi-modality fusion. Array 3, 100004 (2019)
Zou, K., et al.: EvidenceCap: towards trustworthy medical image segmentation via evidential identity cap. arXiv preprint arXiv:2301.00349 (2023)
Acknowledgements
This work was supported by the National Research Foundation, Singapore under its AI Singapore Programme (AISG Award No: AISG2-TC-2021-003), A*STAR AME Programmatic Funding Scheme Under Project A20H4b0141, A*STAR Central Research Fund, the Science and Technology Department of Sichuan Province (Grant No. 2022YFS0071 & 2023YFG0273), and the China Scholarship Council (No. 202206240082).
Author information
Authors and Affiliations
Corresponding authors
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zou, K. et al. (2023). Reliable Multimodality Eye Disease Screening via Mixture of Student’s t Distributions. In: Greenspan, H., et al. Medical Image Computing and Computer Assisted Intervention – MICCAI 2023. MICCAI 2023. Lecture Notes in Computer Science, vol 14226. Springer, Cham. https://doi.org/10.1007/978-3-031-43990-2_56
Download citation
DOI: https://doi.org/10.1007/978-3-031-43990-2_56
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43989-6
Online ISBN: 978-3-031-43990-2
eBook Packages: Computer ScienceComputer Science (R0)