Abstract
Brain network analysis is a useful approach to studying human brain disorders because it can distinguish patients from healthy people by detecting abnormal connections. Due to the complementary information from multiple modal neuroimages, multimodal fusion technology has a lot of potential for improving prediction performance. However, effective fusion of multimodal medical images to achieve complementarity is still a challenging problem. In this paper, a novel hierarchical structural-functional connectivity fusing (HSCF) model is proposed to construct brain structural-functional connectivity matrices and predict abnormal brain connections based on functional magnetic resonance imaging (fMRI) and diffusion tensor imaging (DTI). Specifically, the prior knowledge is incorporated into the separators for disentangling each modality of information by the graph convolutional networks (GCN). And a disentangled cosine distance loss is devised to ensure the disentanglement’s effectiveness. Moreover, the hierarchical representation fusion module is designed to effectively maximize the combination of relevant and effective features between modalities, which makes the generated structural-functional connectivity more robust and discriminative in the cognitive disease analysis. Results from a wide range of tests performed on the public Alzheimer’s Disease Neuroimaging Initiative (ADNI) database show that the proposed model performs better than competing approaches in terms of classification evaluation. In general, the proposed HSCF model is a promising model for generating brain structural-functional connectivities and identifying abnormal brain connections as cognitive disease progresses.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
References
Tsentidou, G., Moraitou, D., Tsolaki, M.: Cognition in vascular aging and mild cognitive impairment. J. Alzheimers Dis. 72(1), 55–70 (2019)
Davatzikos, C., Bhatt, P., Shaw, L.M., Batmanghelich, K.N., Trojanowski, J.Q.: Prediction of mci to ad conversion, via MRI, CSF biomarkers, and pattern classification. Neurobiol. Aging 32(12), 2322 (2011)
Peres, M.A., et al.: Oral diseases: a global public health challenge. Lancet 394(10194), 249–260 (2019)
Keren-Shaul, H., et al.: A unique microglia type associated with restricting development of Alzheimer’s disease. Cell 169(7), 1276–1290 (2017)
Wang, S.-Q., Li, X., Cui, J.-L., Li, H.-X., Luk, K.D., Hu, Y.: Prediction of myelopathic level in cervical spondylotic myelopathy using diffusion tensor imaging. J. Magn. Reson. Imaging 41(6), 1682–1688 (2015)
Shen, Y., Huang, X., Kwak, K.S., Yang, B., Wang, S.: Subcarrier-pairing-based resource optimization for OFDM wireless powered relay transmissions with time switching scheme. IEEE Trans. Signal Process. 65(5), 1130–1145 (2016)
Gibson, E., et al.: NiftyNet: a deep-learning platform for medical imaging. Comput. Methods Programs Biomed. 158, 113–122 (2018)
Wang, S., Shen, Y., Zeng, D., Hu, Y.: Bone age assessment using convolutional neural networks. In: 2018 International Conference on Artificial Intelligence and Big Data (ICAIBD), pp. 175–178. IEEE (2018)
Wang, S., et al.: Skeletal maturity recognition using a fully automated system with convolutional neural networks. IEEE Access 6, 29979–29993 (2018)
Hong, J., et al.: Brain age prediction of children using routine brain MR images via deep learning. Front. Neurol. 11, 584682 (2020)
Wang, S., et al.: An ensemble-based densely-connected deep learning system for assessment of skeletal maturity. IEEE Trans. Syst., Man, Cybern.: Syst. 52(1), 426–437 (2020)
Hu, S., Yu, W., Chen, Z., Wang, S.: Medical image reconstruction using generative adversarial network for Alzheimer disease assessment with class-imbalance problem. In: 2020 IEEE 6th International Conference on Computer and Communications (ICCC), pp. 1323–1327. IEEE (2020)
Yu, W., Lei, B., Ng, M.K., Cheung, A.C., Shen, Y., Wang, S.: Tensorizing GAN with high-order pooling for Alzheimer’s disease assessment. IEEE Trans. Neural Netw. Learn. Syst. 33(9), 4945–4959 (2021)
Zeng, D., Wang, S., Shen, Y., Shi, C.: A GA-based feature selection and parameter optimization for support tucker machine. Procedia Comput. Sci. 111, 17–23 (2017)
Zuo, Q., Pun, C. M., Zhang, Y., Wang, H., Hong, J.: Multi-resolution spatiotemporal enhanced transformer denoising with functional diffusive gans for constructing brain effective connectivity in MCI analysis. arXiv preprint: arXiv:2305.10754 (2023)
Zuo, Q., Lei, B., Shen, Y., Liu, Y., Feng, Z., Wang, S.: Multimodal representations learning and adversarial hypergraph fusion for early Alzheimer’s disease prediction. In: Ma, H., et al. (eds.) PRCV 2021. LNCS, vol. 13021, pp. 479–490. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88010-1_40
Wang, S., Shen, Y., Chen, W., Xiao, T., Hu, J.: Automatic recognition of mild cognitive impairment from MRI images using expedited convolutional neural networks. In: Lintas, A., Rovetta, S., Verschure, P.F.M.J., Villa, A.E.P. (eds.) ICANN 2017. LNCS, vol. 10613, pp. 373–380. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-68600-4_43
Wang, S., Hu, Y., Shen, Y., Li, H.: Classification of diffusion tensor metrics for the diagnosis of a myelopathic cord using machine learning. Int. J. Neural Syst. 28(02), 1750036 (2018)
Hong, J., Yu, S.C.-H., Chen, W.: Unsupervised domain adaptation for cross-modality liver segmentation via joint adversarial learning and self-learning. Appl. Soft Comput. 121, 108729 (2022)
Lei, B., et al.: Predicting clinical scores for Alzheimer’s disease based on joint and deep learning. Expert Syst. Appl. 187, 115966 (2022)
Hirjak, D., et al.: Multimodal magnetic resonance imaging data fusion reveals distinct patterns of abnormal brain structure and function in catatonia. Schizophr. Bull. 46(1), 202–210 (2020)
Zuo, Q., et al.: Brain functional network generation using distribution-regularized adversarial graph autoencoder with transformer for dementia diagnosis (2023)
Honey, C.J., et al.: Predicting human resting-state functional connectivity from structural connectivity. Proc. Natl. Acad. Sci. 106(6), 2035–2040 (2009)
Zuo, Q., et al.: Hemisphere-separated cross-connectome aggregating learning via VAE-GAN for brain structural connectivity synthesis. IEEE Access 11, 48493–48505 (2023)
Wang, S., Wang, H., Shen, Y., Wang, X.: Automatic recognition of mild cognitive impairment and Alzheimers disease using ensemble based 3D densely connected convolutional networks. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), pp. 517–523. IEEE (2018)
Hu, S., Yuan, J., Wang, S.: Cross-modality synthesis from MRI to PET using adversarial U-net with different normalization. In: 2019 International Conference on Medical Imaging Physics and Engineering (ICMIPE), pp. 1–5. IEEE (2019)
Wang, S., Wang, H., Cheung, A.C., Shen, Y., Gan, M.: Ensemble of 3D densely connected convolutional network for diagnosis of mild cognitive impairment and Alzheimer’s disease. In: Wani, M.A., Kantardzic, M., Sayed-Mouchaweh, M. (eds.) Deep Learning Applications. AISC, vol. 1098, pp. 53–73. Springer, Singapore (2020). https://doi.org/10.1007/978-981-15-1816-4_4
Hu, S., Lei, B., Wang, S., Wang, Y., Feng, Z., Shen, Y.: Bidirectional mapping generative adversarial networks for brain MR to pet synthesis. IEEE Trans. Med. Imaging 41(1), 145–157 (2021)
Yu, W., et al.: Morphological feature visualization of Alzheimer’s disease via multidirectional perception GAN. IEEE Trans. Neural Netw. Learn. Syst. (2022)
You, S. et al.: Fine perceptive GANs for brain MR image super-resolution in wavelet domain. IEEE Trans. Neural Netw. Learn. Syst. (2022)
Lei, B., et al.: Diagnosis of early Alzheimer’s disease based on dynamic high order networks. Brain Imaging Behav. 15, 276–287 (2021)
Zuo, Q., Lei, B., Zhong, N., Pan, Y., Wang, S.: Brain structure-function fusing representation learning using adversarial decomposed-VAE for analyzing MCI. arXiv preprint: arXiv:2305.14404 (2023)
Zuo, Q., Lu, L., Wang, L., Zuo, J., Ouyang, T.: Constructing brain functional network by adversarial temporal-spatial aligned transformer for early AD analysis. Front. Neurosci. 16, 1087176 (2022)
Zuo, Q., Lei, B., Wang, S., Liu, Y., Wang, B., Shen, Y.: A prior guided adversarial representation learning and hypergraph perceptual network for predicting abnormal connections of Alzheimer’s disease. arXiv preprint: arXiv:2110.09302 (2021)
Zong, Y., Jing, C., Zuo, Q.: Multiscale autoencoder with structural-functional attention network for Alzheimer’s disease prediction. In: Yu, S., et al. (eds.) PRCV 2022. Lecture Notes in Computer Science, vol. 13535, pp. 286–297. Springer, Cham (2022)
Hong, J., Zhang, Y.-D., Chen, W.: Source-free unsupervised domain adaptation for cross-modality abdominal multi-organ segmentation. Knowl.-Based Syst. 250, 109155 (2022)
Yu, S., et al.: Multi-scale enhanced graph convolutional network for early mild cognitive impairment detection. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12267, pp. 228–237. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59728-3_23
Hu, S., Shen, Y., Wang, S., Lei, B.: Brain MR to PET synthesis via bidirectional generative adversarial network. In: Martel, A.L., et al. (eds.) MICCAI 2020. LNCS, vol. 12262, pp. 698–707. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-59713-9_67
Liu, L., Wang, Y.-P., Wang, Y., Zhang, P., Xiong, S.: An enhanced multi-modal brain graph network for classifying neuropsychiatric disorders. Med. Image Anal. 81, 102550 (2022)
Zhang, L., et al.: Deep fusion of brain structure-function in mild cognitive impairment. Med. Image Anal. 72, 102082 (2021)
Wang, S.-Q.: A variational approach to nonlinear two-point boundary value problems. Comput. Math. Appl. 58(11–12), 2452–2455 (2009)
Mo, L.-F., Wang, S.-Q.: A variational approach to nonlinear two-point boundary value problems. Nonlinear Anal. Theory Methods Appl. 71(12), e834–e838 (2009)
Kingma, D.P., et al.: An introduction to variational autoencoders. Found. Trends® Mach. Learn. 12(4), 307–392 (2019)
Atwood, J., Towsley, D.: Diffusion-convolutional neural networks. In: Advances in Neural Information Processing Systems, vol. 29 (2016)
Zhang, X., He, L., Chen, K., Luo, Y., Zhou, J., Wang, F.: Multi-view graph convolutional network and its applications on neuroimage analysis for Parkinson’s disease. In: AMIA Annual Symposium Proceedings, vol. 2018, p. 1147. American Medical Informatics Association (2018)
Lei, B., et al.: Self-calibrated brain network estimation and joint non-convex multi-task learning for identification of early Alzheimer’s disease. Med. Image Anal. 61, 101652 (2020)
Acknowledgements
This work was supported in part by the Guangdong Social Science Planning Project under Grant GD21YWW01, in part by the Special Fund for Young Teachers in Basic Scientific Research Business Expenses of Central Universities under Grant 2023qntd28, in part by the Young Doctoral Research Start-Up Fund of Hubei University of Economics under Grant XJ22BS28.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Zuo, Q., Zhu, Y., Lu, L., Yang, Z., Li, Y., Zhang, N. (2023). Fusing Structural and Functional Connectivities Using Disentangled VAE for Detecting MCI. In: Liu, F., Zhang, Y., Kuai, H., Stephen, E.P., Wang, H. (eds) Brain Informatics. BI 2023. Lecture Notes in Computer Science(), vol 13974. Springer, Cham. https://doi.org/10.1007/978-3-031-43075-6_1
Download citation
DOI: https://doi.org/10.1007/978-3-031-43075-6_1
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-43074-9
Online ISBN: 978-3-031-43075-6
eBook Packages: Computer ScienceComputer Science (R0)