More Web Proxy on the site http://driver.im/

research-article

MMF-NNs: : Multi-modal Multi-granularity Fusion Neural Networks for brain networks and its application to epilepsy identification

Authors:

Jiashuang Huang,

Mingliang Wang,

Daoqiang ZhangAuthors Info & Claims

Volume 157, Issue C

https://doi.org/10.1016/j.artmed.2024.102990

Published: 01 November 2024 Publication History

Abstract

Structural and functional brain networks are generated from two scan sequences of magnetic resonance imaging data, which can provide different perspectives for describing pathological changes caused by brain diseases. Recent studies found that fusing these two types of brain networks improves performance in brain disease identification. However, traditional fusion models combine these brain networks at a single granularity, ignoring the natural multi-granularity structure of brain networks that can be divided into the edge, node, and graph levels. To this end, this paper proposes a Multi-modal Multi-granularity Fusion Neural Networks (MMF-NNs) framework for brain networks, which integrates the features of the multi-modal brain network from global (i.e., graph-level) and local (i.e., edge-level and node-level) granularities to take full advantage of the topological information. Specifically, we design an interactive feature learning module at the local granularity to learn feature maps of structural and functional brain networks at the edge-level and the node-level, respectively. In that way, these two types of brain networks are fused during the feature learning process. At the global granularity, a multi-modal decomposition bilinear pooling module is designed to learn the graph-level joint representation of these brain networks. Experiments on real epilepsy datasets demonstrate that MMF-NNs are superior to several state-of-the-art methods in epilepsy identification.

Highlights

•

We design a general end-to-end neural network framework for fusing structural and functional brain networks to identify brain diseases.

•

We consider multi-granularity properties during the process of multi-modal brain network fusion for improved recognition performance.

•

Our proposed method is verified on a real epilepsy dataset.

References

[1]

Sporns O., Structure and function of complex brain networks, in: Dialogues in clinical neuroscience, Taylor & Francis, 2022.

[2]

Yan W., Calhoun V., Song M., Cui Y., Yan H., Liu S., Fan L., Zuo N., Yang Z., Xu K., et al., Discriminating schizophrenia using recurrent neural network applied on time courses of multi-site FMRI data, EBioMedicine 47 (2019) 543–552.

[3]

Fiorenzato E., Strafella A.P., Kim J., Schifano R., Weis L., Antonini A., Biundo R., Dynamic functional connectivity changes associated with dementia in Parkinson’s disease, Brain 142 (9) (2019) 2860–2872.

[4]

Planchuelo-Gomez A., Garcia-Azorin D., Guerrero A.L., Aja-Fernandez S., Rodriguez M., de Luis-Garcia R., Structural connectivity alterations in chronic and episodic migraine: A diffusion magnetic resonance imaging connectomics study, Cephalalgia 40 (4) (2020) 367–383.

[5]

Odusami M., Maskeliūnas R., Damaševičius R., Misra S., Machine learning with multimodal neuroimaging data to classify stages of Alzheimer’s disease: A systematic review and meta-analysis, Cognit Neurodyn 18 (3) (2024) 775–794.

[6]

Su T., Gong J., Tang G., Qiu S., Chen P., Chen G., Wang J., Huang L., Wang Y., Structural and functional brain alterations in anorexia nervosa: A multimodal meta-analysis of neuroimaging studies, Hum Brain Mapp 42 (15) (2021) 5154–5169.

[7]

Ma G., Lu C.-T., He L., Philip S.Y., Ragin A.B., Multi-view graph embedding with hub detection for brain network analysis, in: 2017 IEEE international conference on data mining, ICDM, IEEE, 2017, pp. 967–972.

[8]

Popp J.L., Thiele J.A., Faskowitz J., Seguin C., Sporns O., Hilger K., Structural-functional brain network coupling predicts human cognitive ability, NeuroImage 290 (2024).

[9]

Kulik S.D., Nauta I.M., Tewarie P., Koubiyr I., van Dellen E., Ruet A., Meijer K.A., de Jong B.A., Stam C.J., Hillebrand A., et al., Structure-function coupling as a correlate and potential biomarker of cognitive impairment in multiple sclerosis, Netw Neurosci 6 (2) (2022) 339–356.

[10]

Ma J., Liu F., Yang B., Xue K., Wang P., Zhou J., Wang Y., Niu Y., Zhang J., Selective aberrant functional–structural coupling of multiscale brain networks in subcortical vascular mild cognitive impairment, Neurosci Bull 37 (2021) 287–297.

[11]

Suárez L.E., Markello R.D., Betzel R.F., Misic B., Linking structure and function in macroscale brain networks, Trends Cognit Sci 24 (4) (2020) 302–315.

[12]

Oechslin M.S., Gschwind M., James C.E., Tracking training-related plasticity by combining fMRI and DTI: the right hemisphere ventral stream mediates musical syntax processing, Cerebral Cortex 28 (4) (2018) 1209–1218.

[13]

Sokolov A.A., Zeidman P., Erb M., Ryvlin P., Pavlova M.A., Friston K.J., Linking structural and effective brain connectivity: structurally informed parametric empirical Bayes (si-PEB), Brain Struct Funct 224 (2019) 205–217.

[14]

Wee C.-Y., Yap P.-T., Zhang D., Denny K., Browndyke J.N., Potter G.G., Welsh-Bohmer K.A., Wang L., Shen D., Identification of MCI individuals using structural and functional connectivity networks, Neuroimage 59 (3) (2012) 2045–2056.

[15]

Guo S., Huang C.-C., Zhao W., Yang A.C., Lin C.-P., Nichols T., Tsai S.-J., Combining multi-modality data for searching biomarkers in schizophrenia, PLoS One 13 (2) (2018).

[16]

Zhang L., Wang L., Gao J., Risacher S.L., Yan J., Li G., Liu T., Zhu D., Initiative A.D.N., et al., Deep fusion of brain structure-function in mild cognitive impairment, Med Image Anal 72 (2021).

[17]

Betzel R.F., Bassett D.S., Multi-scale brain networks, Neuroimage 160 (2017) 73–83.

[18]

Sendi M.S., Zendehrouh E., Fu Z., Liu J., Du Y., Mormino E., Salat D.H., Calhoun V.D., Miller R.L., Disrupted dynamic functional network connectivity among cognitive control networks in the progression of Alzheimer’s disease, Brain Connect 13 (6) (2023) 334–343.

[19]

Zhu Y., Lu T., Xie C., Wang Q., Wang Y., Cao X., Su Y., Wang Z., Zhang Z., Functional disorganization of small-world brain networks in patients with ischemic leukoaraiosis, Front. Aging Neurosci. 12 (2020) 203.

[20]

Praveen RG, de Melo WC, Ullah N, Aslam H, Zeeshan O, Denorme T, Pedersoli M, Koerich AL, Bacon S, Cardinal P, et al. A joint cross-attention model for audio-visual fusion in dimensional emotion recognition. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022, p. 2486–95.

[21]

Yang X., Xi X., Yang L., Xu C., Song Z., Nie X., Qiao L., Li C., Shi Q., Yin Y., Multi-modality relation attention network for breast tumor classification, Comput Biol Med 150 (2022).

[22]

An Y., Zhang H., Sheng Y., Wang J., Chen X., MAIN: Multimodal attention-based fusion networks for diagnosis prediction, in: 2021 IEEE international conference on bioinformatics and biomedicine, BIBM, IEEE, 2021, pp. 809–816.

[23]

Lei B., Cheng N., Frangi A.F., Tan E.-L., Cao J., Yang P., Elazab A., Du J., Xu Y., Wang T., Self-calibrated brain network estimation and joint non-convex multi-task learning for identification of early Alzheimer’s disease, Med Image Anal 61 (2020).

[24]

Zuo Q., Shen Y., Zhong N., Chen C.P., Lei B., Wang S., Alzheimer’s disease prediction via brain structural-functional deep fusing network, IEEE Trans Neural Syst Rehabil Eng 31 (2023) 4601–4612.

[25]

Zhang W., Zhan L., Thompson P., Wang Y., Deep representation learning for multimodal brain networks, in: Medical image computing and computer assisted intervention–MICCAI 2020: 23rd international conference, Lima, Peru, October 4–8, 2020, proceedings, part VII 23, Springer, 2020, pp. 613–624.

[26]

Tang H., Guo L., Fu X., Wang Y., Mackin S., Ajilore O., Leow A.D., Thompson P.M., Huang H., Zhan L., Signed graph representation learning for functional-to-structural brain network mapping, Med Image Anal 83 (2023).

[27]

Ye K., Tang H., Dai S., Guo L., Liu J.Y., Wang Y., Leow A., Thompson P.M., Huang H., Zhan L., Bidirectional mapping with contrastive learning on multimodal neuroimaging data, in: International conference on medical image computing and computer-assisted intervention, Springer, 2023, pp. 138–148.

[28]

Wang H., Pan H., Zhang K., He S., Chen C., M2FNet: multi-granularity feature fusion network for medical visual question answering, in: Pacific rim international conference on artificial intelligence, Springer, 2022, pp. 141–154.

[29]

Peng H., Ma Y., Li Y., Cambria E., Learning multi-grained aspect target sequence for Chinese sentiment analysis, Knowl-Based Syst 148 (2018) 167–176.

[30]

Xia H., Lu L., Song S., Feature fusion of multi-granularity and multi-scale for facial expression recognition, Vis Comput (2023) 1–13.

[31]

Kawahara J., Brown C.J., Miller S.P., Booth B.G., Chau V., Grunau R.E., Zwicker J.G., Hamarneh G., BrainNetCNN: Convolutional neural networks for brain networks; towards predicting neurodevelopment, NeuroImage 146 (2017) 1038–1049.

[32]

Dyrba M., Grothe M., Kirste T., Teipel S.J., Multimodal analysis of functional and structural disconnection in Alzheimer’s disease using multiple kernel SVM, Hum Brain Mapp 36 (6) (2015) 2118–2131.

[33]

Lu H., Plataniotis K.N., Venetsanopoulos A.N., MPCA: Multilinear principal component analysis of tensor objects, IEEE Trans Neural Netw 19 (1) (2008) 18–39.

Digital Library

[34]

Atwood J., Towsley D., Diffusion-convolutional neural networks, Adv Neural Inf Process Syst 29 (2016).

[35]

Zhang X., He L., Chen K., Luo Y., Zhou J., Wang F., Multi-view graph convolutional network and its applications on neuroimage analysis for Parkinson’s disease, in: AMIA annual symposium proceedings, Vol. 2018, American Medical Informatics Association, 2018, p. 1147.

[36]

Song X., Zhou F., Frangi A.F., Cao J., Xiao X., Lei Y., Wang T., Lei B., Multicenter and multichannel pooling GCN for early AD diagnosis based on dual-modality fused brain network, IEEE Trans Med Imaging 42 (2) (2022) 354–367.

[37]

Lopez C., Heydrich L., Seeck M., Blanke O., Abnormal self-location and vestibular vertigo in a patient with right frontal lobe epilepsy, Epilepsy Behav 17 (2) (2010) 289–292.

[38]

Schaper F.L., Zhao Y., Janssen M.L., Wagner G.L., Colon A.J., Hilkman D.M., Gommer E., Vlooswijk M.C., Hoogland G., Ackermans L., et al., Single-cell recordings to target the anterior nucleus of the thalamus in deep brain stimulation for patients with refractory epilepsy, Int J Neural Syst 29 (04) (2019).

[39]

Shankar A., Khaing H.K., Dandapat S., Barma S., Epileptic seizure classification based on gramian angular field transformation and deep learning, in: 2020 IEEE applied signal processing conference, ASPCON, IEEE, 2020, pp. 147–151.

[40]

Gonen O.M., Moffat B.A., Desmond P.M., Lui E., Kwan P., O’Brien T.J., Seven-tesla quantitative magnetic resonance spectroscopy of glutamate, γ-aminobutyric acid, and glutathione in the posterior cingulate cortex/precuneus in patients with epilepsy, Epilepsia 61 (12) (2020) 2785–2794.

[41]

Selvaraju RR, Cogswell M, Das A, Vedantam R, Parikh D, Batra D. Grad-cam: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the IEEE international conference on computer vision. 2017, p. 618–26.

[42]

Huang J., Zhu Q., Wang M., Zhou L., Zhang Z., Zhang D., Coherent pattern in multi-layer brain networks: Application to epilepsy identification, IEEE J Biomed Health Inf 24 (9) (2019) 2609–2620.

Index Terms

MMF-NNs: Multi-modal Multi-granularity Fusion Neural Networks for brain networks and its application to epilepsy identification
1. Applied computing
  1. Life and medical sciences
2. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Index terms have been assigned to the content through auto-classification.

Recommendations

Infra-Slow Electroencephalogram Power Associates with Reaction Time in Simple Discrimination Tasks
Neural Information Processing
Abstract
Infra-slow (<0.1 Hz) electroencephalography (EEG) activity is recently thought to be an important clue for the elucidation of the default mode network (DMN), one of large-scale brain networks, which is known to activate during relaxed non-task ...
Route Adjustment of Functional Brain Network in Mental Arithmetic Using Task-Evoked FMRI
Brain Informatics
Abstract
A large number of studies on altered functional brain network tend to focus only on the alternation in topological metric of functional connectivity, rather than on the details of graph adjustment that cause topological metric changes, such as ...
Synchronization in functional brain networks of children suffering from ADHD based on Hindmarsh-Rose neuronal model
Abstract
Attention Deficit Hyperactivity Disorder (ADHD) is one of the common psychological disorders in children, which causes improper recognition of others' emotions. These children often have very poor social interaction and a lack of self-...
Highlights
- Improper facial emotion recognition in ADHD children.
- Synchronization ...

Comments

Please enable JavaScript to view thecomments powered by Disqus.

Information & Contributors

Information

Published In

cover image Artificial Intelligence in Medicine

Artificial Intelligence in Medicine Volume 157, Issue C

Nov 2024

404 pages

Issue’s Table of Contents

Elsevier B.V.

Publisher

Elsevier Science Publishers Ltd.

United Kingdom

Publication History

Published: 01 November 2024

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

0
Total Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 28 Jan 2025

Other Metrics

View Author Metrics

Citations

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents