[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to main content
Springer Nature Link
Log in

Spatiotemporal Pooling on Appropriate Topological Maps Represented as Two-Dimensional Images for EEG Classification

  • Conference paper
  • First Online:
Computer Vision – ACCV 2024 (ACCV 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 15473))

Included in the following conference series:

  • 98 Accesses

Abstract

Motor imagery classification based on electroencephalography (EEG) signals is one of the most important brain-computer interface applications, although it need further improvement. Several methods have attempted to obtain useful information from EEG signals by using recent deep learning techniques such as transformers. To improve the classification accuracy, this study proposes a novel EEG-based motor imagery classification method with three key features: generation of a topological map represented as a two-dimensional image from EEG signals with coordinate transformation based on t-SNE, use of the InternImage to extract spatial features, and use of spatiotemporal pooling inspired by PoolFormer to exploit spatiotemporal information concealed in a sequence of EEG images. Experimental results using the PhysioNet EEG Motor Movement/Imagery dataset showed that the proposed method achieved the best classification accuracy of 88.57%, 80.69%, and 70.20% on two-, three-, and four-class motor imagery tasks in cross-individual validation.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
GBP 19.95
Price includes VAT (United Kingdom)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
GBP 99.99
Price includes VAT (United Kingdom)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
GBP 129.99
Price includes VAT (United Kingdom)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

References

  1. Alfeld, P.: A trivariate Clough-Tocher scheme for tetrahedral data. Comput. Aided Geom. Des. 1(2), 169–181 (1984). https://doi.org/10.1016/0167-8396(84)90029-3

    Article  Google Scholar 

  2. Ali, O., Saif-ur Rehman, M., Glasmachers, T., Iossifidis, I., Klaes, C.: ConTraNet: A single end-to-end hybrid network for EEG-based and EMG-based human machine interfaces. arXiv preprint arXiv:2206.10677 (2022). 10.48550/arXiv.2206.10677

  3. Altaheri, H., Muhammad, G., Alsulaiman, M., Amin, S.U., Altuwaijri, G.A., Abdul, W., Bencherif, M.A., Faisal, M.: Deep learning techniques for classification of electroencephalogram (EEG) motor imagery (MI) signals: a review. Neural Comput. Appl. , 1–42 (2021). https://doi.org/10.1007/s00521-021-06352-5

  4. Amin, S.U., Alsulaiman, M., Muhammad, G., Bencherif, M.A., Hossain, M.S.: Multilevel Weighted Feature Fusion Using Convolutional Neural Networks for EEG Motor Imagery Classification. IEEE Access 7, 18940–18950 (2019). https://doi.org/10.1109/ACCESS.2019.2895688

    Article  Google Scholar 

  5. Bashivan, P., Rish, I., Yeasin, M., Codella, N.: Learning Representations from EEG with Deep Recurrent-Convolutional Neural Networks. In: ICLR. pp. 1–15 (2016)

    Google Scholar 

  6. Brunner, C., Leeb, R., Müller-Putz, G., Schlögl, A., Pfurtscheller, G.: BCI Competition 2008–Graz data set A. Institute for Knowledge Discovery (Laboratory of Brain-Computer Interfaces), Graz University of Technology 16, 1–6 (2008)

    Google Scholar 

  7. Carlson, T., del R. Millan, J.: Brain-Controlled Wheelchairs: A Robotic Architecture. IEEE Robot. Autom. Mag. 20(1), 65–73 (2013). 10.1109/MRA.2012.2229936

    Google Scholar 

  8. Cho, H., Ahn, M., Ahn, S., Kwon, M., Jun, S.C.: EEG datasets for motor imagery brain-computer interface. GigaScience 6(7), gix034 (2017). 10.1093/gigascience/gix034

    Google Scholar 

  9. Craik, A., He, Y., Contreras-Vidal, J.L.: Deep learning for electroencephalogram (EEG) classification tasks: a review. J. Neural Eng. 16(3), 031001 (2019). https://doi.org/10.1088/1741-2552/ab0ab5

    Article  Google Scholar 

  10. Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: CVPR. pp. 702–703 (2020)

    Google Scholar 

  11. Dose, H., Møller, J.S., Iversen, H.K., Puthusserypady, S.: An end-to-end deep learning approach to MI-EEG signal classification for BCIs. Expert Syst. Appl. 114, 532–542 (2018). https://doi.org/10.1016/j.eswa.2018.08.031

    Article  Google Scholar 

  12. Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In: ICLR (2021)

    Google Scholar 

  13. Du, Y., Xu, Y., Wang, X., Liu, L., Ma, P.: EEG temporal-spatial transformer for person identification. Sci. Rep. 12(1), 14378 (2022). https://doi.org/10.1038/s41598-022-18502-3

    Article  Google Scholar 

  14. Ehrsson, H.H., Geyer, S., Naito, E.: Imagery of Voluntary Movement of Fingers, Toes, and Tongue Activates Corresponding Body-Part-Specific Motor Representations. J. Neurophysiol. 90(5), 3304–3316 (2003). https://doi.org/10.1152/jn.01113.2002

    Article  Google Scholar 

  15. Fadel, W., Kollod, C., Wahdow, M., Ibrahim, Y., Ulbert, I.: Multi-Class Classification of Motor Imagery EEG Signals Using Image-Based Deep Recurrent Convolutional Neural Network. In: BCI. pp. 1–4 (2020). 10.1109/BCI48061.2020.9061622

    Google Scholar 

  16. Farooq, F., Rashid, N., Farooq, A., Ahmed, M., Zeb, A., Iqbal, J.: Motor Imagery based Multivariate EEG Signal Classification for Brain Controlled Interface Applications. In: ICOM. pp. 1–6. IEEE (2019). 10.1109/ICOM47790.2019.8952008

    Google Scholar 

  17. Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.K., Stanley, H.E.: PhysioBank, PhysioToolkit, and PhysioNet Components of a New Research Resource for Complex Physiologic Signals. Circulation 101(23), e215–e220 (2000). https://doi.org/10.1161/01.CIR.101.23.e215

    Article  Google Scholar 

  18. Jeong, J.H., Lee, B.H., Lee, D.H., Yun, Y.D., Lee, S.W.: EEG Classification of Forearm Movement Imagery Using a Hierarchical Flow Convolutional Neural Network. IEEE Access 8, 66941–66950 (2020). https://doi.org/10.1109/ACCESS.2020.2983182

    Article  Google Scholar 

  19. Ji, Y., Li, F., Fu, B., Zhou, Y., Wu, H., Li, Y., Li, X., Shi, G.: A novel hybrid decoding neural network for EEG signal representation. Pattern Recognition p. 110726 (2024). https://doi.org/10.1016/j.patcog.2024.110726

    Google Scholar 

  20. Kant, P., Laskar, S.H., Hazarika, J., Mahamune, R.: CWT Based Transfer Learning for Motor Imagery Classification for Brain computer Interfaces. J. Neurosci. Methods 345, 108886 (2020). https://doi.org/10.1016/j.jneumeth.2020.108886

    Article  Google Scholar 

  21. Kataoka, H., Okayasu, K., Matsumoto, A., Yamagata, E., Yamada, R., Inoue, N., Nakamura, A., Satoh, Y.: Pre-Training Without Natural Images. Int. J. Comput. Vision 130(4), 990–1007 (2022). https://doi.org/10.1007/s11263-021-01555-8

    Article  Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimization (2015)

    Google Scholar 

  23. Kostas, D., Aroca-Ouellette, S., Rudzicz, F.: BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data. Front. Hum. Neurosci. 15, 653659 (2021). https://doi.org/10.3389/fnhum.2021.653659

    Article  Google Scholar 

  24. Lake, B.M., Salakhutdinov, R., Tenenbaum, J.B.: Human-level concept learning through probabilistic program induction. Science 350(6266), 1332–1338 (2015). https://doi.org/10.1126/science.aab3050

    Article  MathSciNet  Google Scholar 

  25. Lawhern, V.J., Solon, A.J., Waytowich, N.R., Gordon, S.M., Hung, C.P., Lance, B.J.: EEGNet: a compact convolutional neural network for EEG-based brain-computer interfaces. J. Neural Eng. 15(5), 056013 (2018). https://doi.org/10.1088/1741-2552/aace8c

    Article  Google Scholar 

  26. Leeb, R., Brunner, C., Müller-Putz, G., Schlögl, A., Pfurtscheller, G.: BCI Competition 2008-Graz data set B. Graz University of Technology, Austria 16, 1–6 (2008)

    Google Scholar 

  27. Li, M.A., Han, J.F., Duan, L.J.: A Novel MI-EEG Imaging With the Location Information of Electrodes. IEEE Access 8, 3197–3211 (2019). https://doi.org/10.1109/ACCESS.2019.2962740

    Article  Google Scholar 

  28. Li, M.a., Luo, X.y., Yang, J.f.: Extracting the nonlinear features of motor imagery EEG using parametric t-SNE. Neurocomputing 218, 371–381 (2016). 10.1016/j.neucom.2016.08.083

    Google Scholar 

  29. Liu, T., Yang, D.: A Densely Connected Multi-Branch 3D Convolutional Neural Network for Motor Imagery EEG Decoding. Brain Sci. 11(2), 197 (2021). https://doi.org/10.3390/brainsci11020197

    Article  Google Scholar 

  30. Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. In: ICCV. pp. 9992–10002 (2021). 10.1109/ICCV48922.2021.00986

    Google Scholar 

  31. Luo, T.j., Zhou, C.l., Chao, F.: Exploring spatial-frequency-sequential relationships for motor imagery classification with recurrent neural network. BMC Bioinf. 19(1) (2018). 10.1186/s12859-018-2365-1

    Google Scholar 

  32. López-Larraz, E., Sarasola-Sanz, A., Irastorza-Landa, N., Birbaumer, N., Ramos-Murguialday, A.: Brain-machine interfaces for rehabilitation in stroke: A review. NeuroRehabilitation 43(1), 77–97 (2018). https://doi.org/10.3233/nre-172394

    Article  Google Scholar 

  33. Ma, Q., Wang, M., Hu, L., Zhang, L., Hua, Z.: A Novel Recurrent Neural Network to Classify EEG Signals for Customers’ Decision-Making Behavior Prediction in Brand Extension Scenario. Front. Hum. Neurosci. 15, 610890 (2021). https://doi.org/10.3389/fnhum.2021.610890

    Article  Google Scholar 

  34. Ma, X., Qiu, S., Wei, W., Wang, S., He, H.: Deep Channel-Correlation Network for Motor Imagery Decoding From the Same Limb. IEEE Trans. Neural Syst. Rehabil. Eng. 28(1), 297–306 (2020). https://doi.org/10.1109/TNSRE.2019.2953121

    Article  Google Scholar 

  35. Van der Maaten, L., Hinton, G.: Visualizing Data using t-SNE. JMLR 9(86), 2579–2605 (2008)

    Google Scholar 

  36. McInnes, L., Healy, J., Saul, N., Großberger, L.: UMAP: Uniform Manifold Approximation and Projection. J. Open Source Softw. 3(29), 861 (2018). 10.21105/joss.00861

    Google Scholar 

  37. Olivas-Padilla, B.E., Chacon-Murguia, M.I.: Classification of multiple motor imagery using deep convolutional neural networks and spatial filters. Appl. Soft Comput. 75, 461–472 (2019). https://doi.org/10.1016/j.asoc.2018.11.031

    Article  Google Scholar 

  38. Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine Learning in Python. JMLR 12, 2825–2830 (2011)

    MathSciNet  Google Scholar 

  39. Planelles, D., Hortal, E., Costa, Á., Úbeda, A., Iáñez, E., Azorín, J.M.: Evaluating Classifiers to Detect Arm Movement Intention from EEG Signals. Sensors 14(10), 18172–18186 (2014). https://doi.org/10.3390/s141018172

    Article  Google Scholar 

  40. Roots, K., Muhammad, Y., Muhammad, N.: Fusion Convolutional Neural Network for Cross-Subject EEG Motor Imagery Classification. Computers 9(3), 72 (2020). https://doi.org/10.3390/computers9030072

    Article  Google Scholar 

  41. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. Int. J. Comput. Vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y

    Article  MathSciNet  Google Scholar 

  42. Schalk, G., McFarland, D., Hinterberger, T., Birbaumer, N., Wolpaw, J.: BCI2000: a general-purpose brain-computer interface (BCI) system. IEEE Trans. Biomed. Eng. 51(6), 1034–1043 (2004). https://doi.org/10.1109/TBME.2004.827072

    Article  Google Scholar 

  43. Song, Y., Jia, X., Yang, L., Xie, L.: Transformer-based Spatial-Temporal Feature Learning for EEG Decoding. arXiv preprint arXiv:2106.11170 (2021). 10.48550/arXiv.2106.11170

  44. Sun, J., Cao, R., Zhou, M., Hussain, W., Wang, B., Xue, J., Xiang, J.: A hybrid deep neural network for classification of schizophrenia using EEG Data. Sci. Rep. 11(1), 4706 (2021)

    Article  Google Scholar 

  45. Tabar, Y.R., Halici, U.: A novel deep learning approach for classification of EEG motor imagery signals. J. Neural Eng. 14(1), 016003 (2016). https://doi.org/10.1088/1741-2560/14/1/016003

    Article  Google Scholar 

  46. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is All you Need. In: NeurIPS. vol. 30 (2017)

    Google Scholar 

  47. Wairagkar, M., Hayashi, Y., Nasuto, S.J.: Modeling the Ongoing Dynamics of Short and Long-Range Temporal Correlations in Broadband EEG During Movement. Front. Syst. Neurosci. 13, 66 (2019). https://doi.org/10.3389/fnsys.2019.00066

    Article  Google Scholar 

  48. Wairagkar, M., Hayashi, Y., Nasuto, S.J.: Dynamics of Long-Range Temporal Correlations in Broadband EEG During Different Motor Execution and Imagery Tasks. Front. Neurosci. 15 (2021). 10.3389/fnins.2021.660032

    Google Scholar 

  49. Wang, P., Jiang, A., Liu, X., Shang, J., Zhang, L.: LSTM-Based EEG Classification in Motor Imagery Tasks. IEEE Trans. Neural Syst. Rehabil. Eng. 26(11), 2086–2095 (2018). https://doi.org/10.1109/TNSRE.2018.2876129

    Article  Google Scholar 

  50. Wang, Q., Li, B., Xiao, T., Zhu, J., Li, C., Wong, D.F., Chao, L.S.: Learning Deep Transformer Models for Machine Translation. In: ACCL (2019). 10.18653/v1/p19-1176

    Google Scholar 

  51. Wang, W., Dai, J., Chen, Z., Huang, Z., Li, Z., Zhu, X., Hu, X., Lu, T., Lu, L., Li, H., et al.: InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions. In: CVPR. pp. 14408–14419 (2023). 10.1109/cvpr52729.2023.01385

    Google Scholar 

  52. Wang, Z., Cao, L., Zhang, Z., Gong, X., Sun, Y., Wang, H.: Short time Fourier transformation and deep neural networks for motor imagery brain computer interface recognition. Concurrency Comput. Pract. Exp. 30(23), e4413 (2018). https://doi.org/10.1002/cpe.4413

    Article  Google Scholar 

  53. Wei, Y., Liu, Y., Li, C., Cheng, J., Song, R., Chen, X.: TC-Net: A Transformer Capsule Network for EEG-based emotion recognition. Comput. Biol. Med. 152, 106463 (2023). https://doi.org/10.1016/j.compbiomed.2022.106463

    Article  Google Scholar 

  54. Xie, J., Zhang, J., Sun, J., Ma, Z., Qin, L., Li, G., Zhou, H., Zhan, Y.: A Transformer-Based Approach Combining Deep Learning Network and Spatial-Temporal Information for Raw EEG Classification. IEEE Trans. Neural Syst. Rehabil. Eng. 30, 2126–2136 (2022). https://doi.org/10.1109/TNSRE.2022.3194600

    Article  Google Scholar 

  55. Xu, B., Zhang, L., Song, A., Wu, C., Li, W., Zhang, D., Xu, G., Li, H., Zeng, H.: Wavelet Transform Time-Frequency Image and Convolutional Network-Based Motor Imagery EEG Classification. IEEE Access 7, 6084–6093 (2018). https://doi.org/10.1109/ACCESS.2018.2889093

    Article  Google Scholar 

  56. Yang, J., Yao, S., Wang, J.: Deep Fusion Feature Learning Network for MI-EEG Classification. IEEE Access 6, 79050–79059 (2018). https://doi.org/10.1109/ACCESS.2018.2877452

    Article  Google Scholar 

  57. Yu, W., Luo, M., Zhou, P., Si, C., Zhou, Y., Wang, X., Feng, J., Yan, S.: MetaFormer is Actually What You Need for Vision. In: CVPR. pp. 10819–10829 (2022). 10.1109/cvpr52688.2022.01055

    Google Scholar 

  58. Yu, Y., Zhou, Z., Yin, E., Jiang, J., Tang, J., Liu, Y., Hu, D.: Toward brain-actuated car applications: Self-paced control with a motor imagery-based brain-computer interface. Comput. Biol. Med. 77, 148–155 (2016). https://doi.org/10.1016/j.compbiomed.2016.08.010

    Article  Google Scholar 

  59. Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. In: ICCV. pp. 6023–6032 (2019)

    Google Scholar 

  60. Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond Empirical Risk Minimization. In: ICLR (2017)

    Google Scholar 

  61. Zhang, X., Yao, L., Kanhere, S.S., Liu, Y., Gu, T., Chen, K.: MindID: Person Identification from Brain Waves through Attention-based Recurrent Neural Network. IMWUT 2(3), 1–23 (2018). https://doi.org/10.1145/3264959

    Article  Google Scholar 

  62. Zhao, X., Zhao, J., Liu, C., Cai, W.: Deep Neural Network with Joint Distribution Matching for Cross-Subject Motor Imagery Brain-Computer Interfaces. Biomed. Res. Int. 2020, 1–15 (2020). https://doi.org/10.1155/2020/7285057

    Article  Google Scholar 

  63. Zhao, X., Zhang, H., Zhu, G., You, F., Kuang, S., Sun, L.: A Multi-Branch 3D Convolutional Neural Network for EEG-Based Motor Imagery Classification. IEEE Trans. Neural Syst. Rehabil. Eng. 27(10), 2164–2177 (2019). https://doi.org/10.1109/TNSRE.2019.2938295

    Article  Google Scholar 

  64. Zhou, B., Lapedriza, A., Khosla, A., Oliva, A., Torralba, A.: Places: A 10 Million Image Database for Scene Recognition. IEEE Trans. Pattern Anal. Mach. Intell. 40(6), 1452–1464 (2018). https://doi.org/10.1109/TPAMI.2017.2723009

    Article  Google Scholar 

  65. Zhu, K., Wang, S., Zheng, D., Dai, M.: Study on the effect of different electrode channel combinations of motor imagery EEG signals on classification accuracy. J. Eng. 2019(23), 8641–8645 (2019). https://doi.org/10.1049/joe.2018.9073

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science, Graduate School of Science and Technology, Meiji University, Kawasaki, Japan

    Takuto Fukushima

  2. Department of Computer Science, School of Science and Technology, Meiji University, Kawasaki, Japan

    Ryusuke Miyamoto

Authors
  1. Takuto Fukushima
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ryusuke Miyamoto
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Ryusuke Miyamoto .

Editor information

Editors and Affiliations

  1. Pohang University of Science and Technology (POSTECH), Pohang, Korea (Republic of)

    Minsu Cho

  2. Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, United Arab Emirates

    Ivan Laptev

  3. Google, Mountain View, CA, USA

    Du Tran

  4. National University of Singapore, Singapore, Singapore

    Angela Yao

  5. Peking University, Beijing, China

    Hongbin Zha

1 Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary material 1 (pdf 182 KB)

Rights and permissions

Reprints and permissions

Copyright information

© 2025 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Fukushima, T., Miyamoto, R. (2025). Spatiotemporal Pooling on Appropriate Topological Maps Represented as Two-Dimensional Images for EEG Classification. In: Cho, M., Laptev, I., Tran, D., Yao, A., Zha, H. (eds) Computer Vision – ACCV 2024. ACCV 2024. Lecture Notes in Computer Science, vol 15473. Springer, Singapore. https://doi.org/10.1007/978-981-96-0901-7_4

Download citation

  • DOI: https://doi.org/10.1007/978-981-96-0901-7_4

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-96-0900-0

  • Online ISBN: 978-981-96-0901-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation