[go: up one dir, main page]
More Web Proxy on the site http://driver.im/ Skip to main content

Advertisement

Springer Nature Link
Log in

Modeling and classification of voluntary and imagery movements for brain–computer interface from fNIR and EEG signals through convolutional neural network

  • Research
  • Published:
Health Information Science and Systems Aims and scope Submit manuscript

Abstract

Practical brain–computer interface (BCI) demands the learning-based adaptive model that can handle diverse problems. To implement a BCI, usually functional near-infrared spectroscopy (fNIR) is used for measuring functional changes in brain oxygenation and electroencephalography (EEG) for evaluating the neuronal electric potential regarding the psychophysiological activity. Since the fNIR modality has an issue of temporal resolution, fNIR alone is not enough to achieve satisfactory classification accuracy as multiple neural stimuli are produced by voluntary and imagery movements. This leads us to make a combination of fNIR and EEG with a view to developing a BCI model for the classification of the brain signals of the voluntary and imagery movements. This work proposes a novel approach to prepare functional neuroimages from the fNIR and EEG using eight different movement-related stimuli. The neuroimages are used to train a convolutional neural network (CNN) to formulate a predictive model for classifying the combined fNIR–EEG data. The results reveal that the combined fNIR–EEG modality approach along with a CNN provides improved classification accuracy compared to a single modality and conventional classifiers. So, the outcomes of the proposed research work will be very helpful in the implementation of the finer BCI system.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
£29.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (United Kingdom)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13
Fig. 14
Fig. 15
Fig. 16
Fig. 17
Fig. 18

Similar content being viewed by others

References

  1. Fantini S. Dynamic model for the tissue concentration and oxygen saturation of hemoglobin in relation to blood volume, flow velocity, and oxygen consumption: implications for functional neuroimaging and coherent hemodynamics spectroscopy (CHS). NeuroImage. 2014;85:202–21. https://doi.org/10.1016/j.neuroimage.2013.03.065.

    Article  Google Scholar 

  2. Baker JM, Bruno JL, Gundran A, Hadi Hosseini SM, Reiss AL. fNIRS measurement of cortical activation and functional connectivity during a visuospatial working memory task. PLoS ONE. 2018;13(8):1–22. https://doi.org/10.1371/journal.pone.0201486.

    Article  Google Scholar 

  3. Shin J, Lühmann AV, Kim DW, Mehnert J, Hwang HJ, Müller KR. Data descriptor: simultaneous acquisition of EEG and NIRS during cognitive tasks for an open access dataset. Sci Data. 2018;5(180003):1–16. https://doi.org/10.1038/sdata.2018.3.

    Article  Google Scholar 

  4. Aghajani H, Garbey M, Omurtag A. Measuring mental workload with EEG + fNIRS. Front Hum Neurosci. 2017;11(359):1–20. https://doi.org/10.3389/fnhum.2017.00359.

    Article  Google Scholar 

  5. Hong K, Khan MJ, Hong MJ. Feature extraction and classification methods for hybrid fNIRS-EEG brain-computer interfaces. Front Hum Neurosci. 2018;12:246. https://doi.org/10.3389/fnhum.2018.00246.

    Article  Google Scholar 

  6. Abdulkader SN, Atia A, Mostafa MSM. Brain computer interfacing: applications and challenges. Egypt Inform J. 2015;16:213–30. https://doi.org/10.1016/j.eij.2015.06.002.

    Article  Google Scholar 

  7. Burle B, Spieser L, Roger C, Casini L, Hasbroucq T, Vidal F. Spatial and temporal resolutions of EEG: is it really black and white? A scalp current density view. Int J Psychophysiol. 2015;97(3):210–20. https://doi.org/10.1016/j.ijpsycho.2015.05.004.

    Article  Google Scholar 

  8. Basic principles of magnetoencephalography. MIT Class Notes. http://web.mit.edu/kitmitmeg/whatis.html. Accessed 24 Oct 2006.

  9. Ariely D, Berns GS. Neuromarketing: the hope and hype of neuroimaging in business. Nat Rev Neurosci. 2010;11:284–92. https://doi.org/10.1038/nrn2795.

    Article  Google Scholar 

  10. Ayaz H, Onaral B, Izzetoglu K, Shewokis PA, McKendrick R, Parasuraman R. Continuous monitoring of brain dynamics with functional near infrared spectroscopy as a tool for neuro ergonomic research: empirical examples and a technological development. Front Hum Neurosci. 2013;7(871):1–13. https://doi.org/10.3389/fnhum.2013.00871.

    Article  Google Scholar 

  11. Ernst LH, Plichta MM, Lutz E, Zesewitz AK, Tupak SV, Dresler T, Ehlis AC, Fallgatter AJ. Prefrontal activation patterns of automatic and regulated approach avoidance reactions: a functional near-infrared spectroscopy (fNIRS) study. Cortex. 2013;49(1):131–42. https://doi.org/10.1016/j.cortex.2011.09.013.

    Article  Google Scholar 

  12. Jöbsis FF. Noninvasive infrared monitoring of cerebral and myocardial oxygen sufficiency and circulatory parameters. Science. 1977;198(4323):1264–7. https://doi.org/10.1126/science.929199.

    Article  Google Scholar 

  13. Ferrari M, Quaresima V. A brief review on the history of human functional near-infrared spectroscopy (fNIRS) development and fields of application. NeuroImage. 2012;63(2):921–35. https://doi.org/10.1016/j.neuroimage.2012.03.049.

    Article  Google Scholar 

  14. Ferrari M, Giannini I, Carpi A, Fasella P, Fieschi C, Zanette E. Non-invasive infrared monitoring of tissue oxygenation and circulatory parameters. In: XII world congress of angiology 1980, Athens, September 7–12.

  15. Giannini I, Ferrari M, Carpi A, Fasella P. Rat brain monitoring by near-infrared spectroscopy: an assessment of possible clinical significance. Physiol Chem Phys. 1982;14(3):295–305.

    Google Scholar 

  16. Cui X, Bray S, Bryant DM, Glover GH, Reiss AL. A quantitative comparison of NIRS and fMRI across multiple cognitive tasks. NeuroImage. 2011;54(4):2808–21. https://doi.org/10.1016/j.neuroimage.2010.10.069.

    Article  Google Scholar 

  17. Noah JA, Ono Y, Nomot Y, Shimada S, Tachibana A, Zhang X, Bronner S, Hirsch J. fMRI validation of fNIRS measurements during a naturalistic task. J Vis Exp. 2015. https://doi.org/10.3791/52116.

    Article  Google Scholar 

  18. Allison BZ, Brunner C, Kaiser V, Muller-Putz GR, Neuper C, Pfurtscheller G. Toward a hybrid brain-computer interface based on imagined movement and visual attention. J Neural Eng. 2010. https://doi.org/10.1088/1741-2560/7/2/026007.

    Article  Google Scholar 

  19. Pfurtscheller G, Allison BZ, Brunner C, Bauernfeind G, Solis-Escalante T, Scherer R, Zander TO, Mueller-Putz G, Neuper C, Birbaumer N. The hybrid BCI. Front Neurosci. 2010. https://doi.org/10.3389/fnpro.2010.00003.

    Article  Google Scholar 

  20. Fazli S, Mehnert J, Steinbrink J, Curio G, Villringer A, Müller KR, Blankertz B. Enhanced performance by a hybrid NIRS–EEG brain-computer interface. NeuroImage. 2012;59(1):519–29. https://doi.org/10.1016/j.neuroimage.2011.07.084.

    Article  Google Scholar 

  21. Lee MH, Fazli S, Mehnert J, Lee SW. Improving the performance of brain-computer interface using multi-modal neuroimaging. In: 2nd IAPR Asian conference on pattern recognition, Naha, 2013, pp. 511–15. https://doi.org/10.1109/acpr.2013.132.

  22. Lee MH, Fazli S, Mehnert J, Lee SW. Subject-dependent classification for robust idle state detection using multi-modal neuroimaging and data-fusion techniques in BCI. Pattern Recognit. 2015;48(8):2725–37. https://doi.org/10.1016/j.patcog.2015.03.010.

    Article  Google Scholar 

  23. Buccino P, Keles HO, Omurtag A. Hybrid EEG-fNIRS asynchronous brain-computer interface for multiple motor tasks. PLoS ONE. 2016. https://doi.org/10.1371/journal.pone.0146610.

    Article  Google Scholar 

  24. World Bank and WHO. World report on disability. World Bank and WHO; 2015. http://www.who.int/disabilities/world_report/2011/report.pdf.

  25. Batula M, Kim YE, Ayaz H. Virtual and actual humanoid robot control with four-class motor-imagery-based optical brain-computer interface. Comput Intell Neurosci. 2017;2017(1463512):1–13. https://doi.org/10.1155/2017/1463512.

    Article  Google Scholar 

  26. Batula AM, Ayaz H, Kim YE. Evaluating a four-class motor-imagery-based optical brain-computer interface. In: 36th annual international conference of the IEEE engineering in medicine and biology society, Chicago, IL, 2014, pp. 2000–03. https://doi.org/10.1109/embc.2014.6944007.

  27. Abbas W, Khan NA. FBCSP-based multi-class motor imagery classification using BP and TDP features. In: 2018 40th annual international conference of the IEEE engineering in medicine and biology society (EMBC), Honolulu, HI, 2018, pp. 215–18.

  28. Mahmood A, Zainab R, Ahmad RB, Saeed M, Kamboh AM. Classification of multi-class motor imagery EEG using four band common spatial pattern. In: Annual international conference of the IEEE engineering in medicine and biology society (EMBC), Seogwipo, 2017, pp. 1034–37. https://doi.org/10.1109/embc.2017.8037003.

  29. Mishra PK, Jagadish B, Kiran MPRS, Rajalakshmi P, Reddy DS. A novel classification for EEG based four class motor imagery using kullback-leibler regularized Riemannian manifold. In: 2018 IEEE 20th international conference on e-Health networking, applications and services (Healthcom), Ostrava, 2018, pp. 1–5. https://doi.org/10.1109/healthcom.2018.8531086.

  30. Ge S, Wang R, Yu D. Classification of four-class motor imagery employing single-channel electroencephalography. PLoS ONE. 2014;9(6):1–7. https://doi.org/10.1371/journal.pone.0098019.

    Article  Google Scholar 

  31. León CL. Multilabel classification of EEG-based combined motor imageries implemented for the 3D control of a robotic arm. PhD thesis, Université de Lorraine, 2017.

  32. Rahman MA. Matlab based graphical protocol. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/69162-matlab-based-graphical-protocol. Accessed 20 Oct 2018.

  33. World medical association declaration of Helsinki-ethical principles for medical research involving human subjects. Adopted by 64th WMA General Assembly, Fortaleza, Brazil, Special Communication: Clinical Review & Education, 2013.

  34. Rahman MA, Rashid MA, Ahmad M. Selecting the optimal conditions of Savitzky-Golay filter for fNIRS signal. Biocybern Biomed Eng. 2019;39(3):624–37. https://doi.org/10.1016/j.bbe.2019.06.004.

    Article  Google Scholar 

  35. Wiriessnegger SC, Kurzmann J, Neuper C. Spatio-temporal differences in brain oxygenation between movement execution and imagery: a multichannel near-infrared spectroscopy study. Int J Psychophysiol. 2008;67(1):54–63. https://doi.org/10.1016/j.ijpsycho.2007.10.004.

    Article  Google Scholar 

  36. Batula AM, Mark JA, Kim YE, Ayaz H. Comparison of brain activation during motor imagery and motor movement using fNIRS. Comput Intell Neurosci. 2017. https://doi.org/10.1155/2017/5491296.

    Article  Google Scholar 

  37. Nyhof L. Biomedical signal filtering for noisy environments. PhD thesis, Deakin University, Australia, 2014. http://hdl.handle.net/10536/DRO/DU:30079016.

  38. Chavan MS, Agarwala R, Uplane MD. Digital elliptic filter application for noise reduction in ECG signal. WSEAS Trans Electron. 2006;3(1):210–6.

    Google Scholar 

  39. Lutovac MD, Tosic DV, Evans BL. Filter design for signal processing. Upper Saddle River: Prentice Hall; 2001.

    Google Scholar 

  40. Vlcek M, Unbehauen R. Degree, ripple, and transition width of elliptic filters. IEEE Trans Circ Syst. 1989;36(3):469–72. https://doi.org/10.1109/31.17602.

    Article  MathSciNet  Google Scholar 

  41. Orfanidis SJ. Introduction to signal processing. Upper Saddle River: Prentice Hall; 1996.

    Google Scholar 

  42. Mammone N, Morabito FC. Enhanced automatic wavelet independent component analysis for electroencephalographic artifact removal. Entropy. 2014;16(12):6553–72. https://doi.org/10.3390/e16126553.

    Article  Google Scholar 

  43. Khanam F, Rahman MA, Ahmad M. Evaluating alpha relative power of EEG signal during psychophysiological activities in Salat. In: International conference on innovations in science, engineering and technology (ICISET), 2018, Bangladesh, pp. 1–6. https://doi.org/10.1109/iciset.2018.8745614.

  44. Ifeachor EC, Jervis BW. Digital signal processing: a practical approach. Boston: Addison Wesley; 1993.

    Google Scholar 

  45. Rahman MA, Haque MM, Anjum A, Mollah MN, Ahmad M. Classification of motor imagery events from prefrontal hemodynamics for BCI application. Algorithms for intelligent system. Singapore: Springer; 2018. https://doi.org/10.1007/978-981-13-7564-4_2.

    Book  Google Scholar 

  46. Zhijie B, Qiuli L, Lei W, Chengbiao L, Shimin Y, Xiaoli L. Relative power and coherence of EEG series are related to amnestic mild cognitive impairment in diabetes. Front Aging Neurosci. 2014. https://doi.org/10.3389/fnagi.2014.00011.

    Article  Google Scholar 

  47. Krizhevsky A, Sutskever I, Hinton GE. Imagenet classification with deep convolutional neural networks. Adv Neural Inf Process Syst. 2012. https://doi.org/10.1145/3065386.

    Article  Google Scholar 

  48. Matsugu M, Mori K, Mitari Y, Kaneda Y. Subject independent facial expression recognition with robust face detection using a convolutional neural network. Neural Netw. 2003. https://doi.org/10.1016/S0893-6080(03)00115-1.

    Article  Google Scholar 

  49. Rahman MA, Ahmad M. Lie detection from single feature of functional near infrared spectroscopic (fNIRS) signals. In: 2nd international conference on electrical & electronic engineering (ICEEE 2017), 27–29 December, Rajshahi University of Engineering & Technology (RUET), Rajshahi, Bangladesh. https://doi.org/10.1109/ceee.2017.8412900.

  50. Rahman MA, Rashid MMO, Khanam F, Alam MK, Ahmad M. EEG based brain alertness monitoring by statistical and artificial neural network approach. Int J Adv Comput Sci Appl. 2019. https://doi.org/10.14569/ijacsa.2019.0100157.

    Article  Google Scholar 

  51. Rahman MA, Khanam F, Ahmad M. Detection of effective temporal window for classification of motor imagery events from prefrontal hemodynamics. In: International conference on electrical, computer and communication engineering (ECCE), Cox’s Bazar, Bangladesh, 2019. https://doi.org/10.1109/ecace.2019.8679317.

  52. Vakkuri A, Yli-Hankala A, Talja P, Mustola S, Tolvanen-Laakso H, Sampson T, Viertiö-Oja H. Time-frequency balanced spectral entropy as a measure of anesthetic drug effect in central nervous system during sevoflurane, propofol, and thiopental anesthesia. Acta Anaesthesiol Scand. 2004;48(2):145–53. https://doi.org/10.1111/j.0001-5172.2004.00323.x.

    Article  Google Scholar 

  53. Rahman MA. Topoplot for B-Alert X-10 9-channel EEG signal. MATLAB Central File Exchange. https://www.mathworks.com/matlabcentral/fileexchange/69991-topoplot-for-b-alert-x-10-9-channel-eeg-signal. Accessed 2 Apr 2019.

  54. Rahman MA, Hossain MK, Khanam F, Alam MK, Ahmad M. Four-class motor imagery EEG signal classification using PCA, wavelet, and two-stage neural network. Int J Adv Comput Sci Appl. 2019. https://doi.org/10.14569/ijacsa.2019.0100562.

    Article  Google Scholar 

  55. Shin J, Jeong J. Multiclass classification of hemodynamic responses for performance improvement of functional near-infrared spectroscopy-based brain–computer interface. J Biomed Opt. 2014;19(6):067009-1–9. https://doi.org/10.1117/1.jbo.19.6.067009.

    Article  Google Scholar 

Download references

Acknowledgements

This work was partially supported by the Higher Education Quality Enhancement Project (HEQEP), UGC, Bangladesh; under Subproject “Postgraduate Research in BME”, CP#3472, KUET, Bangladesh.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, Khulna University of Engineering & Technology (KUET), Khulna, Bangladesh

    Md. Asadur Rahman

  2. Department of Computer Science and Engineering, Jahangirnagar University, Dhaka, Bangladesh

    Mohammad Shorif Uddin

  3. Department of Electrical and Electronic Engineering, Khulna University of Engineering & Technology (KUET), Khulna, Bangladesh

    Mohiuddin Ahmad

Authors
  1. Md. Asadur Rahman
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Mohammad Shorif Uddin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Mohiuddin Ahmad
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Md. Asadur Rahman.

Ethics declarations

Conflict of interest

This research work has no conflict of interest to anyone.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix

Appendix

figure a

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Rahman, M.A., Uddin, M.S. & Ahmad, M. Modeling and classification of voluntary and imagery movements for brain–computer interface from fNIR and EEG signals through convolutional neural network. Health Inf Sci Syst 7, 22 (2019). https://doi.org/10.1007/s13755-019-0081-5

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1007/s13755-019-0081-5

Keywords

Search

Navigation