CN113918008B - Brain-computer interface system based on source space brain magnetic signal decoding and application method - Google Patents
Brain-computer interface system based on source space brain magnetic signal decoding and application method Download PDFInfo
- Publication number
- CN113918008B CN113918008B CN202111004557.9A CN202111004557A CN113918008B CN 113918008 B CN113918008 B CN 113918008B CN 202111004557 A CN202111004557 A CN 202111004557A CN 113918008 B CN113918008 B CN 113918008B
- Authority
- CN
- China
- Prior art keywords
- brain
- signals
- magnetic
- magnetoencephalography
- signal
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 210000004556 brain Anatomy 0.000 title claims abstract description 91
- 238000000034 method Methods 0.000 title claims abstract description 21
- 238000002582 magnetoencephalography Methods 0.000 claims abstract description 42
- 230000008035 nerve activity Effects 0.000 claims abstract description 23
- 238000010223 real-time analysis Methods 0.000 claims abstract description 12
- 230000000638 stimulation Effects 0.000 claims abstract description 10
- 210000005036 nerve Anatomy 0.000 claims abstract description 9
- 238000007781 pre-processing Methods 0.000 claims abstract description 7
- 238000012545 processing Methods 0.000 claims abstract description 3
- 230000001939 inductive effect Effects 0.000 claims abstract 2
- 230000001537 neural effect Effects 0.000 claims description 23
- 210000003128 head Anatomy 0.000 claims description 16
- 239000000523 sample Substances 0.000 claims description 15
- 238000013528 artificial neural network Methods 0.000 claims description 14
- 238000012549 training Methods 0.000 claims description 10
- 239000011159 matrix material Substances 0.000 claims description 8
- 230000008569 process Effects 0.000 claims description 8
- 230000004886 head movement Effects 0.000 claims description 7
- 230000033001 locomotion Effects 0.000 claims description 7
- 210000004761 scalp Anatomy 0.000 claims description 7
- 238000010586 diagram Methods 0.000 claims description 6
- 125000004122 cyclic group Chemical group 0.000 claims description 5
- 230000006698 induction Effects 0.000 claims description 5
- 210000003625 skull Anatomy 0.000 claims description 5
- 230000001054 cortical effect Effects 0.000 claims description 4
- 238000001914 filtration Methods 0.000 claims description 4
- 210000004884 grey matter Anatomy 0.000 claims description 4
- 210000000976 primary motor cortex Anatomy 0.000 claims description 4
- 230000000306 recurrent effect Effects 0.000 claims description 4
- 238000012360 testing method Methods 0.000 claims description 4
- 230000033228 biological regulation Effects 0.000 claims description 3
- 230000007613 environmental effect Effects 0.000 claims description 3
- 238000002360 preparation method Methods 0.000 claims description 3
- 238000000926 separation method Methods 0.000 claims description 3
- 238000012546 transfer Methods 0.000 claims description 3
- 230000002567 autonomic effect Effects 0.000 claims description 2
- 230000004424 eye movement Effects 0.000 claims description 2
- 238000012880 independent component analysis Methods 0.000 claims description 2
- 230000003183 myoelectrical effect Effects 0.000 claims description 2
- 230000002490 cerebral effect Effects 0.000 claims 1
- 238000012856 packing Methods 0.000 claims 1
- 238000000605 extraction Methods 0.000 abstract description 4
- 238000004458 analytical method Methods 0.000 abstract description 3
- 230000001149 cognitive effect Effects 0.000 description 6
- 241000238366 Cephalopoda Species 0.000 description 4
- 230000003993 interaction Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000002610 neuroimaging Methods 0.000 description 3
- 238000005057 refrigeration Methods 0.000 description 3
- 230000003925 brain function Effects 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 229910052734 helium Inorganic materials 0.000 description 2
- 239000001307 helium Substances 0.000 description 2
- SWQJXJOGLNCZEY-UHFFFAOYSA-N helium atom Chemical compound [He] SWQJXJOGLNCZEY-UHFFFAOYSA-N 0.000 description 2
- 238000003384 imaging method Methods 0.000 description 2
- 239000007788 liquid Substances 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000011160 research Methods 0.000 description 2
- 230000035945 sensitivity Effects 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 208000000094 Chronic Pain Diseases 0.000 description 1
- 208000019022 Mood disease Diseases 0.000 description 1
- 208000021384 Obsessive-Compulsive disease Diseases 0.000 description 1
- 208000002193 Pain Diseases 0.000 description 1
- 206010033799 Paralysis Diseases 0.000 description 1
- 208000028017 Psychotic disease Diseases 0.000 description 1
- 208000020339 Spinal injury Diseases 0.000 description 1
- 230000001133 acceleration Effects 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 150000001340 alkali metals Chemical group 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000002238 attenuated effect Effects 0.000 description 1
- 230000002457 bidirectional effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000007177 brain activity Effects 0.000 description 1
- 230000037410 cognitive enhancement Effects 0.000 description 1
- 230000006998 cognitive state Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 230000001276 controlling effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000002996 emotional effect Effects 0.000 description 1
- 230000001037 epileptic effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000005358 geomagnetic field Effects 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 238000002513 implantation Methods 0.000 description 1
- 230000004807 localization Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000004007 neuromodulation Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000004806 packaging method and process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 230000008054 signal transmission Effects 0.000 description 1
- 239000013589 supplement Substances 0.000 description 1
- 208000024891 symptom Diseases 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
- G06F3/015—Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/242—Detecting biomagnetic fields, e.g. magnetic fields produced by bioelectric currents
- A61B5/245—Detecting biomagnetic fields, e.g. magnetic fields produced by bioelectric currents specially adapted for magnetoencephalographic [MEG] signals
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7203—Signal processing specially adapted for physiological signals or for diagnostic purposes for noise prevention, reduction or removal
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7225—Details of analogue processing, e.g. isolation amplifier, gain or sensitivity adjustment, filtering, baseline or drift compensation
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
Landscapes
- Health & Medical Sciences (AREA)
- Engineering & Computer Science (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- General Health & Medical Sciences (AREA)
- Signal Processing (AREA)
- Biomedical Technology (AREA)
- Surgery (AREA)
- Veterinary Medicine (AREA)
- Public Health (AREA)
- Biophysics (AREA)
- Pathology (AREA)
- Animal Behavior & Ethology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Molecular Biology (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physiology (AREA)
- Psychiatry (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Neurology (AREA)
- Mathematical Physics (AREA)
- Dermatology (AREA)
- Fuzzy Systems (AREA)
- Neurosurgery (AREA)
- Evolutionary Computation (AREA)
- Human Computer Interaction (AREA)
- General Physics & Mathematics (AREA)
- Power Engineering (AREA)
- Measurement And Recording Of Electrical Phenomena And Electrical Characteristics Of The Living Body (AREA)
- Magnetic Resonance Imaging Apparatus (AREA)
Abstract
The invention discloses a brain-computer interface system based on source space brain magnetic signal decoding and an application method thereof. The system comprises a magnetoencephalography signal acquisition device, a data acquisition workstation and a magnetoencephalography signal acquisition device, wherein the magnetoencephalography signal acquisition device is used for being worn on the head of a subject, acquiring magnetoencephalography signals of the subject and sending the magnetoencephalography signals to the data acquisition workstation; the data acquisition workstation is used for synchronously receiving the multichannel magnetoencephalography signals acquired by the magnetoencephalography signal acquisition device and sending the multichannel magnetoencephalography signals to the real-time analysis workstation; the real-time analysis workstation is used for carrying out real-time preprocessing, tracing and decoding on the received brain magnetic signals and sending decoding information to the multi-mode stimulation presentation device and the external controlled equipment; the multi-mode stimulation presentation device is used for presenting stimulation information for inducing the brain nerve activity of the subject or analyzing nerve feedback signals decoded by the workstation in real time; and the external controlled device is used for carrying out corresponding processing according to the received decoding signal. The invention can realize the real-time extraction, pretreatment and traceability analysis of the multichannel high-flux full brain nerve activity magnetic signals.
Description
Technical Field
The invention relates to the field of brain-computer interface research, in particular to a brain-computer interface system for realizing direct interaction with the outside and closed-loop nerve regulation and control by decoding a brain magnetic signal after tracing and a use method thereof.
Background
By detecting and decoding Brain neural activity, brain-computer interface (Brain-Computer Interface, BCI) can directly establish a connection path between the human Brain and external equipment, and a neural signal closed loop for Brain-computer interaction is constructed. With the rapid development of the wearable brain imaging device towards miniaturization and portability, the brain-computer interface has become a hotspot of international science and industry layout, and is an important competitive field of the core high strength of various science and technology nations.
Early brain-computer interface applications were primarily unidirectional connections from the human brain to external devices, such as to assist in reestablishing connections to the outside world for persons with impaired motor abilities or severe disabilities (e.g., stroke, paralyzed patients with spinal injuries), typically by decoding neural activity of the motor sensory cortex of the brain to manipulate the external devices, thereby achieving interaction with the outside world. In recent years, the rapid advance of brain imaging and artificial intelligence technology greatly expands the application scene of brain-computer interface technology, so that the brain-computer interface application is increased from feeling and perception to a high-level cognitive activity level. By constructing the real-time bidirectional connection between the human brain and the computer, the direct control of the human brain on external equipment can be realized, and conversely, the brain activity can be monitored and regulated. By automatically monitoring advanced cognitive activities and states of the human brain (including internal speech, emotional tendency, cognitive load, and attention states, etc.). In addition, with closed loop nerve feedback (closed-loop neurofeedback), brain-machine interface technology can also be used for nerve rehabilitation, improving psychotic symptoms (obsessive-compulsive disorder, mood disorders, chronic pain) and cognitive enhancement. Implementing these advanced brain-computer interface applications requires accurate localization, extraction and decoding of neural activity signals of the relevant brain region or brain network in real time throughout the brain.
The brain function imaging technology is the basis of brain-computer interfaces, and accurately detecting the brain nerve activity with high complexity and dynamic characteristics is a precondition for realizing high-efficiency human-computer interaction. Clinically, the placement/implantation of electrodes in the subcolumns of the brain to record neural activity, while providing the advantage of high temporal and spatial resolution, is only useful as an invasive measurement for a specific patient population (e.g., epileptic patients). Furthermore, due to clinical requirements and safety considerations, implanted electrodes (arrays) can only cover local brain regions, and synchronous recording of full brain neural activity cannot be achieved, which is difficult to use for brain-computer interface applications involving advanced cognitive activities in multiple brain regions. The electroencephalogram can directly record the electrical signals generated by the brain nerve activity outside the scalp in a noninvasive manner, and has the characteristics of high time resolution, high wearability and good portability. However, due to the characteristics of poor conductivity and anisotropy of brain medium, the propagation of nerve activity electrical signals in the brain can be greatly distorted and lost, so that the signal-to-noise ratio of the electroencephalogram is low, the effective signal bandwidth is limited (the attenuation of signals above 50Hz is serious after passing through the skull), the spatial resolution is poor, the brain region where the nerve activity occurs is difficult to accurately trace back and the spatial information is provided, and the application scene of the brain-computer interface is limited. In contrast, the conduction of the magnetic signals of the brain nerve activity is hardly influenced by brain media, so that the specific position of the nerve activity can be traced relatively accurately, and the nerve activity decoding with both time and spatial resolution is completed in a signal source space. The noninvasive property, ultra-high time resolution and excellent spatial resolution after tracing of the magnetoencephalography make the magnetoencephalography an ideal brain function imaging technology for advanced application of brain-computer interfaces.
Traditional magnetoencephalography based on superconducting quantum interferometers (Superconducting Quantum INTERFERENCE DEVICE, SQUID) requires maintenance of working conditions for low-temperature superconductivity. Therefore, due to the liquid helium refrigeration equipment, the SQUID magnetoencephalography detector is fixed in position and cannot be flexibly adjusted to be close to the scalp, and the detected neural activity magnetic signals are seriously attenuated in the propagation process, so that the magnetoencephalography signals from weak neural activity and deep brain regions cannot be effectively detected. In addition, the bulky volume of liquid helium refrigeration equipment cannot meet the wearable portability requirements of brain-computer interfaces. In recent years, miniaturization of ultra-high sensitivity atomic magnetometers (Atomic Magnetometer, AM) has enabled portable wearable magnetoencephalography. The AM brain magnetic diagram can detect weak magnetic fields based on the action of alkali metal atoms and laser under the room temperature environment, and has higher sensitivity. In addition, the position of the probe on the AM brain magnetic image detector array can be flexibly adjusted to be close to the scalp because the probe is not limited by huge refrigeration equipment, so that the attenuation of brain magnetic signals in the propagation process is reduced to the greatest extent, and the effective detection of weak nerve activity and deep brain region nerve activity is realized. Meanwhile, the wearable AM magnetoencephalography system also allows the user to perform free movement, and is more suitable for real life scenes compared with SQUID magnetoencephalography.
In summary, as a non-invasive brain imaging technique covering the whole brain, the detection of brain neural activity by magnetoencephalography can compromise both temporal and spatial resolution. The brain-computer interface based on the wearable AM magnetoencephalography can be applied to clinical and cognitive neuroscience under complex situations, but no complete brain-computer interface system scheme and application method based on AM magnetoencephalography in source space decoding with practicability are provided at present.
Disclosure of Invention
In view of the above, the present invention aims to provide a brain-computer interface system based on source space magnetoencephalography signals and an application method thereof, which can realize real-time extraction, preprocessing and traceability analysis of multichannel high-flux magnetoencephalography signals, and realize various brain-computer interface applications by utilizing source space magnetoencephalography signals with space specificity and time resolution.
The invention adopts the technical scheme that:
A brain-computer interface system for decoding a brain magnetic signal based on a source space comprises the following modules:
The brain magnetic signal acquisition device comprises a detector for acquiring magnetic signals generated by brain nerve activity and matched devices (such as a digital signal processing acquisition card, a front/rear end amplifier and the like); the implementation mode of the brain magnetic signal acquisition device can be through a traditional superconducting quantum interferometer magnetometer or a new generation atomic magnetometer; the surface of the magnetoencephalography signal acquisition device is provided with a plurality of detectors, each detector provides a signal channel, and signals acquired by the detectors form a multichannel magnetoencephalography signal and are sent to the data acquisition workstation;
A magnetic shielding means for shielding an ambient magnetic field (for example, a geomagnetic field, an interference magnetic signal of an electronic apparatus) which is irrelevant to a brain neural activity;
The data acquisition workstation is used for synchronously receiving the multichannel magnetoencephalic signals acquired by the magnetoencephalic signal acquisition device, segmenting and packaging the continuous data stream and sending the continuous data stream to the real-time analysis workstation;
the real-time analysis workstation is used for carrying out real-time preprocessing, tracing and decoding on the brain magnetic signals and sending decoding information to the multi-mode stimulation presentation device;
A multi-modal stimulus presentation means for presenting stimulus information that induces brain neural activity to the subject or presenting neural feedback signals (such as intensity of neural activity of a specific brain region, control intention of the subject, etc.) decoded by the real-time analysis workstation, the stimulus information or feedback signals being presented by a plurality of modalities (e.g., visual, auditory, and tactile);
an external controlled device, an external device (e.g., a prosthetic, a voice synthesizer, etc.) controlled by the decoded neural signal.
The relation among the modules of the hardware system is shown in figure 1, and the signal transmission adopts TCP/IP protocol. Based on the system, not only can the control of external equipment be realized, but also a nerve feedback closed loop for nerve regulation/rehabilitation can be realized. In actual use, the implementation of each module can be flexibly adjusted according to actual conditions and requirements.
Preferably, the magnetoencephalography device uses a wearable magnetoencephalography of an atomic magnetometer based on the principle of spin-free exchange relaxation (Spin Exchange Relaxation Free, SERF), but the system is also applicable to traditional SQUID magnetoencephalography.
Preferably, the magnetic shielding device can use a multi-layered magnetic shielding barrel having a better shielding effect or a magnetic shielding room providing a free movement space. For wearable AM magnetoencephalography, an active compensation coil can be added on each AM probe to counteract the environmental remanence interference signal caused by motion.
Further, the data acquisition workstation synchronously integrates the received multichannel brain magnetic signals (sampling rate >1000Hz, channel number > 96) and the stimulation time sequence information in the buffer; the real-time analysis workstation adopts GPU acceleration optimization, and can realize neural decoding based on a deep neural network complex model.
Preferably, the multi-modal stimulation presentation device may simulate multi-sensory stimulation signals in a real environment in a virtual reality manner and perform neural feedback in the same manner.
Further, the steps of preprocessing the magnetoencephalography signals in real time and decoding the magnetoencephalography signals in the source space by the real-time analysis workstation are as follows (as shown in fig. 2):
1) Acquiring magnetic resonance head structure images of a user, and constructing a head model comprising scalp, the inner and outer surfaces of skull and cortical gray matter;
2) Constructing a source space of brain nerve activity on the basis of the head model, and calculating a transfer matrix from the source space to a probe array space by combining the relative positions of the head of a user and the brain magnetic map probe array;
3) Signal-space separation (Signal-space separation) of multichannel magnetoencephalic signals (in units of time windows of not more than 50 ms) received from a data acquisition workstation for removing environmental interference signals other than brain neural activity;
4) The multichannel brain magnetic signals are further preprocessed, including band-stop filtering for removing power frequency (50 Hz) in the time domain, band-pass filtering for extracting needed frequency components, independent component analysis for removing eye movement, electrocardio and myoelectric noise near the probe, and compensation for influence caused by head movement of a user by utilizing head position information recorded in real time;
5) Calculating noise covariance matrix of the probe according to the data of the non-sampled magnetoencephalography without a user, and carrying out real-time magnetoencephalography by adopting minimum norm estimation (minimum norm estimate, MNE) in combination with the transmission matrix;
6) The neural activity signal of the source space is decoded, the user's intention is extracted for controlling an external device, or the neural activity signal is directly fed back to the user for autonomic neuromodulation.
The brain-computer interface system based on source space brain magnetic signal decoding has the following advantages:
1. The system can realize real-time extraction, pretreatment and traceability analysis of the magnetic signals of the whole brain nerve activity, and the brain magnetic diagram is used as a non-invasive measurement means and is suitable for almost all healthy people and clinical patients;
2. The traced brain magnetic signals have space specificity and high time resolution, and the nerve decoding can be carried out in the source space, so that nerve activity signals of specific brain regions/networks or specific frequency components can be used in a targeted manner, the decoding accuracy is further improved, and the application range of brain-computer interfaces is expanded.
3. By combining the wearable magnetoencephalography with the virtual reality equipment, the system can restore the ecological effectiveness (ecological validity) of the use scene to the greatest extent, and realize the application requirements of various brain-computer interfaces such as the research of cognitive neuroscience, the auxiliary clinical nerve rehabilitation and the like in the real scene.
Drawings
FIG. 1 is a schematic diagram of a hardware system of the present invention;
FIG. 2 is a flow chart of the real-time tracing and decoding of the magnetoencephalography signals;
Fig. 3 is a schematic diagram of a brain-computer interface application of a source space magnetoencephalography signal real-time decoding language.
Detailed Description
In the following description, the brain-computer interface system of the present invention is further described in terms of specific embodiments to facilitate a more thorough understanding of the features and advantages of the present invention by those skilled in the art. It should be noted that the following description is merely representative of one exemplary application. It will be apparent that the invention is not limited to any particular structure, function, device, and method described herein, but may have other embodiments, or combinations of other embodiments. The software/hardware modules depicted in the present invention or as shown in the accompanying drawings may also be flexibly adapted as desired.
According to one embodiment of the invention, a brain-computer interface application for real-time decoding of language by source space magnetoencephalography signals, the application comprising three parts of a preparation phase, a training phase and an application phase.
Preparation stage
The brain magnetic cap carrying the AM probe is customized according to the shape of the head of a user. A high-definition magnetic resonance head structure image (spatial resolution of at least 1x 1cm 3) of a user is acquired, and a head model including scalp, inner and outer surfaces of skull and cortical gray matter is constructed by a boundary element method (Boundary Element Method, BEM). Signals (not less than 2 minutes) of the brain magnetic diagram when no load is acquired are used for calculating a probe array noise covariance matrix.
2000 Sentences used for daily conversations (single sentence is not more than 20 words) are extracted from the corpus, corresponding pinyin is transcribed for each sentence, and a training expectation set (for example, "what is you going in the morning", "ni zao shang qu na le.
Training phase
The user wears the brain magnetic cap (not less than 306 channels) carrying the AM probe in a magnetic shielding environment, and the head movement of the user is recorded in real time through a position coil on the brain magnetic cap and is used for correcting and compensating the influence of the head movement on the brain magnetic signals.
The training corpus is divided into 10 trials, each trial comprising 200 sentences and their corresponding pinyin. A sentence and a corresponding pinyin form are presented on a display each time in a visual input mode, a user looks at a screen, imagines pinyin letters of each word are written in sequence, each part of the body does not need to move, and brain nerve activity magnetic signals (namely brain magnetic signals) in the process are recorded in real time in the whole course of a brain magnetic map. Because the length of each sentence in the corpus is different, each sentence has no fixed imagination writing time, and the user feeds back to the next sentence through keys after finishing.
In addition to the training corpus described above, the user also needs to imagine writing 26 pinyin letters and four symbols (comma ",", period ",", space "_", and question mark ". Each character was repeatedly imagined to be written 50 times for a total of 1500 characters. The random scrambling of the sequence was followed by a 10 test run, each test run comprising 150 characters. The brain nerve activity magnetic signal of the character is recorded by the brain magnetic map in real time.
After all trials were completed, a magnetic signal of brain neural activity imagining 2000 sentences and 1450 characters was obtained. And extracting the time series of the magnetoencephalic signals (N voxels in total, as shown in figure 3) of each voxel in the primary motor cortex, the primary body induction cortex and the motor auxiliary area of the left and right hemispheres of the brain related to imagination writing after tracing the magnetoencephalic signals. Using the magnetoencephalic signal time series of each voxel in these brain regions (i.e. the magnetoencephalic signal time series of each voxel in the brain's left and right hemispheres, primary body induction cortex, and motion assistance region) a cyclic neural network (Recurrent Neural Network, RNN) can be trained, whose input is the magnetoencephalic signal time series in units of a 50 millisecond time window, and whose output is the probabilities p (X) of each of 26 pinyin letters and 4 punctuations, where X is any character. The training of this recurrent neural network is performed on-line.
Stage of actual use
In practical use, the wearing position of the magnetoencephalography cap cannot be strictly ensured to be identical every time, so that the system calibration is needed first. Specifically, after wearing the magnetoencephalic cap, a user imagines to write 26 pinyin letters and 4 punctuation marks one by one according to the screen prompt, and the magnetoencephalic signals acquired in the process are used for calibrating a circulating neural network trained in advance after tracing.
After the calibration is completed, the user can imagine writing the Chinese pinyin corresponding to any sentence to be expressed. The brain-computer interface system can record the magnetoencephalic signals of a user when the user imagines writing in real time, trace the source by taking a time window of 50 millimeters as a unit, and extract signals of each voxel in a primary motor cortex, a primary body induction cortex and a motion auxiliary region which are related to the imagining writing in a source space. The traced signals are immediately input into a trained cyclic neural network, so that the probability of writing a character in a current time window is obtained, and among all characters exceeding a probability threshold (for example, 0.4), the character with the highest probability is output (the system judges whether the current input is a new character according to the probability-time curve of each character). If there are no more characters than the probability threshold in the time window, no output is made.
Decoding of individual pinyin letters in the original signal output by the recurrent neural network may be erroneous, as shown by "nj_hao" in fig. 3. Therefore, the system further adopts a deep neural network language model pre-trained based on big data to correct pinyin obtained by decoding the cyclic neural network in real time, corrects individual characters with decoding errors, supplements pinyin tones by using context information, and finally outputs characters.
The above embodiments are merely illustrative of the principles of the present invention and its effectiveness, and are not intended to limit the invention. Modifications and variations may be made to the above-described embodiments by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, the scope of the invention is to be indicated by the appended claims.
Claims (8)
1. A brain-computer interface system based on source space magnetoencephalography signal decoding, comprising
The brain magnetic signal acquisition device is used for being worn on the head of a subject, acquiring brain magnetic signals generated by brain nerve activity of the subject and sending the brain magnetic signals to the data acquisition workstation;
The data acquisition workstation is used for synchronously receiving the multichannel magnetoencephalography signals acquired by the magnetoencephalography signal acquisition device and sending the multichannel magnetoencephalography signals to the real-time analysis workstation;
The real-time analysis workstation is used for carrying out real-time preprocessing, tracing and decoding on the received brain magnetic signals and sending decoding information to the multi-mode stimulation presentation device and the external controlled equipment; the method for preprocessing, tracing and decoding the brain magnetic signals in real time by the real-time analysis workstation comprises the following steps: 1) Acquiring magnetic resonance head structure images of a subject, and constructing a head model comprising scalp, inner and outer surfaces of skull and cortical gray matter; 2) Constructing a source space of brain nerve activity on the basis of a head model, and calculating a transfer matrix from the source space to a brain magnetic pattern probe array space by combining the relative positions of the head of a subject and the brain magnetic pattern probe array of a brain magnetic signal acquisition device; 3) Carrying out signal space separation on the multichannel brain magnetic signals to remove environmental interference signals except brain nerve activities; 4) Preprocessing multichannel brain magnetic signals, including removing band-stop filtering of power frequency in a time domain, extracting band-pass filtering of needed frequency components, removing independent component analysis of eye movement, electrocardio and myoelectric noise near a probe, and compensating influence of head movement of a subject by utilizing head position information recorded in real time; 5) Calculating a noise covariance matrix of a brain magnetic map probe of the brain magnetic signal acquisition device according to brain magnetic map data acquired in an empty condition without a subject, and carrying out real-time brain magnetic signal tracing by adopting minimum norm estimation in combination with the transfer matrix; 6) Decoding the neural activity signal of the source space, extracting decoding information for controlling external controlled equipment, or feeding back the neural activity signal obtained by decoding to the subject for autonomic regulation;
The multi-mode stimulation presentation device is used for presenting stimulation information for inducing the brain nerve activity of the subject or presenting nerve feedback signals decoded by the real-time analysis workstation;
And the external controlled device is used for carrying out corresponding processing according to the received decoding signal.
2. The brain-computer interface system according to claim 1, further comprising a magnetic shielding device for providing shielding of the subject from ambient magnetic fields unrelated to brain neural activity.
3. The brain-computer interface system according to claim 2, wherein the magnetic shielding device is a multi-layer magnetic shielding barrel or a magnetic shielding room providing free movement space.
4. The brain-computer interface system according to claim 1, 2 or 3, wherein the data acquisition workstation performs slicing and packing on the received continuous data stream and transmits the continuous data stream to the real-time analysis workstation.
5. The brain-computer interface system according to claim 1, wherein the externally controlled device includes a prosthetic, a speech synthesizer.
6. The brain-computer interface system according to claim 1, wherein the magnetoencephalography device employs an atomic magnetometer based on a spin-free relaxation principle.
7. A brain-computer interface system application method based on source space brain magnetic signal decoding comprises the following steps:
1) The preparation stage: acquiring magnetic resonance head structure images of a user by using a cerebral magnetic cap, and constructing a head model comprising scalp, the inner and outer surfaces of skull and cortical gray matter; calculating a noise covariance matrix of the probe array according to the signals when the acquired brain magnetic diagram is empty; constructing a training anticipation set corresponding to the characters and the pinyin;
2) Training phase: the head movement of the user is recorded in real time by utilizing a position coil on the magnetoencephalic cap worn by the user, and the head movement is used for correcting and compensating the influence of the head movement on the signal; dividing the training corpus into M test orders, wherein each test order comprises a plurality of sentences and corresponding pinyin thereof; then, a sentence and a corresponding pinyin form are presented on a display, and magnetic signals of brain nerve activity in the process that a user looks at a screen to imagine to write pinyin letters of each word in sequence are collected; collecting magnetic signals of brain nerve activity in the process of imagining writing each pinyin letter and setting punctuation marks by a user; then extracting the time sequence of the magnetoencephalic signals of each voxel in the primary motor cortex, the primary body induction cortex and the motor auxiliary area which are related to imagination writing of the left hemisphere and the right hemisphere of the brain according to the acquired magnetoencephalic signals in a tracing way; then training a cyclic neural network by using the extracted magnetoencephalic signal time sequence, wherein magnetoencephalic signals in a time window with set length are input, and the probabilities of each Pinyin letter and set punctuation mark are output;
3) The using stage is as follows: tracing the source according to the magnetoencephalic signals of the process of writing each pinyin letter and each set punctuation mark according to the screen prompt after the magnetoencephalic cap is worn by the acquired user, and calibrating the circulating neural network; then, for any unexpectedly expressed sentence i, collecting a magnetoencephalic signal of a Chinese pinyin process corresponding to the sentence i which is imagined by a user, and tracing the magnetoencephalic signal to obtain signals of each voxel in a primary motor cortex, a primary body induction cortex and a motion auxiliary region which are related to imagined writing in a source space; and inputting the signals obtained by tracing into the calibrated cyclic neural network to obtain the probability of the character written by the current time window, and outputting the character with the maximum probability.
8. The method of claim 7, wherein the pinyin decoded from the recurrent neural network is corrected in real time using a deep neural network language model pre-trained based on big data, individual characters decoded incorrectly are corrected, pinyin tones are supplemented with context information, and words are output.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111004557.9A CN113918008B (en) | 2021-08-30 | 2021-08-30 | Brain-computer interface system based on source space brain magnetic signal decoding and application method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111004557.9A CN113918008B (en) | 2021-08-30 | 2021-08-30 | Brain-computer interface system based on source space brain magnetic signal decoding and application method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113918008A CN113918008A (en) | 2022-01-11 |
| CN113918008B true CN113918008B (en) | 2024-06-25 |
Family
ID=79233477
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111004557.9A Active CN113918008B (en) | 2021-08-30 | 2021-08-30 | Brain-computer interface system based on source space brain magnetic signal decoding and application method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113918008B (en) |
Families Citing this family (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114052668B (en) * | 2022-01-17 | 2022-06-17 | 北京航空航天大学杭州创新研究院 | Brain function analysis method based on magnetoencephalogram data |
| CN114791768B (en) * | 2022-06-24 | 2022-11-15 | 清华大学 | Computing system and electronic equipment based on brain-computer interface |
| CN117100277B (en) * | 2022-07-25 | 2024-04-30 | 北京大学 | Multi-mode synchronous brain imaging method based on magnetoencephalography and near infrared brain imaging |
| CN115018018B (en) * | 2022-08-05 | 2022-11-11 | 北京航空航天大学杭州创新研究院 | Double spatial filtering method for inhibiting background noise |
| CN115344122B (en) * | 2022-08-15 | 2024-06-18 | 中国科学院深圳先进技术研究院 | Acoustic non-invasive brain-computer interface and control method |
| CN115359236B (en) * | 2022-10-19 | 2023-03-28 | 之江实验室 | A Machine Learning-Based Magnetoencephalogram Classification System for Working Memory Tasks |
| CN115721323A (en) * | 2022-11-22 | 2023-03-03 | 中国科学院苏州生物医学工程技术研究所 | Brain-computer interface signal identification method and system and electronic equipment |
| CN116088686B (en) * | 2023-01-17 | 2023-07-25 | 北京融域智胜科技有限公司 | Electroencephalogram tracing motor imagery brain-computer interface training method and system |
| CN116369950B (en) * | 2023-05-25 | 2024-01-26 | 武汉理工大学 | Target detection method based on electroencephalogram tracing and multi-feature extraction |
| CN116491960B (en) * | 2023-06-28 | 2023-09-19 | 南昌大学第一附属医院 | Brain transient monitoring device, electronic device, and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109199376A (en) * | 2018-08-21 | 2019-01-15 | 北京工业大学 | The coding/decoding method of Mental imagery EEG signals based on the imaging of OA-WMNE brain source |
| CN111543949A (en) * | 2020-05-13 | 2020-08-18 | 北京航空航天大学 | A diagnostic device for children's ASD based on magnetoencephalography and electroencephalography |
Family Cites Families (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9468541B2 (en) * | 2010-05-05 | 2016-10-18 | University Of Maryland College Park | Time domain-based methods for noninvasive brain-machine interfaces |
| WO2012012746A2 (en) * | 2010-07-22 | 2012-01-26 | Washington University | Multimodal brain computer interface |
| KR101507382B1 (en) * | 2013-09-11 | 2015-04-01 | 한국표준과학연구원 | Magnetoencephalography Measuring Apparatus And Magnetoencephalography Measuring Method |
| CN109567785A (en) * | 2018-12-05 | 2019-04-05 | 北京昆迈生物医学研究院有限公司 | A kind of quantum magneticencephalogram system and method based on magnetic shielding cylinder |
| JP7211132B2 (en) * | 2019-02-06 | 2023-01-24 | 株式会社リコー | Information display device, information display method, and information display system |
| US11801003B2 (en) * | 2020-02-12 | 2023-10-31 | Hi Llc | Estimating the magnetic field at distances from direct measurements to enable fine sensors to measure the magnetic field from the brain using a neural detection system |
| CN111973180B (en) * | 2020-09-03 | 2021-09-17 | 北京航空航天大学 | Brain structure imaging system and method based on MEG and EEG fusion |
| CN113143293B (en) * | 2021-04-12 | 2023-04-07 | 天津大学 | Continuous speech envelope nerve entrainment extraction method based on electroencephalogram source imaging |
-
2021
- 2021-08-30 CN CN202111004557.9A patent/CN113918008B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109199376A (en) * | 2018-08-21 | 2019-01-15 | 北京工业大学 | The coding/decoding method of Mental imagery EEG signals based on the imaging of OA-WMNE brain source |
| CN111543949A (en) * | 2020-05-13 | 2020-08-18 | 北京航空航天大学 | A diagnostic device for children's ASD based on magnetoencephalography and electroencephalography |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113918008A (en) | 2022-01-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN113918008B (en) | Brain-computer interface system based on source space brain magnetic signal decoding and application method | |
| Pfurtscheller et al. | 15 years of BCI research at Graz University of Technology: current projects | |
| Fingelkurts et al. | Cortical operational synchrony during audio–visual speech integration | |
| Cheng et al. | Motion imagery-BCI based on EEG and eye movement data fusion | |
| CN104771255A (en) | Implementation method for identifying motion models based on cerebral-cortex hemoglobin information | |
| Abdulwahab et al. | A systematic review of brain-computer interface based eeg | |
| Fu et al. | Single-trial decoding of imagined grip force parameters involving the right or left hand based on movement-related cortical potentials | |
| Dovgialo et al. | Assessment of statistically significant command-following in pediatric patients with disorders of consciousness, based on visual, auditory and tactile event-related potentials | |
| Nawrocka et al. | Brain-computer interface based on steady-state visual evoked potentials (SSVEP) | |
| Mohan et al. | Exploring the exciting potential and challenges of brain computer interfaces | |
| Démas et al. | Assessing spino-cortical proprioceptive processing in childhood unilateral cerebral palsy with corticokinematic coherence | |
| Rajyalakshmi et al. | Exploration of recent advances in the field of brain computer interfaces | |
| Tang et al. | Wearable intelligent throat enables natural speech in stroke patients with dysarthria | |
| Kaur et al. | Analyzing eeg based neurological phenomenon in bci systems | |
| Ramakuri et al. | The virtual keyboard is accessible using wireless EEG device for patient’s with paralysis | |
| Gangwar et al. | Statistical learning for brain–computer interface | |
| Adsawinnawanawa et al. | Decode brain signal into thai word using EEG and L-SVM | |
| Kambhamettu et al. | Brain–Computer Interface‐Assisted Automated Wheelchair Control Management–Cerebro: A BCI Application | |
| Sahu et al. | Applying auto regression techniques on amyotrophic lateral sclerosis patients EEG dataset with P300 speller | |
| Gilford | Localisation of brain activity for SSVEP-based BCIs: an fMRI and EEG study | |
| Saeed et al. | Investigation of Feature Extraction Method for EEG Signal Processing | |
| Martišius | Data acquisition and signal processing methods for brain–computer interfaces | |
| Singh et al. | Futuristic Design and Intelligent Computational Techniques in Neuroscience and Neuroengineering | |
| Devi | The Fusion of Human-Computer Interaction and Artificial Intelligence Leads to the Emergence of Brain Computer Interaction | |
| Battapady et al. | Single trial detection of human movement intentions from SAM-filtered MEG signals for a high performance two-dimensional BCI |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |