CN115316955A - Light-weight and quick decoding method for motor imagery electroencephalogram signals - Google Patents
Light-weight and quick decoding method for motor imagery electroencephalogram signals Download PDFInfo
- Publication number
- CN115316955A CN115316955A CN202211087807.4A CN202211087807A CN115316955A CN 115316955 A CN115316955 A CN 115316955A CN 202211087807 A CN202211087807 A CN 202211087807A CN 115316955 A CN115316955 A CN 115316955A
- Authority
- CN
- China
- Prior art keywords
- time
- motor imagery
- convolution
- module
- electroencephalogram
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Withdrawn
Links
- 238000000034 method Methods 0.000 title claims abstract description 46
- 238000011176 pooling Methods 0.000 claims abstract description 29
- 238000013136 deep learning model Methods 0.000 claims abstract description 25
- 238000007781 pre-processing Methods 0.000 claims abstract description 6
- 230000006870 function Effects 0.000 claims description 32
- 238000004422 calculation algorithm Methods 0.000 claims description 31
- 238000012549 training Methods 0.000 claims description 20
- 238000012360 testing method Methods 0.000 claims description 16
- 238000001914 filtration Methods 0.000 claims description 10
- 238000004364 calculation method Methods 0.000 claims description 9
- 238000013507 mapping Methods 0.000 claims description 9
- 238000010606 normalization Methods 0.000 claims description 9
- 238000012545 processing Methods 0.000 claims description 8
- 230000002123 temporal effect Effects 0.000 claims description 8
- 210000004556 brain Anatomy 0.000 claims description 7
- 239000011159 matrix material Substances 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 6
- 230000003595 spectral effect Effects 0.000 claims description 6
- ORILYTVJVMAKLC-UHFFFAOYSA-N Adamantane Natural products C1C(C2)CC3CC1CC2C3 ORILYTVJVMAKLC-UHFFFAOYSA-N 0.000 claims description 3
- 238000001228 spectrum Methods 0.000 claims 1
- 238000013527 convolutional neural network Methods 0.000 description 28
- 238000002474 experimental method Methods 0.000 description 12
- 230000004913 activation Effects 0.000 description 8
- 238000013135 deep learning Methods 0.000 description 8
- 238000005070 sampling Methods 0.000 description 5
- 238000000537 electroencephalography Methods 0.000 description 4
- 210000002569 neuron Anatomy 0.000 description 4
- 210000004027 cell Anatomy 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 239000000284 extract Substances 0.000 description 3
- 238000000605 extraction Methods 0.000 description 3
- 230000004927 fusion Effects 0.000 description 3
- 238000010801 machine learning Methods 0.000 description 3
- 238000011160 research Methods 0.000 description 3
- 238000004458 analytical method Methods 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000010200 validation analysis Methods 0.000 description 2
- 238000012800 visualization Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 238000012067 mathematical method Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000003058 natural language processing Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000003534 oscillatory effect Effects 0.000 description 1
- 238000000513 principal component analysis Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 230000000717 retained effect Effects 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Images
Classifications
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/24—Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
- A61B5/316—Modalities, i.e. specific diagnostic methods
- A61B5/369—Electroencephalography [EEG]
- A61B5/372—Analysis of electroencephalograms
-
- A—HUMAN NECESSITIES
- A61—MEDICAL OR VETERINARY SCIENCE; HYGIENE
- A61B—DIAGNOSIS; SURGERY; IDENTIFICATION
- A61B5/00—Measuring for diagnostic purposes; Identification of persons
- A61B5/72—Signal processing specially adapted for physiological signals or for diagnostic purposes
- A61B5/7235—Details of waveform analysis
- A61B5/7264—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
- A61B5/7267—Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems involving training the classification device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/049—Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Health & Medical Sciences (AREA)
- Life Sciences & Earth Sciences (AREA)
- Physics & Mathematics (AREA)
- Artificial Intelligence (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Theoretical Computer Science (AREA)
- General Health & Medical Sciences (AREA)
- Molecular Biology (AREA)
- Mathematical Physics (AREA)
- Evolutionary Computation (AREA)
- Software Systems (AREA)
- Public Health (AREA)
- General Physics & Mathematics (AREA)
- Computing Systems (AREA)
- Data Mining & Analysis (AREA)
- Computational Linguistics (AREA)
- Veterinary Medicine (AREA)
- General Engineering & Computer Science (AREA)
- Animal Behavior & Ethology (AREA)
- Psychiatry (AREA)
- Surgery (AREA)
- Pathology (AREA)
- Heart & Thoracic Surgery (AREA)
- Medical Informatics (AREA)
- Signal Processing (AREA)
- Physiology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Fuzzy Systems (AREA)
- Psychology (AREA)
- Feedback Control In General (AREA)
- Manipulator (AREA)
Abstract
The invention discloses a light and rapid motor imagery electroencephalogram signal decoding method, which comprises the following steps: s1, constructing a deep learning model, wherein the deep learning model comprises a space-time convolution module, a pooling module and a full-connection module; the space-time convolution module is composed of time convolution layers for reducing trainable parameter quantity and space depth convolution layers for reducing channel connection; the pooling module is a stack of pooling layers to reduce the dimensionality and complexity of the model; the full connection module is used for final classification; s2, preprocessing the original electroencephalogram signals, and then classifying and decoding the preprocessed electroencephalogram signals by using a deep learning model. The invention can obtain better decoding performance with less trainable parameters, and maintains relative balance between classification precision and model complexity.
Description
Technical Field
The invention relates to the technical field of electroencephalogram signal analysis, in particular to a light-weight and rapid motor imagery electroencephalogram signal decoding method.
Background
In a motor imagery electroencephalogram decoding task, the most typical of the traditional machine learning algorithm for manually extracting features is a common space mode, a filter bank common space mode, short-time Fourier transform, principal component analysis and the like. The basic idea of the CSP algorithm is to find a spatial filter and maximize the distance in the motor imagery electroencephalogram four-classification task. Similarly, the FBCSP method is an extension of CSP technology and is also often used in electroencephalogram decoding tasks. The algorithm extracts optimal spatial features through a set of band pass filters, thereby selecting and classifying the features. Meanwhile, in the past study on motor imagery electroencephalogram, classification methods such as linear classifiers, support vector machines, multi-layer perceptrons, and random forests are often used. Although these methods have achieved good results in motor imagery electroencephalogram decoding tasks, they separate feature extraction and classification into two stages.
However, the deviation of the conventional machine learning algorithm caused by the need of manually extracting features has its defect in classification accuracy. With the wide application of the deep learning method in various fields and the capability of efficiently extracting more meaningful features to obtain a better effect, the convolutional neural network also has come into play in the aspect of electroencephalogram classification. In contrast, deep learning combines feature extraction and classification into one step. According to the characteristics of time sequence of electroencephalogram signals, the long-period and short-period memory network has the capability of extracting time characteristics, particularly has unique advantages in the aspect of processing time sequences and has wide understanding in the fields of speech recognition and natural language processing. By introducing gate functions in the cell structure, the LSTM can solve the trouble of data that the general RNN cannot learn because the relevant input information is too large. The LSTM structure contains two important branches, a memory cell and a nonlinear gating cell. At present, research attempts have been made to extract temporal features and obtain good classification results by using BLSTM to memorize the relationship change of two channels in a specific time. Surprisingly, experiments show that the convolutional neural network can better extract time-space domain and frequency domain characteristics from the motor imagery electroencephalogram. Notably, in the research contribution that has appeared, most scholars use CNNs or fusion models to extract motor imagery electroencephalograms, but do not consider the consumption of resources and the complexity of model computation, occupy more trainable parameters especially in the fusion models with higher complexity, and do not visualize the extracted features and the convolution kernel output in their solutions.
Disclosure of Invention
The invention aims to provide a light and quick decoding method for a motor imagery electroencephalogram signal. The invention can obtain better decoding performance with less trainable parameters, and maintains relative balance between classification precision and model complexity.
The technical scheme of the invention is as follows: a light-weight and quick decoding method for motor imagery electroencephalogram signals is carried out according to the following steps:
s1, constructing a deep learning model, wherein the deep learning model comprises a space-time convolution module, a pooling module and a full-connection module; the space-time convolution module is composed of a time convolution layer for reducing the trainable parameter number and a space depth convolution layer for reducing channel connection; the pooling module is a stack of pooling layers to reduce the dimensionality and complexity of the model; the full connection module is used for final classification;
s2, preprocessing the original electroencephalogram signals, and then classifying and decoding the preprocessed electroencephalogram signals by using a deep learning model.
According to the lightweight and rapid motor imagery electroencephalogram signal decoding method, the space-time convolution module is used for extracting the space and spectral characteristics of electroencephalogram input; the time convolution layer defines the kernel value of the time convolution kernel through a parameterized function, so that the kernel value of the time convolution layer is described by the time filteringSubset of devices to reduce the number of trainable parameters and reduce resource consumption, wherein the EEG signal is associated with
The one-dimensional convolution formula between the time convolution kernels is as follows:
in the formula,
is the first
An electrode signal and
one-dimensional convolution between time convolution kernels;
is the total number;
is shown as
The signal of each electrode is transmitted to the electrode,
represents the length of the filter along the time dimension of the one-dimensional convolution,
the number of corresponding temporal convolution kernels;
is an intermediate amount;
by parameterising functions
The kernel value of the time convolution kernel is defined, and the amplitude of the band-pass filter in the frequency domain range is expressed as:
in the formula:
is the frequency;
is the first
An inferior value of the cut-off frequency of the band-pass filtering;
is that
A figure of merit for the cut-off frequency of the bandpass filtering;
the amplitude of the band pass filter in the time domain is as follows:
in the formula:
is a sine function;
the depth-space convolutional layers with filter size (1, 65) and 32 band-pass temporal filters were stacked after the temporal convolutional layer to learn spectral and spatial features.
In the foregoing lightweight and fast decoding method for electroencephalogram signals based on motor imagery, the pooling module takes a sampling signal as an input with a size of (1, 109) and a step size of (1, 23) and investigates a specific hyper-parameter, and converts the output of the pooling module by using a flat layer and connects to a full connection layer.
In the lightweight and rapid motor imagery electroencephalogram signal decoding method, the fully-connected module uses a softmax function for classification, and trains hyper-parameters by using a supervised learning algorithm to generate real output values corresponding to four classes of motor imagery; passing function
Processes, functions, mapped to real classes to represent convolutions
Represents the number of electrodes;
in order to be a step of time,
is the total input data; the conditional probability for a particular subject for each class label that converts an output to a given output by the softmax function is expressed as:
in the formula:
is the conditional probability of the tag;
is a first
The input data for each of the tests was,
；
parameters representing weights and bias functions;
is a mapping function;
minimizing the sum of the losses per sample by assigning a high probability to correctly output labels, the calculation formula is shown as:
in the formula:
for the output class of each trial,
the value of (c) is determined by the negative logarithm probability, and the specific calculation expression is as follows:
in the lightweight and rapid motor imagery electroencephalogram signal decoding method, in the step S2, the processing of the original electroencephalogram signal is to convert the original electroencephalogram signal into a channel of a cross-time sample C × T matrix, then apply a band-pass filter between 8Hz and 30Hz for filtering, and then perform batch normalization processing on the filtered electroencephalogram signal.
In the lightweight and fast decoding method for the motor imagery electroencephalogram signals, the original electroencephalogram signals are expressed as follows:
in the formula:
represents the first
Total number of trials recorded, first of motor imagery signals
The secondary test is closely connected with the output class; for a given test
Of 1 at
Mapping the output of the secondary input test to an output class label
To respectively refer to motor imagery four classification tasks.
According to the lightweight and rapid motor imagery electroencephalogram signal decoding method, the training of a deep learning model is realized by utilizing a BrainDecode framework; dividing an original electroencephalogram signal into 750 samples as input of a deep learning model, setting the value of batch at each time to be 58 at first, and terminating training when an epochs execution standard exceeds a preset maximum value; and simultaneously, an early stopping strategy is combined with an Adam optimizer to improve the classification precision.
The lightweight and rapid motor imagery electroencephalogram signal decoding method prevents the over-fitting phenomenon by using a batch normalization regularization method.
Compared with the prior art, the invention provides the efficient deep learning module for the motor imagery electroencephalogram decoding, the deep learning module can obtain better decoding performance with less trainable parameters, and the relative balance between the classification precision and the model complexity is kept. The invention greatly improves the decoding precision of the motor imagery electroencephalogram by taking the variable parameters into consideration. According to the experimental result demonstration of the specific subject of the electroencephalogram, the method obtains about 78.42% of average classification precision, and has stronger robustness compared with other SOA algorithms. In addition, the invention can solve the general problem based on electroencephalogram in neuroscience, and paves the way for establishing practical clinical application.
Drawings
FIG. 1 is a diagram of a deep learning model according to the present invention;
FIG. 2 is a schematic flow chart of the present invention;
FIG. 3 is a data set BCI contest
2a training set and testing set single experiment paradigm;
FIG. 4 shows the competition of the FBCSP algorithm in BCI
2a average confusion matrix of different output classes between all subjects on the dataset;
FIG. 5 shows a Mixed-ConvNet model in BCI competition
2a average confusion matrix of different output classes among all subjects on the dataset;
FIG. 6 shows a BCI race
2a comparing the performance of decoding accuracy and standard error between different models on the data set;
FIG. 7 is a comparison between the depth model of the present invention and two other SOA algorithms for different performance metrics.
Detailed Description
The following examples further illustrate the invention but are not to be construed as limiting the invention.
Example 1: a lightweight and rapid decoding method for a motor imagery electroencephalogram signal is disclosed, as shown in figure 1, and comprises the following steps:
s1, acquiring an original electroencephalogram signal, and constructing a deep learning model, wherein the deep learning model comprises a space-time convolution module, a pooling module and a full-connection module; the space-time convolution module is composed of a time convolution layer for reducing the trainable parameter number and a space depth convolution layer for reducing channel connection; the pooling module is a stack of pooling layers to reduce the dimensionality and complexity of the model; the full connection module is used for final classification;
s2, preprocessing the original electroencephalogram signals, and then classifying and decoding the preprocessed electroencephalogram signals by using a deep learning model.
Specifically, the raw brain electrical signal is represented as:
in the formula:
represents the first
Total number of trials recorded, first of motor imagery signals
The secondary test is closely connected with the output class; for a given test
，
(ii) a Wherein
Represents the number of electrodes;
in order to be a step of time,
for the total input data, the first
Mapping the output of the secondary input test to an output class label
To respectively refer to motor imagery four classification tasks.
As shown in fig. 2, the preprocessing of the raw brain electrical signal is to convert the raw brain electrical signal into channels of a cross-time sample C × T matrix, then apply a band-pass filter between 8Hz and 30Hz for filtering, and then perform batch normalization processing on the filtered brain electrical signal. In the preprocessing process, noise and artifacts are filtered for the original electroencephalogram signals, and three EOG bad channels are removed. In the present invention, these frequencies are updated to vary within the range, initialized randomly from a uniform distribution of the frequency bands of interest at [0,38] or [4,38] Hz.
In this embodiment, the global hyper-parameter, the number of output shapes, the number of parameters, and the detailed information of the activation function used by each layer of the deep learning model are described in tables 1 to 3.
TABLE 1 input and spatio-temporal convolution module for deep learning model
TABLE 2 pooling Module
TABLE 3 full connection Module
Tables 1-3 show the detailed hyper-parameters of the deep learning model of the present invention. Tables 1-3 highlight the main modules of the inventive smart application time-sinusoidal convolutional layer model and the convolutional or pooling layers used by each module, the global hyper-parameters of each layer, the output shape, the trainable parameters of each layer and the activation functions employed. Where C denotes the number of channels, T denotes the number of samples, K and F denote the number and size of convolution kernels, S and P denote the step size and pooling size, D denotes multiplier in the depth convolution layer, and m denotes regularization using BatchNormParameter values, and
represents the droop rate of the hybrid convolutional neural network and,
meaning the number of EEG signal feature classifications. At the same time, the sampling frequency is 250Hz, and an ELU exponential linear unit activation function is also used in the experiment.
In the field of electroencephalogram signal and processing, CNNs are often used for processing high-latitude data, and bias values and weights are given to all neurons with convolution windows, namely kernel sizes. Instead, CNNs skillfully deepens the concept of weight sharing, learns a set of weights and a single bias value applied to hidden layer neurons, and the mathematical method of this process is expressed as follows:
wherein,
to indicate the first in the hidden layer
A first of the filter
The activation output of the individual neurons is,
in response to the activation function being used,
finger filter
Is determined by the shared offset value of (a),
is the size of the kernel, and is,
is a vector of shared weights that is,
the idea is to predict the output vector of the neuron,
representing a transpose operation.
In order to extract the spatial and spectral characteristics of the electroencephalogram input, time and space convolution layers are respectively designed. The time convolution layer defines the kernel value of the time convolution kernel through a parameterization function, so that the kernel value description of the time convolution layer belongs to the subset of the time filter, the trainable parameter quantity is reduced, and the resource consumption is reduced, wherein the electroencephalogram signal and the second time convolution kernel
The one-dimensional convolution formula between the time convolution kernels is as follows:
in the formula,
is the first
An electrode signal and
a one-dimensional convolution between the time convolution kernels;
is the total number;
is shown as
The signal of each electrode is transmitted to the electrode,
represents the length of the filter along the time dimension of the one-dimensional convolution,
the number of corresponding temporal convolution kernels;
is an intermediate amount;
by parameterising functions
The kernel values of the time convolution kernel are defined, and in order to describe the band pass filtering in the frequency domain, the amplitude is expressed as:
in the formula:
is the frequency;
is the first
Bad value of cut-off frequency of band-pass filtering;
is that
A figure of merit for the cut-off frequency of the bandpass filter;
this computation reduces the number of trainable parameters for each kernel of the time convolution layer in the time domain, which, similarly, has the following amplitude for band pass filtering in the time domain:
in the formula:
is a sine function;
the depth-space convolutional layers with filter size (1, 65) and 32 band-pass temporal filters were stacked after the temporal convolutional layer to learn spectral and spatial features. In order to prevent the occurrence of overfitting, the invention flexibly applies the batch normalization regularization technology to the feature map dimension.
The output of the space-time convolution module is used as the input of the pooling module, the dimensionality of each feature map is reduced by a pooling layer on the premise of retaining original meaningful information, and the space pooling can be divided into a sub-sampling type and a down-sampling type, wherein the most famous method is maximum pooling and average pooling. The present invention sets a batch normalization technique after each convolutional layer and applies the Dropout method after the pooling layer. In the average pooling module, the sampled signal is taken as input and the specific hyper-parameter is investigated with a size of (1, 109) and a step size of (1, 23).
Finally, the deep learning model also leverages the flat layer to transform the output of the module and connect to the fully connected layer, resulting in an extracted one-dimensional feature vector. The full-connection module is used for classifying by using a softmax function, training a hyper-parameter by using a supervised learning algorithm, and generating a real output value corresponding to the motor imagery four-classification; the process of mapping to the true class by a function to represent the convolution, the specific subject conditional probability of each class label that converts the output to a given output by the softmax function is expressed as:
in the formula:
is the conditional probability of the tag;
is as follows
Input data for each test;
parameters representing the weights and the bias function,
an exponential function;
is a mapping function;
represents the number of electrodes;
in order to be a step of time,
as total input data
Minimizing the sum of the losses per sample by assigning a high probability to correctly output labels, the calculation formula is shown as:
in the formula:
for the output class of each trial,
the value of (c) is determined by the negative log likelihood, and the specific calculation expression is as follows:
the invention skillfully realizes the training of Mixed-ConvNet by means of the BrainDecode framework which is completely used for processing the motor imagery electroencephalogram classification task problem and is proposed by Schirrmeister. Because the difference between the brain electrical subjects is changed along with the change of time, the invention sets a series of variable hyper-parameters and searches the optimal parameter setting value for each subject. 750 samples of the division of the original 22 electroencephalogram channel signals are taken as input of the model, the value of the batch is set to 58 at the beginning each time, and the training is terminated when the epochs execution standard exceeds a preset maximum value. Meanwhile, an early stopping strategy and an Adam optimizer are combined to improve the classification accuracy of the model, a frequency band with a low frequency of 0 or 4Hz is randomly selected in the interested distribution, and in order to enable the model training to be more stable and reliable, the batch normalization regularization technology is the key point for preventing the over-fitting phenomenon and the value of the over-fitting phenomenon is set to be 0.99. By the original brain electrical signal
With the preprocessed EEG signal
For example, when
And
are respectively a matrix
And
row i of (1), then normalization operationIs obviously represented by the following formula:
wherein,
and with
Mean and standard deviation runs are represented, respectively. After the output results are regularized, the activation functions are used skillfully and the functions used in this study are exponential linear units and linear activation functions. Wherein, the calculation formula of the ELU may be expressed as:
the Dropout technique randomly sets the output of the previous layer to 0 during each training update and this model sets a Dropout rate of 0.5 after averaging the pooling layers.
In order to verify the accuracy of classification decoding of the invention, the invention visualizes the experiment results of the data set BCI competition 2a and some SOA algorithms, visualizes the convolution kernel, visualizes the convolved results to help understand the function of the convolution kernel, deeply identifies the key functions of which parts in the image classification problem, namely class activation visualization and hidden layer and feature visualization, through a heat map.
1. Description of data sets
As one of the most common datasets in the field of motor imagery electroencephalography, BCI race 2a is a publicly available dataset collected by the university of Greenz from 9 subject experiments using 22 electroencephalographic electrodes (excluding 3 EOG channels) at a sampling frequency of 250 Hz. It is specifically reminded that in the experimental study of the present invention, since the training data of the 4 th subject of the original data set is missing part of the test sample, the experiment was performed by using eight subjects except the 4 th subject. After three additional EOG channels are eliminated, the noise and artifacts are filtered at 0.5 to 100Hz using a band pass filter. The subjects of the data set perform four different Motor imagery tasks of left hand, right hand, double feet and tongue movement, the training set and the test set respectively complete the experimental records on two different days, each group of experiments comprises 288 experiment times, each type of task comprises 72 times, the Motor imagery time is 4s, each experiment generates 750 sample points, the experimental paradigm of which is shown in fig. 3 and comprises a "Beep" sound prompt tone, a "fire scenes" 2-second time cross, the next 1.25 seconds of left, right, upper and lower "Cue" arrow prompts (respectively corresponding to the Motor imagery left hand, right hand, double feet and tongue categories), 3 to 6 seconds of key "Motor imagery time and the last 6 to 7.5 seconds of" Break "short rest time after the cross hours. Previous research on classification attempts of motor imagery related data shows that the individual variability of the subjects of the data set is large, and the decoding effect of the subjects on the model is different from person to person. Moreover, to implement the early stopping strategy, which is the first step in the optimization process, the training set is further partitioned into a small part of the training set and a validation set, where the validation set accounts for 20% of the total training set.
2. Performance evaluation index
In order to evaluate the decoding performance of the proposed method, test data is retained and true predictions are made for the tagged data. The following performance indexes are used in the experiment to evaluate the proposed model, wherein the classification precision is the most frequently adopted measurement index, and the predicted value and the recall ratio are also important methods for identifying the good and bad structure of the algorithm, and the three common index formulas are as follows:
wherein
Is a true example of the case that,
is a true negative example of the case where,
it is a false positive example that,
is a false negative example. In addition, the cohn kappa value is a more commonly used calculation index, and the formula is shown as follows:
wherein,
representing the scale accuracy of the observed uniformity,
implicitly the probability or accuracy of the random guess.
The score value is an index which is adopted frequently finally, and needs to be calculated by combining a predicted value and a recall ratio, and a specific formula is expressed as follows:
3. comparison result
The deep learning model proposed by the invention competes with similar SOA algorithm in data set BCI
Experimental comparisons on phase 2a are detailed in table 4 (subject 4 with partial training data missing omitted).
TABLE 4
In order to improve the persuasion, a traditional machine learning algorithm FBCSP and deep learning ShallowConvNet, C2CM, deepConvNet, waSFConvNet, CM-ConvNet, AMSI-ConvNet and a deployed-model are selected for the experiment, and a series of comparison experiments are implemented on the four-classification motor imagery electroencephalogram data set. The common spatial mode of the filter bank is the most classical traditional manual feature extraction algorithm of motor imagery electroencephalogram, and the four-classification data set task is completed by combining a linear discriminant analysis classifier, so that the decoding precision of 68.59% is obtained on the data set. Fig. 4 and 5 respectively show the average confusion matrix of different output classes between the FBCSP algorithm and all the subjects of the present invention, and both show that the average classification accuracy of the "Left" and "Right" motor imagery tasks is higher than that of the "Foot" and "Tongue" classifications. Notably, the proposed Mixed-ConvNet model is higher than the FBCSP + rLDA structure in four motor imagery classifications, and especially extracts electroencephalogram features which are more obviously differentiable for decoding on the two classifications of 'Foot' and 'Tongue'. It is important to pay attention to that the structure is sufficiently higher than that of the FBCSP traditional algorithm by about 24% in the aspect of 'Tongue' classification and reaches the average decoding precision of 80.4%, so that the efficiency of extracting the features of the proposed Mixed-ConvNet decoding model is proved to be high.
Furthermore, it was surprisingly found from table 4 that the Shallow model Shallow-ConvNet most widely used in the field of motor imagery electroencephalogram and the two-dimensional convolution scene model C2CM similar to most two-dimensional convolution structures obtain average classification accuracy of 74.05% and 75.43% on this data set, respectively. Meanwhile, the Deep-ConvNet model built by Schirrmeister et al stacks several layers of convolution and pooling layers on the basis of a Shallow structure, and on the contrary, not only is the trainable parameter quantity of the model increased, but also the average classification precision is reduced by about 4% compared with Shalow-ConvNet. It is worth mentioning that the model of the spectral power modulation of the electroencephalogram signal, waSF-ConvNet, is directly considered with the wavelet kernel, the network comprises two specific convolutional layers, followed by a pooling layer, plus five complete parts of a dropout layer and a fully connected layer, the amount of parameters is significantly reduced in the process of learning the features and only a relatively low accuracy of 68.96% is obtained on this four-class electroencephalogram data set. Further, the AMSI-ConvNet structure proposes two variants of the space-time convolution module to verify the feasibility of decoding performance, and further benchmark tests show that an average classification accuracy of 76.27% is obtained on the electroencephalogram competition data set. Compared with other baseline algorithms, the average decoding precision of the model provided by the invention is about 10% higher than that of the traditional FBCSP, and the standard deviation value is relatively small, so that the structure is observed to have higher robustness and stability.
Furthermore, the deep learning model provided by the invention competes with other Baseline algorithms in BCI
The experimental pair of decoding accuracy on the 2a data set is shown in fig. 6, where the error bars in fig. 6, which reflect the performance between the eight subjects on this data set, represent the standard deviation (Std). The performance of the Mixed-ConvNet model is superior to that of other SOA methods, the algorithm is higher than that of the Shalow-ConvNet and Deep-ConvNet models by 4% and 8% in accuracy respectively through fully using the sine time convolution module, and the standard deviation of the model on the data set is smaller than that of other SOA algorithms, and experimental demonstration shows that the time-space convolution module indeed improves the electroencephalogram decoding performance. Considering the performance of the model from the direction of an individual subject, compared with other SOA algorithms, the performance of the displayed model on the structure is the best, and the average decoding accuracy of the specific subjects 1, 5 and 9 is respectively 88.54%, 71.87% and 85.76%, so that the proposed structure can be completely verified to be sufficiently applied to the decoding of the oscillatory motor imagery electroencephalogram. Table 3 fully demonstrates the BCI contest
Several similar SOA motor imagery algorithms on the 2a dataset using a subject-specific approach were compared to the kappa value results between each subject of the proposed model.
TABLE 5
As can be readily seen from table 5, the proposed model achieved the optimal performance evaluation index kappa value and reached 0.72, the highest kappa values were obtained in the specific subjects 2, 3, 6 and 9 compared to other conventional algorithms or the shallow model. Fig. 7 shows the comparison results between the model Mixed-ConvNet (deep learning model) and the conventional algorithm FBCSP and the typical Shallow structure shalow-ConvNet model in the classification accuracy, the kappa value, and the F1 score value of different performance metrics (in the histogram of each evaluation index, shalow-ConvNet, FBCSP, and Mixed-ConvNet are sequentially from top to bottom), and it is easy to distinguish from the graph that the model Mixed-ConvNet (deep learning model) performs better than other SOA algorithms (i.e., FBCSP). Moreover, table 6 simply illustrates the competition of the proposed model with other Baseline Algorithm BCI by other performance metrics with Precision and Recall values to fully verify the competitiveness of the proposed model
Comparison of results for Precision, recall values, and F1 scores on 2a datasets:
TABLE 6
Carefully observing the table, the overall average value of the proposed lightweight structure on the performance indexes of the three is superior to that of other comparable SOA algorithms.
Considering the complexity of the model, the detailed comparison result between the deep learning structure proposed by the present invention and the trainable parameters, training time and average decoding precision used by several other baseline algorithms on the BCI competition 2a data set is shown in table 7:
TABLE 7
As can be seen from Table 7, in several shallow and deep fusion algorithms for comparison, the channel mapping mixed size convolution neural network combined with the amplitude disturbance data enhancement method uses a large number of trainable hyperparameters and reaches 8.36 × 10 5 On the premise that the model structure is complex, a large amount of resources are consumed, the calculation cost is increased, and the average decoding precision on the data set does not exceed the proposed model. In addition, the Deep convolution model Deep-ConvNet, the classical Shallow structure Shallow-ConvNet, and the hybrid stacked network AMSI-ConvNet do not provide better balance among trainable parameters, the average training time of the subject (hh: mm: ss), and the average decoding accuracy. Our proposed model introduces only a minimum number of trainable parameters 8324 and achieves the highest average classification accuracy, the average training time between eight subjects on this data set only takes less than 10 minutes, which is sufficient to demonstrate that the proposed lightweight structure is fully applicable in real brain-machine interface application scenarios.
In conclusion, the invention provides the efficient deep learning module for the decoding of the motor imagery electroencephalogram, the deep learning module can obtain better decoding performance with less trainable parameters, and the relative balance between the classification precision and the model complexity is kept. The invention greatly improves the decoding precision of the motor imagery electroencephalogram by taking the variable parameters into consideration. According to the experimental result demonstration of the specific subject of the electroencephalogram, the method obtains about 78.42% of average classification precision, and has stronger robustness compared with other SOA algorithms. In addition, the invention can solve the general problem based on electroencephalogram in neuroscience, and paves the way for establishing practical clinical application.
Claims (8)
1. A light-weight and quick motor imagery electroencephalogram signal decoding method is characterized by comprising the following steps: the method comprises the following steps:
s1, constructing a deep learning model, wherein the deep learning model comprises a space-time convolution module, a pooling module and a full-connection module; the space-time convolution module is composed of a time convolution layer for reducing the trainable parameter number and a space depth convolution layer for reducing channel connection; the pooling module is a stack of pooling layers to reduce the dimensionality and complexity of the model; the full connection module is used for final classification;
s2, preprocessing the original electroencephalogram signals, and then classifying and decoding the preprocessed electroencephalogram signals by using a deep learning model.
2. The lightweight fast decoding method for electroencephalogram signals based on motor imagery, according to claim 1, wherein: the space-time convolution module is used for extracting the space and spectrum characteristics of the electroencephalogram input; the time convolution layer defines the kernel value of the time convolution kernel through a parameterization function, so that the kernel value description of the time convolution layer belongs to the subset of the time filter, the trainable parameter quantity is reduced, and the resource consumption is reduced, wherein the electroencephalogram signal and the second time convolution kernel
The one-dimensional convolution formula between the time convolution kernels is as follows:
in the formula,
is the first
An electrode signal and
a one-dimensional convolution between the time convolution kernels;
is the total number;
denotes the first
The signal of each electrode is transmitted to the electrode,
represents the length of the filter along the time dimension of the one-dimensional convolution,
the number of corresponding temporal convolution kernels;
is an intermediate amount;
by parameterising functions
The kernel value of the time convolution kernel is defined, and the amplitude of the band-pass filter in the frequency domain range is expressed as:
in the formula:
is the frequency;
is the first
Bad value of cut-off frequency of band-pass filtering;
is that
A figure of merit for the cut-off frequency of the bandpass filter;
the amplitude of the band pass filtering in the time domain is as follows:
in the formula:
is a sine function;
the depth-space convolutional layers with filter size (1, 65) and 32 band-pass time filters were stacked after the time convolutional layer to learn spectral and spatial features.
3. The lightweight and fast motor imagery electroencephalogram signal decoding method of claim 1, wherein: the pooling module takes as input the sampled signal at a size of (1, 109) and step size of (1, 23) and investigates certain hyper-parameters, and switches the output of the pooling module with a flat layer and connects to a full connection layer.
4. The lightweight and fast motor imagery electroencephalogram signal decoding method of claim 3, wherein: the full-connection module is used for classifying by using a softmax function, training a hyper-parameter by using a supervised learning algorithm, and generating a real output value corresponding to the motor imagery four-classification; passing function
Mapping to trueReal classes to indicate the process of convolution, in functions
Represents the number of electrodes;
in order to be a step of time,
is the total input data; the conditional probability for a particular subject for each class label that converts an output to a given output by the softmax function is expressed as:
in the formula:
is the conditional probability of the tag;
is a first
The input data for each of the trials was,
；
the parameters representing the weights and the bias function,
an exponential function;
as a function of the mapping；
Minimizing the sum of the losses per sample by assigning a high probability to the correctly output label, the calculation formula is shown as:
in the formula:
for the output class of each trial,
the value of (c) is determined by the negative logarithm probability, and the specific calculation expression is as follows:
5. the lightweight and fast motor imagery electroencephalogram signal decoding method of claim 1, wherein: in the step S2, the original electroencephalogram signals are processed by converting the original electroencephalogram signals into channels of a cross-time sample C x T matrix, then filtering by applying a band-pass filter between 8Hz and 30Hz, and then carrying out batch normalization processing on the filtered electroencephalogram signals.
6. The lightweight fast decoding method for electroencephalogram signals based on motor imagery, according to claim 5, wherein: the raw brain electrical signal is represented as:
in the formula:
represents the first
Total number of trials recorded, first of motor imagery signals
The secondary test is closely connected with the output class; for a given test
First, of
Output mapping of secondary input test into output class label
To respectively refer to motor imagery four classification tasks.
7. The lightweight fast decoding method for electroencephalogram signals based on motor imagery, according to claim 6, wherein: training a deep learning model by using a BrainDecode framework; dividing an original electroencephalogram signal into 750 samples as input of a deep learning model, setting the value of batch at each time to be 58 at first, and terminating training when an epochs execution standard exceeds a preset maximum value; and simultaneously, an early stopping strategy is combined with an Adam optimizer to improve the classification precision.
8. The lightweight and fast motor imagery electroencephalogram signal decoding method of claim 7, wherein: and (3) preventing an overfitting phenomenon by using a batch normalization regularization method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211087807.4A CN115316955A (en) | 2022-09-07 | 2022-09-07 | Light-weight and quick decoding method for motor imagery electroencephalogram signals |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211087807.4A CN115316955A (en) | 2022-09-07 | 2022-09-07 | Light-weight and quick decoding method for motor imagery electroencephalogram signals |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115316955A true CN115316955A (en) | 2022-11-11 |
Family
ID=83929661
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211087807.4A Withdrawn CN115316955A (en) | 2022-09-07 | 2022-09-07 | Light-weight and quick decoding method for motor imagery electroencephalogram signals |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115316955A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117390542A (en) * | 2023-10-13 | 2024-01-12 | 上海韶脑传感技术有限公司 | Domain generalization-based cross-test motor imagery electroencephalogram modeling method |
-
2022
- 2022-09-07 CN CN202211087807.4A patent/CN115316955A/en not_active Withdrawn
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117390542A (en) * | 2023-10-13 | 2024-01-12 | 上海韶脑传感技术有限公司 | Domain generalization-based cross-test motor imagery electroencephalogram modeling method |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Dissanayake et al. | Deep learning for patient-independent epileptic seizure prediction using scalp EEG signals | |
| Yang et al. | Emotion recognition from multi-channel EEG through parallel convolutional recurrent neural network | |
| Yang et al. | A single-channel EEG based automatic sleep stage classification method leveraging deep one-dimensional convolutional neural network and hidden Markov model | |
| Samavat et al. | Deep learning model with adaptive regularization for EEG-based emotion recognition using temporal and frequency features | |
| Taqi et al. | Classification and discrimination of focal and non-focal EEG signals based on deep neural network | |
| Yu et al. | Epileptic seizure prediction using deep neural networks via transfer learning and multi-feature fusion | |
| CN115804602A (en) | Electroencephalogram emotion signal detection method, equipment and medium based on attention mechanism and with multi-channel feature fusion | |
| Ullah et al. | [Retracted] An Effective and Lightweight Deep Electrocardiography Arrhythmia Recognition Model Using Novel Special and Native Structural Regularization Techniques on Cardiac Signal | |
| Berger et al. | Generative adversarial networks in electrocardiogram synthesis: Recent developments and challenges | |
| Farokhah et al. | Simplified 2D CNN architecture with channel selection for emotion recognition using EEG spectrogram | |
| CN118094317A (en) | Motor imagery electroencephalogram signal classification system based on TimesNet and convolutional neural network | |
| CN113128459A (en) | Feature fusion method based on multi-level electroencephalogram signal expression | |
| Cao et al. | IFBCLNet: Spatio-temporal frequency feature extraction-based MI-EEG classification convolutional network | |
| Dissanayake et al. | Multi-stage stacked temporal convolution neural networks (MS-S-TCNs) for biosignal segmentation and anomaly localization | |
| CN115316955A (en) | Light-weight and quick decoding method for motor imagery electroencephalogram signals | |
| Stuchi et al. | A frequency-domain approach with learnable filters for image classification | |
| Xu et al. | Trends and challenges of processing measurements from wearable devices intended for epileptic seizure prediction | |
| Ren et al. | Extracting and supplementing method for EEG signal in manufacturing workshop based on deep learning of time–frequency correlation | |
| Liu et al. | A learnable front-end based efficient channel attention network for heart sound classification | |
| Mardiansyah et al. | Multivariate eeg signal using pca and cnn in post-stroke classification | |
| Parveen et al. | EEG-based Emotion Classification-A Theoretical Perusal of Deep Learning Methods | |
| Cao et al. | BiCCT: a compact convolutional transformer for EEG emotion recognition | |
| Chithra et al. | Heart Disease Detection Using Ensemble Machine Learning Technique Based on Various Risk Factors | |
| Guo et al. | Emotion Recognition based on Physiological Signals Multi-head Attention Contrastive Learning | |
| Li et al. | A Novel Artifact Removal Strategy and Spatial Attention-based Multiscale CNN for MI Recognition |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WW01 | Invention patent application withdrawn after publication |
Application publication date: 20221111 |
|
| WW01 | Invention patent application withdrawn after publication |