KR102143920B1 - Visual stimulation-based brain-computer interface apparatus and method of processing information thereof - Google Patents

Abstract

The present invention relates to a visual stimulation-based brain-computer interface device and an information processing method thereof, wherein in the information processing method performed by the visual stimulation-based brain-computer interface device, a) a display module to which visual stimulation is provided, and Extracting a gaze point of the user based on reference distance information from the user, and calculating an entire viewing area based on the gaze point; b) setting a first range as a central viewing area based on the gaze point in the entire viewing area, and setting a second range excluding the first range as a peripheral viewing area; c) presenting a first visual field stimulus to the central visual field and a second visual field stimulus to the peripheral visual field based on the previously measured visual fatigue; d) measuring a user's EEG by the first visual field stimulation or the second visual field stimulation, and extracting brain signal characteristic information for a frequency band or a time band from the measured EEG; And e) detecting a user intention stimulus gazed by the user by using the brain signal characteristic information, and outputting the detected user intention stimulus through the display module.

Description

Visual stimulation-based brain-computer interface device and its information processing method {VISUAL STIMULATION-BASED BRAIN-COMPUTER INTERFACE APPARATUS AND METHOD OF PROCESSING INFORMATION THEREOF}

The present invention relates to a brain-computer interface device based on a visual stimulus with high recognition performance while presenting different visual stimuli with low visual fatigue, and an information processing method thereof.

Electroencephalogram (EEG) refers to a measurement of a current signal generated in the brain with an electrode. These brain waves can be largely divided into spontaneous brain waves and induced brain waves. Spontaneous brain waves are brain signals that are expressed by the user's will and are generated when they move or imagine movements spontaneously. Induced brain waves are brain signals that are naturally expressed irrespective of the user's will by external stimuli and are induced by visual, auditory, and tactile stimuli.

On the other hand, the brain-computer interface is a research field that allows you to control an input device or a wheelchair, etc. of a computer only with thoughts using brain signals.

One of the representative brain signals, the Steady State Visual Evoked Potential (SSVEP), is a natural response signal to an external stimulus, which is induced in the occipital cortex when a repetitive visual stimulus is concentrated in a specific frequency band. Say the signal. In general, in order to use SSVEP, a method of continuously blinking a visual stimulus of a specific frequency (for example, 10Hz) on an LCD monitor is used. However, the corresponding visual stimulus is repeatedly exposed to the user for a period of typically 3 to 10 seconds, and this method makes the user feel extreme eye strain. That is, the SSVEP-based BCI system gives considerable visual fatigue to the user's eyes, and this visual fatigue leads to a decrease in concentration, making it difficult to use for a long time, and in rare cases, users with photosensitive constitutions have a limitation that may cause seizures.

As a BCI technology for controlling a device by presenting a visual stimulus to a user using this SSVEP and analyzing the user's intention through brain waves, the following technologies have been proposed.

Republic of Korea Patent Registration No. 10-1293863 (name of the invention: QWERTY type character input interface device and character input method using a stable state visual oil power plant) proposes a QWERTY type character interface device, and visually induced potential through brain waves Disclosed is a method of detecting a user's intention from and inputting a character. And Republic of Korea Patent Registration No. 10-1389015 (name of the invention: EEG analysis system using amplitude-modulated steady state visual induced potential visual stimulation), amplitude-modulated time to establish a stable BCI system when using a stable state visual induced potential. A configuration that presents a stimulus is disclosed. In addition, Korean Patent Registration No. 10-1465878 (title of the invention: object control method and apparatus using a stable-state visual evoked potential) is linked with a projection-type display device to allocate a visual stimulus pattern to an object, and Disclosed is a method of controlling an object through an EEG signal according to a pattern gaze.

The above technologies variously propose BCI-based device control using the steady state visual evoked potential, but all of them directly generate SSVEP-based visual stimuli to the central field of view, thereby reducing user fatigue when exposed repeatedly for a long period of time. There is a limit to generating it.

Therefore, there is a need for a visual stimulation-based BCI technology capable of maintaining BCI performance while improving visual stimulation to reduce user fatigue.

Republic of Korea Patent Registration No. 10-1293863 (Name of invention: ＱＷＥＲＴＹ type character input interface device and character input method using stable state visual oil power plant)

Republic of Korea Patent Registration No. 10-1389015 (Name of invention: EEG analysis system using amplitude-modulated stable-state visual stimulus)

Korean Patent Registration No. 10-1465878 (Name of invention: Object control method and apparatus using stable-state visual induced potential)

In order to solve the above problems, according to an embodiment of the present invention, visual stimulation of a specific frequency is provided to the peripheral field of view, avoiding the gaze point, which is the central field of view, thereby reducing visual fatigue, It is to provide a visual stimulation-based brain-computer interface device capable of enhancing recognition performance by providing a potential-based visual stimulation and an information processing method thereof.

However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, the information processing method of the brain-computer interface device based on visual stimulation according to an embodiment of the present invention is an information processing method performed by the brain-computer interface device based on visual stimulation. A method comprising: a) extracting a gaze point of a user based on reference distance information between a display module provided with a visual stimulus and a user, and calculating an entire viewing area based on the gaze point; b) setting a first range as a central viewing area based on the gaze point in the entire viewing area, and setting a second range excluding the first range as a peripheral viewing area; c) presenting a first visual field stimulus to the central visual field and a second visual field stimulus to the peripheral visual field based on the previously measured visual fatigue; d) measuring a user's EEG by the first visual field stimulation or the second visual field stimulation, and extracting brain signal characteristic information for a frequency band or a time band from the measured EEG; And e) detecting a user intention stimulus gazed by the user by using the brain signal characteristic information, and outputting the detected user intention stimulus through the display module.

In addition, a visual stimulation-based brain-computer interface device according to another embodiment of the present invention includes a visual stimulation-based brain-computer interface device, comprising: a display module providing visual stimulation; A memory in which a program for performing an information processing method for analyzing a user intention stimulus by processing an EEG signal of a user by different visual stimuli provided through the display module is recorded; And a processor for executing the program, wherein the processor extracts a gaze point of the user based on reference distance information between the display module and the user by executing the program, and based on the gaze point A first range is set as a central viewing area, a second range excluding the first range is set as a peripheral viewing area, and a first viewing stimulus is presented in the central viewing area based on a pre-measured visual fatigue level, Presenting a second field of view stimulus to the peripheral field of view, measuring the user's EEG by the first field of view stimulation or the second field of view stimulation, and extracting brain signal characteristic information for a frequency band or time band from the measured EEG After that, the brain signal characteristic information is used to detect the user's intentional stimulus that the user stares at.

According to the above-described problem solving means of the present invention, by providing a visual stimulus based on a stable-state visual-induced potential to the peripheral field of view, avoiding the gaze point, which is a central field of view, and providing a visual stimulus based on an event-related potential to the central field of view, the user With low visual fatigue, they can control external devices using brain waves with concentration, and recognition performance can be improved while reducing visual fatigue by using recognition results of different visual stimuli for user intention analysis.

Accordingly, in the present invention, in addition to the safety state visual induced potential and event-related potential, coupling with other visual stimuli can be facilitated, and the BCI device can be used for a long time by reducing visual fatigue.

1 is a diagram illustrating a configuration of a brain-computer interface device based on visual stimulation according to an embodiment of the present invention.
2 is a flowchart illustrating an information processing method of a brain-computer interface device based on visual stimulation according to an exemplary embodiment of the present invention.
FIG. 3 is a diagram illustrating a central viewing area and a peripheral viewing area according to FIG. 2.
FIG. 4 is a diagram illustrating a screen providing visual field stimulation in the form of a combination of SSVEP stimulation and ERP stimulation according to FIG. 2.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, the present invention may be implemented in various different forms and is not limited to the embodiments described herein. In the drawings, parts irrelevant to the description are omitted in order to clearly describe the present invention, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" with another part, this includes not only "directly connected" but also "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, it means that other components may be further included, and one or more other features, not excluding other components, unless specifically stated to the contrary. It is to be understood that it does not preclude the presence or addition of any number, step, action, component, part, or combination thereof.

The following examples are detailed descriptions for aiding understanding of the present invention, and do not limit the scope of the present invention. Accordingly, the invention of the same scope performing the same function as the present invention will also belong to the scope of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram illustrating a configuration of a brain-computer interface device based on visual stimulation according to an embodiment of the present invention.

Referring to FIG. 1, the brain-computer interface device 100 based on visual stimulation includes, but is not limited to, a display module 110, a memory 120, and a processor 130.

The display module 110 provides different visual stimuli on the monitor, and outputs and displays the user intention stimulus analyzed by the processor 130.

The display module 110 includes a visual stimulus presentation unit 111 and a stimulus output unit 112, wherein the visual stimulus presentation unit 111 provides different visual stimuli to the user through a monitor, and the stimulus output unit 112 User intention stimulus is output through EEG analysis and displayed on the monitor.

The memory 120 records a program for performing an information processing method for analyzing user intention stimuli by processing the user's EEG signals by different visual stimuli. In addition, the processor 130 performs a function of temporarily or permanently storing the data processed. Here, the memory 120 may include a volatile storage medium or a non-volatile storage medium, but the scope of the present invention is not limited thereto.

The processor 130 controls the entire process of providing an information processing method for detecting a user intention stimulus using a steady state visual evoked potential (SSVEP) and an event-related potential (ERP) based on a visual stimulus.

The processor 130 includes an EEG signal measuring unit 131, a feature extracting unit 132, a feature combining unit 133, and a user intention analysis unit 134.

The EEG measurement unit 131 extracts EEG from 27 or more channels according to the international electrode system. When the user wears the cap with the microchip attached to the scalp in a non-invasive manner, the EEG measurement unit 131 puts a gel that is harmless to the human body to lower the impedance value, measure a significant EEG signal, and consider different EEG signals for each user. A reference channel and a ground channel are used.

In order to extract a strong signal from the visual stimulus provided through the display module 110, the feature extracting unit 132 converts the brain signal characteristic information in the frequency band and the brain signal characteristic information in the new book band from the brain wave measured through fast Fourier transform. Extract each.

The feature combiner 133 combines brain signal feature information in a frequency band and a time band so that all brain signal features resulting from different stimuli can be used.

The user intention analysis unit 134 analyzes which character the user intends and gazes by using the characteristic of the brain signal, determines the user intention stimulus, and outputs it.

Each of the steps performed by the EEG signal measurement unit 131, the feature extraction unit 132, the feature combination unit 133, and the user intention analysis unit 134 of the processor 130 will be described later with reference to FIG. 2. do.

Here, the processor 130 may include all types of devices capable of processing data, such as a processor. Here, the'processor' may refer to a data processing device embedded in hardware, which has a circuit physically structured to perform a function represented by a code or instruction included in a program. As an example of a data processing device built into the hardware as described above, a microprocessor, a central processing unit (CPU), a processor core, a multiprocessor, and an application-specific integrated (ASIC) circuit), a field programmable gate array (FPGA), and the like, but the scope of the present invention is not limited thereto.

The visual stimulation-based brain-computer interface device 100 includes a communication module (not shown) that provides a communication interface to transmit and receive data, such as an external device or a cap for measuring an EEG signal worn by a user, and each user. It may further include a database (not shown) for storing the measured brain signal characteristic information and data for visual stimulation. Here, the communication module may be a device including hardware and software necessary for transmitting and receiving a signal such as a control signal or a data signal through a wired or wireless connection with another device.

FIG. 2 is a flowchart illustrating an information processing method of a brain-computer interface device based on a visual stimulus according to an embodiment of the present invention, and FIG. 3 is a diagram illustrating a central viewing area and a peripheral viewing area according to FIG. 2, FIG. 4 is a diagram illustrating a screen providing visual field stimulation in the form of a combination of SSVEP stimulation and ERP stimulation according to FIG. 2.

First, referring to FIGS. 2 to 4, the information processing method of the brain-computer interface device based on visual stimulation extracts a gaze point of a user based on reference distance information between a display module provided with visual stimulation and a user, The entire field of view is calculated based on the extracted gaze point. In this case, the first range is set as the central viewing area based on the gaze point, and the second range excluding the first range is set as the peripheral viewing area (S110).

When the reference distance information between the display module and the user is 70 cm, the first range (a) is set within 1.25° of the viewing angle based on the gaze point, and the second range (b) is the viewing angle based on the gaze point. Set within 2.5° from 1.25°.

Therefore, as shown in FIG. 3, the entire field of view is in the form of an outer circle with a diameter of 6 cm, the central field of view is in the form of an inner circle with a diameter of 3 cm, and the peripheral field of view is in the form of a circular band excluding the inner circle from the outer circle. Becomes. That is, the peripheral viewing area is formed in the shape of a circular band having a thickness of 3 cm by subtracting the diameter of the inner circle of 3 cm from the diameter of the outer circle of 6 cm. Meanwhile, such specific values are exemplary for convenience of description, and the present invention is not limited thereto.

The peripheral visual field formed in this way is a region capable of extracting sufficient neural signals from the brain while reducing the mental burden because it is neurophysiologically less sensitive to visual information.

Therefore, the visual stimulation-based brain-computer interface device presents the first visual field stimulus with low visual fatigue in the central visual field based on the pre-measured visual fatigue level, and the visual fatigue is somewhat less than the first visual fatigue stimulus in the peripheral visual field. A second high visual field stimulation is presented (S120).

In this case, visual fatigue may be measured by visual fatigue through a visual fatigue analysis method based on various biological signals or brain wave analysis, or eye image analysis based on camera vision.

ERP is a signal that appears as a result of a user's response when the same external stimulus such as sight and hearing is repeatedly provided to a user, and refers to a potential difference occurring after 300 ms at the time of providing a specific stimulus. The ERP-based visual stimulus is an EEG signal with high resolution in a time band, and the ERP-based BCI device has the advantage of being slow but low visual fatigue.

In addition, the SSVEP-based visual stimulus is information of a frequency band due to the characteristic of the EEG signal, and the ERP-based visual stimulus can be divided into information of a time band due to the characteristic of the EEG signal. That is, the SSVEP-based visual stimulus analyzes the time signal in terms of frequency, and compares two data using canonical phase-gon analysis to calculate a correlation value. In addition, the ERP-based visual stimulus performs a linear discriminant analysis in terms of time to calculate a discriminant function, and analyzes brain waves through the discriminant function.

Therefore, as shown in FIG. 4, the brain-computer interface device based on visual stimulation provides ERP-based visual stimulation generated by blinking (i.e., Flash) once in a random order in the central field of view. It provides an SSVEP-based visual stimulus that occurs by repeated flickering at a specific frequency in the field of view. To this end, the visual stimulation-based brain-computer interface device simultaneously outputs SSVEP-based visual stimulus and ERP-based visual stimulus to a monitor operating at 60 Hz or 120 Hz.

The visual stimulus-based brain-computer interface device generates an SSVEP frequency domain (eg, a divisor of the monitor frequency) that can be output in the frequency domain through the display module, and inserts the SSVEP-based visual stimulus between each frame. At the same time, ERP-based visual stimulus generates time between ERP-based visual stimuli ((Inter-Stimulus Interval, ISI), ERP-based visual stimulus output time (Stimulus-Time Interval, STI) and ERP-based between each frame. After inserting the visual stimulus, the two different visual stimuli within a single system are consistently output to the monitor without error.

Referring back to FIG. 2, the brain-computer interface device based on visual stimulation measures the user's brain waves by the first visual field stimulation or the second visual field stimulation (S130). In a state where a user wears a cap with a microchip attached to the scalp in a non-invasive manner, a brain-computer interface device based on visual stimulation extracts EEG from more than 27 channels according to the international electrode system.

The brain-computer interface device based on visual stimulation extracts brain signal feature information in a frequency band and brain signal feature information in a time band through fast Fourier transform in order to extract a robust signal from the measured brain wave signal (S140).

The EEG signal measured through SSVEP-based visual stimulation is calculated by calculating the first analysis data and the second analysis data through CCA through signal analysis in the frequency band, and comparing the calculated first analysis data and the second analysis data. Calculate the correlation value. In addition, the ERP signal measured through the ERP-based visual stimulus generated at 300 ms after the stimulation is provided is calculated using the LDA in the time band.

Thereafter, the visual stimulation-based brain-computer interface device probably calculates the stimulus gazed by the user by combining the correlation value, which is the characteristic information of the brain signal in the frequency band, and the discrimination function, which is the characteristic information of the brain signal in the time band ( S150).

Therefore, the brain-computer interface device based on visual stimulus analyzes what stimulus the user intended and gazes by using both the characteristics of brain signals in the frequency band and the time band, and detects the user's intended stimulus that the user gazes at, and Is output through the display module (S160).

As described above, the present invention combines the SSVEP stimulus in the peripheral field of view, not the user's gaze point, and the ERP stimulus in the central field of view including the gaze point, and presents it to the user, so that the recognition result through the CCA in the frequency band and the LDA in the time band. By effectively combining the recognition results, the stimulus intended by the user can be accurately detected.

The present invention is applied to a brain-computer interface technology in which a computer, a smart phone, a robot, etc. controls and adjusts an external device with brain waves, so that a TV channel change, door opening/closing, smart home control technology or user's thoughts It can be used as a variety of communication aids, such as interpreting as a picture.

The information processing method of the brain-computer interface device based on visual stimulation according to the embodiment of the present invention described above is also implemented in the form of a recording medium including instructions executable by a computer such as a program module executed by a computer. Can be. Such recording media include computer-readable media, and computer-readable media may be any available media that can be accessed by a computer, and include both volatile and nonvolatile media, and removable and non-removable media. In addition, computer-readable media includes computer storage media, which are volatile and nonvolatile implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. , Both removable and non-removable media.

The above description of the present invention is for illustrative purposes only, and those of ordinary skill in the art to which the present invention pertains will be able to understand that other specific forms can be easily modified without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative and non-limiting in all respects. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as being distributed may also be implemented in a combined form.

The scope of the present invention is indicated by the claims to be described later rather than the detailed description, and all changes or modified forms derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

100: visual stimulation-based brain-computer interface device
110: display module 120: memory
130: processor

Claims (12)

In the information processing method performed by a visual stimulation-based brain-computer interface device,
a) extracting a gaze point of the user based on the reference distance information between the display module provided with the visual stimulus and the user, and calculating the entire viewing area based on the gaze point;
b) setting a first range as a central viewing area based on the gaze point in the entire viewing area, and setting a second range excluding the first range as a peripheral viewing area;
c) presenting a first visual field stimulus to the central visual field and a second visual field stimulus to the peripheral visual field based on the previously measured visual fatigue;
d) measuring a user's EEG by the first visual field stimulation or the second visual field stimulation, and extracting brain signal characteristic information for a frequency band or a time band from the measured EEG; And
e) detecting a user intention stimulus gazed by the user using the brain signal characteristic information, and outputting the detected user intention stimulus through the display module,
The first visual field stimulation is an event related potential (ERP) based visual field stimulation,
The second visual field stimulation is a visual field stimulation based on Steady State Visual Evoked Potential (SSVEP),
Information processing method of brain-computer interface device based on visual stimulation. delete The method of claim 1,
The step c),
As the first visual field stimulus, a visual stimulus randomly generated in a preset time band for analysis of an EEG signal in a time band is provided,
As the second visual field stimulation, a visual stimulation that is repeatedly generated in a preset frequency band for analysis of an EEG signal in a frequency band is provided. The method of claim 1,
Step b),
The visual stimulation-based brain-computer interface device is to set the viewing angle within 1.25° as the first range based on the gaze point and the viewing angle within 1.25° to 2.5° as the second range based on the gaze point Of information processing. The method of claim 4,
The entire viewing area is formed in the shape of an outer circle having a first radius based on the reference distance information,
The central viewing area is formed in an inner circle shape having a second radius based on the reference distance information,
The peripheral field of view is the entire field of view, except for the central field of view, and is formed in a circular band shape. The method of claim 1,
Step d)
In the case of the first visual field stimulation, a discriminant function is calculated through linear discriminant analysis (LDA) in a time band, and an EEG is analyzed through the calculated discriminant function,
In the case of the second field of view stimulus, the first analysis data and the second analysis data are respectively calculated through the signal analysis in the frequency band and the canonical correlation analysis (CCA), and the calculated first analysis data and the second analysis data are 2 Comparing analysis data to extract a correlation value to analyze brain waves, visual stimulation-based brain-computer interface device information processing method. The method of claim 6,
Step e)
Visual stimulus-based brain that probabilistically calculates the user's intention stimulus by combining characteristic information of a brain signal for a time band using the discrimination function and characteristic information of a brain signal for a frequency band using the correlation value -Information processing method of computer interface devices. In the brain-computer interface device based on visual stimulation,
A display module to which visual stimulation is provided;
A memory in which a program for performing an information processing method for analyzing a user intention stimulus by processing an EEG signal of a user by different visual stimuli provided through the display module is recorded; And
Includes; a processor for executing the program,
The processor, by executing the program,
The user's gaze point is extracted based on the reference distance information between the display module and the user, a first range is set as a central viewing area based on the gaze point, and a second range excluding the first range is a peripheral view. Set to the area,
Presenting a first field of view stimulus to the central field of view based on a pre-measured visual fatigue level, and a second field of view stimulus to the peripheral field of view,
After measuring the user's EEG by the first visual field stimulation or the second visual field stimulation, extracting brain signal characteristic information for a frequency band or a time band from the measured brain wave, the user gazes by using the brain signal characteristic information. To detect user intention stimulus,
The first visual field stimulation is an event related potential (ERP) based visual field stimulation,
The second visual field stimulation is a visual field stimulation based on Steady State Visual Evoked Potential (SSVEP),
Brain-computer interface device based on visual stimulation. The method of claim 8,
The processor sets a viewing angle within 1.25° as a first range based on the gaze point, and sets a viewing angle within 2.5° to 1.25° as a second range based on the gaze point, a visual stimulation-based brain- Computer interface device. The method of claim 8,
The display module,
The visual stimulation-based brain-computer interface device comprising a stimulation output unit for outputting and displaying the user intended stimulation. The method of claim 8,
The processor,
An EEG signal measuring unit for measuring a user's EEG signal by the first field stimulus and the second field stimulus through a plurality of channels;
A feature extraction unit for extracting brain signal feature information in a frequency band or a time band from the brain wave signal;
A feature combining unit that combines the brain signal feature information in the frequency band or the time band; And
A visual stimulation-based brain-computer interface device comprising a user intention analysis unit probabilistically calculating the user intention stimulus gazed by the user using the combined brain signal characteristic information to determine and output the user intention stimulus . The method of claim 11,
The feature extraction unit,
When the first visual field stimulus is an event related potential (ERP) based visual field stimulus, a discriminant function is calculated through linear discriminant analysis (LDA) in a time band, and the calculated discriminant function is Analyzing brain waves through,
When the second field stimulus is a field stimulus based on Steady State Visual Evoked Potential (SSVEP), the first analysis data and canonical correlation analysis (CCA) are performed through signal analysis in the frequency band. A brain-computer interface device based on visual stimuli that calculates second analysis data through each, and compares the calculated first analysis data with the second analysis data to extract a correlation value to analyze brain waves.

Images