WO2019235458A1

WO2019235458A1 - Recalled image estimation device, recalled image estimation method, control program, and recording medium

Info

Publication number: WO2019235458A1
Application number: PCT/JP2019/022113
Authority: WO
Inventors: 琢史 ▲柳▼澤; 良平福間; 晴彦貴島; 伸志西本
Original assignee: 国立大学法人大阪大学; 国立研究開発法人情報通信研究機構
Priority date: 2018-06-04
Filing date: 2019-06-04
Publication date: 2019-12-12
Also published as: JPWO2019235458A1; JP7352914B2

Abstract

The present invention estimates, with high accuracy, a target image that is being recalled by a person. A recalled image estimation device (10) is provided with: a multi-point electric potential measurement unit (1) which measures electric characteristics of the brain of a person at multiple measurement points in a brain region including a visual association cortex; a decoder (2) which, from the electric characteristics measured while the subject is visually recognizing candidate images, estimates decoding information indicative of the content of a target image being recalled by the person; and an image determination unit (3) which, on the basis of the estimated decoding information, determines a candidate image for causing the person to perform visual recognition.

Description

Recall image estimation device, recall image estimation method, control program, recording medium

The present invention relates to estimation of a recall image, and more particularly to a recall image estimation device that supports presentation of an arbitrary recalled image.

There are many patients whose body motor function is extremely reduced due to progressive neurological diseases such as amyotrophic lateral sclerosis (ALS) and cerebrovascular disorders. One such assistive technology for patients is the brain-machine-interface (BMI).

BMI measures the action potential of a patient's cranial nerve cells or cortical electroencephalograms to interpret motor intentions, controls the operation of machines such as robot arms, and controls to select and input characters intended by the patient. Technology that makes it possible to According to a questionnaire conducted on ALS patients, there is a strong demand to support patient communication using BMI technology.

For example, in Patent Document 1, the presented image and the electrical characteristics measured at a plurality of measurement points in the region including the visual association area of the brain when the image is presented are measured in association with each other. A communication support apparatus that supports communication by specifying an image to be transmitted based on the electrical characteristics is disclosed.

Non-Patent Document 1 describes a method in which the firing activity of a nerve cell recorded from the hippocampus of a subject is measured, and the subject considers one of the images by overlapping the two images. A technique capable of strongly displaying the image is disclosed.

JP 2010-257343 A (published on November 11, 2010)

However, the conventional techniques as described above have a problem that it is impossible to estimate an arbitrary image recalled with high accuracy.

For example, since the communication support apparatus described in Patent Literature 1 determines an image to be displayed based on the electrical characteristics associated with the presented image, the displayable image is limited to the presented image. , Can not display any recalled image.

Also, in the technique described in Non-Patent Document 1, a subject viewing a state where two images overlap each other causes the image on the side toward which the consciousness is directed to be strongly displayed by directing consciousness to one of the images. Although it can, it does not display any recalled image.

An object of one aspect of the present invention is to realize a recall image estimation apparatus and a recall image estimation method that accurately estimate a target image recalled by a subject.

In order to solve the above problems, a recall image estimation device according to one aspect of the present invention is a multipoint potential that measures electrical characteristics of a subject's brain at a plurality of measurement points in a brain region including a visual association area. Estimated by the measurement unit, a decoder for estimating decoding information indicating the content of the target image recalled by the subject from the electrical characteristics measured while the subject visually recognizes the candidate image, and the decoder An image determining unit that determines a candidate image to be visually recognized by the subject based on the decoded information.

The recall image estimation method according to the present invention is measured at a plurality of measurement points in the brain region including the visual association area while the subject visually recognizes the candidate image in order to solve the above problem. From the electrical characteristics of the brain, an estimation step for estimating decoding information indicating the content of the target image recalled by the subject, and a candidate image to be visually recognized by the subject based on the decoding information estimated in the estimation step An image determining step for determining.

According to one aspect of the present invention, a target image recalled by a subject can be accurately estimated.

It is a functional block diagram which shows the schematic structural example of the recall image estimation apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows an example of the flow of a process of the recall image estimation apparatus. It is a functional block diagram which shows an example of schematic structure of the recall image estimation apparatus which performs the machine learning for producing | generating a decoder. (A) is a flowchart which shows an example of the method of producing | generating a decoder by machine learning, (b) is a flowchart which shows the preparation process of the decoding information which shows the image for learning and the content of each image. It is a figure which shows an example of the explanatory image explaining the content of the image for learning for producing | generating a decoder, and the image for learning. It is an image figure explaining an example of the procedure which produces | generates a decoder using the image for learning. It is a graph which shows the result of one Example of this invention. It is a graph which shows the result of the other Example of this invention. It is a functional block diagram which shows the schematic structural example of the recall image estimation apparatus which concerns on one Embodiment of this invention.

Embodiment 1
Hereinafter, the recall image estimation apparatus 10 according to an embodiment of the present invention will be described in detail. The recall image estimation device 10 is a device that estimates the decoding information indicating the content of the target image of the target image recalled by the subject, and determines a candidate image to be visually recognized by the subject based on the estimated decoding information. . The recall image estimation device 10 does not determine a candidate image based on a one-to-one correspondence between the image visually recognized by the subject and the electrical characteristics of the subject's brain B when the image is viewed. Absent. Therefore, the recall image estimation apparatus 10 can determine an arbitrary candidate image that is not an image visually recognized by the subject in advance as a candidate image to be visually recognized by the subject. Therefore, the recall image estimation device 10 is a device that supports the subject to be able to present any images and images that the subject desires to present outside. Here, the “candidate image” is intended to be an image visually recognized by the subject in order to measure the electrical characteristics of the brain B, and the “target image” is recalled while the subject visually recognizes the candidate image. Is intended (ie, the image that the subject wants to present).

(Configuration of Recall Image Estimation Device 10)
First, the configuration of the recall image estimation device 10 will be described with reference to FIG. FIG. 1 is a functional block diagram showing a schematic configuration example of a recall image estimation apparatus 10 according to an embodiment of the present invention. In the following, a configuration in which the recall image estimation device 10 includes the display unit 5 will be described as an example, but the present invention is not limited to this. For example, a configuration in which an external display device is applied instead of the display unit 5 may be used.

As shown in FIG. 1, the recall image estimation device 10 includes a multipoint potential measurement unit 1, a decoder 2, an image determination unit 3, a display control unit 4, a display unit 5, and a storage unit 6.

The multipoint potential measuring unit 1 measures the electrical characteristics of the subject's brain B at a plurality of measurement points in the region of the brain B including the visual association area. More specifically, the multipoint potential measurement unit 1 includes a plurality of electrodes E, and measures a cortical electroencephalogram (Electro-Cortico-Graphy: ECoG) of the brain B (low invasive configuration). In this case, the electrode E is an ECoG electrode placed under the dura mater. The electrode E is an electrode for detecting the cortical potential generated in the brain B of the subject who is viewing the image. Electrode E can be placed on the surface of the brain B cerebral cortex that contains the visual association area and on the surface of the sulcus. In addition, the number of the electrodes E should just be plural (for example, 100), and is not specifically limited.

The multipoint potential measuring unit 1 is not limited to the configuration for measuring the cortical potential. For example, the multipoint potential measuring unit 1
・ Configuration to measure action potential (Multi-unit Activity: MUA) of nerve cell using electrode inserted into brain B as electrode E (invasive configuration)
・ Structure for measuring electroencephalogram (stereotactic Electro-Graphy: stereotactic EEG) using an insertion electrode in brain B as electrode E (invasive structure)
・ Scalp Electro-Encephalo-Graphy (scalp EEG) measurement using electrode E placed on scalp (non-invasive configuration)
・ Configuration to measure intravascular electro-encephalogram (intravascular EEG) using electrode E placed in cerebral blood vessel (minimally invasive configuration)
・ Either a configuration (non-invasive configuration) for measuring a magnetic field generated by an electrical activity of the brain B using a magnetoencephalogram (Magneto-Encephalo-Graphy: MEG) sensor as the electrode E Good.

However, the sensitivity of the electrical characteristics of the brain B to be measured is generally in the order of scalp EEG <MEG <intravasual EEG <stereotactic EEG <ECoG <MUA. On the other hand, the burden on the subject's body is in the order of scalp EEG = MEG <intravasular EEG <stereotactic EEG = ECoG <MUA. In consideration of the accuracy to be achieved and the burden on the body of the subject, MEG and ECoG are desirable as the multipoint potential measuring unit 1.

When applying a configuration in which the cortical potential is measured by the multipoint potential measuring unit 1, for example, an alpha wave (8 to 13 Hz), a beta wave (13 to 30 Hz), a low frequency gamma wave (30 to 80 Hz), and a high frequency gamma wave An electroencephalogram in each band (80 to 150 Hz) can be applied.

The decoder 2 estimates decoding information indicating the contents of the target image recalled by the subject from the electrical characteristics measured while visually recognizing the candidate image. Here, “decoding information” is information indicating the content and meaning of an image. More specifically, “decoding information” is information representing the content and meaning of an image as a vector in a semantic space (which may be expressed as a “word vector space”). A method of expressing the content of an image as a vector in a semantic space will be described later with a specific example.

The decoder 2 may be a learned neural network. The learning for creating the decoder 2 is generated in advance using a predetermined candidate image and a word vector corresponding to one or more words included in one or more explanatory sentences explaining the contents of the predetermined candidate image. Teacher decoding information is used. In this case, the decoder 2 includes an input layer and an output layer, and when the electrical characteristics of the brain B measured while viewing the predetermined candidate image are input to the input layer, the predetermined candidate Learning is performed so that the teacher decoding information associated with the image is output from the output layer. A process of generating the decoder 2 by learning will be described later with a specific example.

The image determination unit 3 determines a candidate image to be visually recognized by the subject based on the decoding information estimated by the decoder 2. More specifically, the image determination unit 3 causes the candidate image associated with the same or similar decoding information as the decoding information estimated by the decoder 2 to be viewed following the candidate image that is being viewed by the subject. Determine as a candidate image.

The display control unit 4 controls the display unit 5 to display the candidate image determined by the image determination unit 3. Further, the display control unit 4 controls the display unit 5 to display a predetermined candidate image prepared for learning in the process of generating the decoder 2 by learning.

The display unit 5 is a display that displays an image. The subject recalls an arbitrary target image while visually recognizing the image displayed on the display unit 5.

The storage unit 6 stores candidate images to be displayed on the display unit 5. Each candidate image is associated with decoding information indicating the contents of each candidate image. When the recall image estimation device 10 also has a function of performing machine learning (supervised learning) of the decoder 2, the storage unit 6 corresponds to each learning image (predetermined candidate image) and each learning image. The attached decoding information (teacher decoding information) is stored.

According to said structure, the decoding information which shows the content of the image which the said test subject is recalling is estimated from the electrical property of the brain B of the test subject who is visually recognizing the candidate image, and based on the estimated decoding information The subject is made to visually recognize the determined image. Thus, a closed-loop control mechanism can be configured in which the subject image is visually recognized by the subject, the decoding information is estimated, and the next candidate image is determined based on the estimated decoding information. Here, the closed-loop control mechanism is a “closed control mechanism” in which a candidate image to be visually recognized by the subject is determined from the electrical characteristics of the brain B measured when the subject is visually recognizing the candidate image. Is intended.

By applying such a closed-loop control mechanism, the subject repeats the process of visually recognizing the candidate image while recalling the desired target image. Therefore, the top-down control of the brain activity by the subject himself / herself is input to the visual cortex of the brain B, and the electrical characteristics of the brain B when this top-down control is input can be measured. Therefore, the target image recalled by the subject can be accurately estimated.

Here, the top-down control is one of the forms of neural information control when the brain B processes visual information, as in the bottom-up control. Top-down control is control for selecting a target stimulus by actively biasing neural information when there is prior knowledge about a stimulus to be selected from visual information. In contrast, bottom-up control is a control that passively pays attention to a prominent stimulus, such as when a stimulus significantly different from the surrounding stimulus is included among the multiple stimuli included in the visual information. It is.

(Outline of processing flow of the recall image estimation device 10)
Next, the processing flow of the recall image estimation device 10 will be described with reference to FIG. FIG. 2 is a flowchart illustrating an example of a process flow of the recall image estimation apparatus 10.

First, the decoder 2 is generated by machine learning. Specifically, when an electrical characteristic of the brain B measured while viewing a predetermined candidate image is input, so as to output teacher decoding information associated with the predetermined candidate image, The decoder 2 is learned (step S1: decoder generation step). In the recall image estimation apparatus 10 as shown in FIG. 1, the learned decoder 2 is applied.

Next, the display control unit 4 controls the display unit 5 so that candidate images to be visually recognized by the subject are displayed (step S2: candidate image display step). The image may be a moving image including a plurality of images. It does not matter if there is sound. The candidate image that is first visually recognized by the subject is not particularly limited. For example, an arbitrary image such as a screen for notifying that the estimation process by the recall image estimation device 10 has started may be displayed.

Next, the decoder 2 estimates decoding information from the electrical characteristics of the brain B of the subject viewing the displayed candidate image measured by the multipoint potential measuring unit 1 (step S3: estimation step).

Subsequently, the image determination unit 3 determines a candidate image to be visually recognized by the subject based on the decoding information estimated by the decoder 2 (step S4: image determination step).

Then, returning to step 2, the display control unit 4 controls the display unit 5 so that the image determined by the image determination unit 3 is displayed following the subject.

In this way, by repeating the processes of steps S2 to S4, a closed-loop control mechanism is configured in which the subject visually recognizes the candidate image while recalling the desired target image.

(Configuration of Recall Image Estimation Device 10a that Performs Learning for Creating Decoder 2)
Here, the configuration of the recall image estimation apparatus 10a that generates the decoder 2 by machine learning will be described with reference to FIG. FIG. 3 is a functional block diagram illustrating an example of a schematic configuration of the recall image estimation apparatus 10a that performs machine learning for creating the decoder 2. Note that the recall image estimation device 10a may have the same function and the same configuration as the recall image estimation device 10 shown in FIG. 1 (for example, the image determination unit 3 not related to the learning of the decoder 2).

The recall image estimation device 10a includes a decoded information comparison unit 7 and a weight coefficient correction unit 8.

The decoding information comparison unit 7 uses the decoding information estimated by the decoder 2 before learning (or during learning) from the electrical characteristics of the brain B measured from the brain B of the subject viewing the learning image, and the learning The teacher decoding information associated with the image for use is compared.

The weighting factor correction unit 8 corrects the weighting factor of the decoder 2 based on the comparison result by the decoding information comparison unit 7. Specifically, the weight coefficient correction unit 8 is associated with the learning image when the electrical characteristics of the brain B measured from the brain B of the subject who is viewing the learning image are input. The current weighting factor of the decoder 2 is corrected so as to output the teacher decoding information.

By generating the decoder 2 by machine learning in this way, the decoder 2 can estimate with high accuracy decoding information indicating the content of the target image from the electrical characteristics of the brain B of the subject recalling the target image. Can be created.

(Generation of decoder 2 by machine learning)
Next, an outline of a machine learning method for creating the decoder 2 will be described with reference to FIGS. FIG. 4A is a flowchart illustrating an example of a method for generating a decoder by machine learning, and FIG. 4B is a flowchart illustrating a preparation process of a learning image and decoding information indicating the contents of each image. It is. FIG. 5 is a diagram illustrating an example of a learning image for creating the decoder 2 and an explanatory text explaining the content of the learning image.

First, learning images used for machine learning and teacher decoding information indicating the contents of each learning image are prepared (step S11: learning image preparation step).

<Preparation of learning image and teacher decoding information>
Hereinafter, a specific example of the process of preparing the learning image and the teacher decoding information will be described with reference to FIG. Of the steps shown in FIG. 4B, steps S113 to S115 can be performed using a general personal computer.

Step for preparing learning images used for machine learning (step S111).

Step for preparing an explanatory text (caption or annotation) explaining the content and meaning of the learning image for each learning image (step S112). The explanatory text may be a single sentence or may include a plurality of sentences. The explanatory text is preferably a text that simply and accurately describes the content of the image and the impression received when the image is viewed. The explanatory note may be created by showing an image to one or a plurality of people, or may be created artificially using artificial intelligence having an image recognition function. The learning image for creating the decoder 2 and the explanatory text explaining the content of the learning image will be described later with a specific example.

· A step of extracting words included in the explanatory text (step S113). A known morphological analysis engine can be applied to this step. Examples of such a known morphological analysis engine include “MeCab”, “Chasen”, “KyTea”, and the like. This process is a process that is necessary when the explanatory text is written in Japanese. If the explanatory text is written in a language in which each word is separated (for example, a space exists between words), such as English, this step is omitted.

A step of generating a word vector for each extracted word (step S114). In this step, a known tool (for example, artificial intelligence) having a function of outputting a distributed expression of words in the semantic space can be applied. Examples of such known tools include “Word2vec”, “GloVe”, “fastText”, “Doc2Vec”, and “WordNet”. For example, “Word2vec” learned using many existing sentences (for example, descriptions such as “Weekpedia” on the web) means a predetermined dimension (for example, 1000 dimensions) for each word extracted from the explanatory text. The word vector in the space can be output with high accuracy. Note that the word vector is preferably a vector in a linear space in which linear operations can be performed, but may be a word vector in a non-linear space. Note that this step can be performed in the same manner regardless of the type of language used in the description. For example, when the description is written in English, Word2vec or the like may be learned using an English version of Wikipedia or the like, and a word vector may be output using the learned Word2vec.

A step of generating teacher decoding information associated with the learning image as an average of word vectors. For words extracted from the explanatory text explaining the content of the learning image, the vector average of the word vectors generated in step S114 is obtained, and teacher decoding information indicating the content of the explanatory text is generated.

As described above, the teacher decoding information is generated by averaging vectors in the meaning space of words extracted from sentences explaining the contents of each learning image. Note that decoding information is also generated for each of the candidate images provided to the recall image estimation apparatus 10 according to the present embodiment by the processes of S111 to S115.

4 (a), next, the multipoint potential measuring unit 1 measures the electrical characteristics measured in the brain B of the subject who visually recognizes the learning image (step S12: measurement step). In this step, it is desirable that the subject merely visually recognizes the learning image without recalling the target image.

Subsequently, the decoder 2 is trained using the measured electrical characteristics as an input signal and the teacher decoding information indicating the contents of the currently viewed learning image as a teacher signal. Specifically, first, the decoding information comparison unit 7 is estimated by the decoder 2 before learning (or during learning) from the electrical characteristics of the brain B measured from the brain B of the subject viewing the learning image. The decoded information is compared with the teacher decoded information associated with the learning image. Next, when the electrical characteristics of the brain B measured from the brain B of the subject who is visually recognizing the learning image are input, the weighting coefficient correction unit 8 performs the teacher decoding associated with the learning image. The current weighting factor of the decoder 2 is corrected so as to output information.

Note that the steps S11 to S13 shown in FIG. 4 (a) do not have to be performed continuously, and can be performed individually. For example, the process of step S11 may be performed before step S12 is performed, or may be performed after step S12 is performed. Alternatively, the configuration may be such that step S12 is performed, data in which the measured electrical characteristics are associated with the image visually recognized by the subject is stored, and the data is used for the learning of the decoder 2.

<Example of learning image>
The image shown in FIG. 5 is an example of a learning image. For this image, “The family of four of my parents, daughter, and son is going on. The son is wearing a space suit and his father is photographing the situation. "Everyone feels a smile and a fun atmosphere."

In step S112, a plurality of explanatory texts may be created for one learning image (and candidate image). For example, for the learning image shown in FIG. 5, “It seems that three families are taking pictures of people wearing spacesuits with a camera. They seem to have fun and experience wearing spacesuits. "You can see a child wearing a space suit. Dad is taking a picture. I'm glad if you can have this experience."

<Creation of decoder 2>
Next, creation of the decoder 2 will be described with reference to FIG. FIG. 6 is an image diagram illustrating an example of a procedure for generating the decoder 2 using the learning image. Here, the case where the electrical characteristic of the brain B of the subject is a cortical electroencephalogram will be described as an example.

First, the cortical electroencephalogram of the brain B of the subject who is viewing the learning image is measured by the multipoint potential measuring unit 1.

Next, the measured cortical electroencephalogram is frequency-analyzed to determine the power of each band of the alpha wave, the beta wave, and the gamma wave, and these are used as a feature matrix that is input to the decoder 2.

On the other hand, using a morphological analysis engine such as MeCab, a word is extracted from the explanatory text for each image viewed by the subject, and decoding information is generated from the explanatory text. For example, the explanatory text shown in FIG. 6 “The top of the mountain with snow. The sky with clear blue and white clouds, the snowy ground and the exposed waterside mountains. In this case, words such as “snow”, “mountain”, “top”, “mode” are extracted.

Then, decoding information averaged for each element (for example, 1000 dimensions) of the extracted word vector is determined as teacher decoding information. A word vector for each extracted word is generated as a 1000-dimensional word vector using learned Word2vec.

In the learning process of the decoder 2, the weight matrix is corrected so that the teacher decoding information of each image can be output with a desired accuracy when the power of each band of the alpha wave, the beta wave, and the gamma wave is used as an input signal. .

FIG. 6 shows an example of learning to output decoding information for 3600 images using regression processing such as ridge-regulation. As an alternative to ridge-regulation, it is also possible to apply analysis methods such as deep learning and Sparse Logistic Regression (SLR).

[Embodiment 2]
Another embodiment of the present invention will be described below. For convenience of explanation, members having the same functions as those described in the above embodiment are given the same reference numerals, and the description thereof will not be repeated.

(Configuration of Recall Image Estimation Device 10a)
The configuration may be such that the candidate images to be visually recognized by the subject are not determined from the images stored in the storage unit 6 but are acquired from an arbitrary information group to be searched.

The recall image estimation device 10a uses a wide variety of images as candidate images by searching for images from the information group to be searched. First, the recall image estimation device 10a will be described with reference to FIG. FIG. 9 is a functional block diagram illustrating a schematic configuration example of the recall image estimation apparatus 10a according to the embodiment of the present invention.

The recall image estimation device 10a shown in FIG. 9 includes an image search unit 3a (image determination unit) instead of the image determination unit 3.

The image search unit 3a generates a search query using the same or similar decoding information as the decoding information estimated by the decoder 2. In addition, the image search unit 3a uses the generated search query to search for an image associated with the same or similar decoding information as the decoding information from the information group to be searched. The information group to be searched may be an arbitrary information group. For example, as shown in FIG. 9, a website A 60a and a website B 60b existing on the Internet may be included. Next, the image search unit 3a determines an image acquired as a search result as a candidate image. Moreover, the image search part 3a determines the image acquired as a search result as a candidate image which makes a subject visually recognize a candidate image.

Specifically, the decoder 2 estimates decoding information indicating the contents of the target image recalled by the subject from the electrical characteristics measured while viewing the candidate image. The decoder 2 can estimate one or more words close to the estimated decoding information (for example, a vector in the semantic space). Specifically, the decoder 2 selects several words in order of increasing distance between a vector in the semantic space of the estimated decoded information and a vector in the semantic space of each word close to the decoded information.

The image search unit 3a selects several verbs and adjectives from the words estimated by the decoder 2 and uses them for a known image search (for example, Google (registered trademark) image search). Generate a search query. The image search unit 3a can search the web for an image associated with the word estimated by the decoder 2 using the generated search query. The image search unit 3 a determines an image listed at the top in the search result as a candidate image to be displayed on the display unit 5.

If configured in this way, when applying the closed-loop control mechanism, a variety of images of a search target information group including a website existing on the Internet can be used as a candidate image to be presented to the subject. it can.

By applying the closed-loop control mechanism, the subject repeats the process of visually recognizing the candidate image while recalling the desired target image. Therefore, the subject searches for an arbitrary image from a search target information group including a website existing on the Internet by changing an electrical characteristic measured while viewing the candidate image. Can do.

Note that the recall image estimation device 10 a shown in FIG. 9 does not include the storage unit 6 that stores candidate images to be displayed on the display unit 5. However, this is only an example, and the recall image estimation device 10a may be configured to include the storage unit 6 as in the recall image estimation device 10 illustrated in FIG.

In this case, the image search unit 3a acquires the image associated with the same or similar decoding information as the decoding information estimated by the decoder 2 from the storage unit 6, the website A 60a, the website B 60b, and the like.

[Example of software implementation]
The control blocks (particularly the decoder 2, the image determination unit 3, and the display control unit 4) of the recall image estimation device 10 may be realized by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like. It may be realized by software.

In the latter case, the recall image estimation apparatus 10 includes a computer that executes instructions of a program that is software for realizing each function. The computer includes, for example, one or more processors and a computer-readable recording medium storing the program. In the computer, the processor reads the program from the recording medium and executes the program, thereby achieving the object of the present invention. As the processor, for example, a CPU (Central Processing Unit) can be used. As the recording medium, a “non-temporary tangible medium” such as a ROM (Read Only Memory), a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. Further, a RAM (Random Access Memory) for expanding the program may be further provided. The program may be supplied to the computer via an arbitrary transmission medium (such as a communication network or a broadcast wave) that can transmit the program. Note that one embodiment of the present invention can also be realized in the form of a data signal embedded in a carrier wave, in which the program is embodied by electronic transmission.

The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

[Summary]
The recall image estimation apparatus according to aspect 1 of the present invention includes a multipoint potential measurement unit that measures electrical characteristics of a subject's brain at a plurality of measurement points in a brain region including a visual association area, and the subject is a candidate image. From the electrical characteristics measured while visually recognizing, based on the decoding information estimated by the decoder, the decoder for estimating the decoding information indicating the content of the target image recalled by the subject, An image determining unit that determines candidate images to be visually recognized by the subject.

According to the above configuration, the decoding information indicating the content of the image recalled by the subject is estimated from the electrical characteristics of the brain of the subject viewing the candidate image, and is determined based on the estimated decoding information. The subject is made to visually recognize the image. Thus, a closed-loop control mechanism can be configured in which the subject image is visually recognized by the subject, the decoding content is estimated, and the next candidate image is determined based on the estimated decoding information.

By applying such a closed-loop mechanism, the subject repeats the process of visually recognizing the candidate image while recalling the desired target image. Therefore, the top-down control of brain activity by the subject himself / herself is input to the visual cortex of the brain, and the electrical characteristics of the brain when this top-down control is input can be measured. Therefore, the target image recalled by the subject can be accurately estimated.

The recall image estimation device according to aspect 2 of the present invention is the recall image estimation apparatus according to aspect 1, in which the image determination unit determines an image associated with the decoding information that is the same as or similar to the decoding information estimated by the decoder, You may determine as a candidate image made to visually recognize after the said candidate image.

In the recall image estimation device according to aspect 3 of the present invention, the image determination unit generates and generates a search query using the decoding information that is the same as or similar to the decoding information estimated by the decoder. The search query is used to search the information group to be searched for an image associated with the decoding information that is the same as or similar to the decoding information, and an image acquired as a search result is used as the candidate image. May be determined as

In the recall image estimation device according to aspect 4 of the present invention, in the aspect 3, the image determination unit determines the image acquired as the search result as a candidate image to be visually recognized following the candidate image. Also good.

By searching for images from the information group to be searched, a wide variety of images can be used as candidate images. Note that the search target information group may include websites on the Internet.

The recall image estimation device according to aspect 5 of the present invention is the recall image estimation device according to any one of the aspects 1 to 4, wherein a word vector corresponding to one or more words included in one or more explanatory texts describing the contents of a predetermined candidate image. The teacher decoding information generated in advance and the predetermined candidate image are associated with each other, and the decoder receives the electrical characteristics of the brain measured while viewing the predetermined candidate image. In such a case, the learning may be performed so that the teacher decoding information associated with the predetermined candidate image is output.

In this way, by generating the decoder by learning, a decoder capable of estimating the decoding information indicating the content of the target image with high accuracy from the electrical characteristics of the brain of the subject recalling the target image is generated. be able to.

The recall image estimation apparatus according to aspect 6 of the present invention is the recall image estimation apparatus according to any one of the aspects 1 to 5, wherein the decoder measures the cortical potential of the brain and the electrical brain, which are measured while viewing the candidate image. The decoding information indicating the contents of the candidate image may be estimated using at least one of the magnetic fields generated by the active activity.

In addition, in order to solve the above-described problem, the recall image estimation method according to an aspect of the present invention provides a plurality of measurement points in a brain region including a visual association area while a subject visually recognizes a candidate image. Based on the measured electrical characteristics of the brain, the estimation step of estimating the decoding information indicating the content of the target image recalled by the subject, and the subject is made to visually recognize based on the decoding information estimated in the estimation step And an image determining step for determining a candidate image.

A control program for causing a computer to function as the recall image estimation device according to any one of the above aspects 1 to 6, the control program for causing the computer to function as the decoder and the image determining unit, and A computer-readable recording medium recording the control program is also included in the technical scope of the present invention.

An embodiment of the present invention will be described below.

<Verification of estimation accuracy>
The cortical electroencephalogram of the subject's brain B was measured by the multipoint potential measuring unit 1 while allowing the subject to visually recognize a 60-minute moving image including various types of meaning content. The videos to be viewed by the subjects were prepared by connecting the edited videos by dividing the introduction video of the movie into short segments. In a 60-minute video, various videos including the same video appear several times in random order. The subject was instructed to view the video without fixing the viewpoint.

The moving image visually recognized by the subject was converted into a still image (scene) every second. For each scene, explanations explaining the contents of the scene were created by a plurality of people. Moreover, the power of each band of an alpha wave, a beta wave, and a gamma wave was analyzed about the cortical electroencephalogram measured in the same 1 second.

A word was extracted from the description for each scene using MeCab. For each extracted word, a 1000-dimensional word vector was generated using Word2vec learned using Wikipedia. Each scene was associated with decoding information generated as an average of word vectors for words extracted from the explanatory text.

Using the power of cortical electroencephalogram as an input signal, machine learning was performed using the decoding information of each scene as a teacher signal, and decoder 2 was created.

In this example, it was possible to estimate decoding information indicating the content of an image with statistically significant accuracy for 3600 scenes using ridge-regulation.

This will be described with reference to FIG. The solid black line in FIG. 7 shows the frequency distribution of the correlation coefficient between the decoding information estimated from the cortical brain waves of the brain B of the subject viewing the scene and the decoding information (that is, the correct answer) associated with the scene. Is shown. On the other hand, the gray line in FIG. 7 shows the correlation between the shuffled label of the decoding information associated with each scene and the decoding information estimated from the cortical electroencephalogram of the brain B of the subject viewing the scene. The frequency distribution of numbers is shown. According to FIG. 7, it was demonstrated that the decoding information associated with the scene can be estimated with significantly high accuracy from the cortical electroencephalogram of the brain B of the subject viewing the scene.

<Demonstration of recall image estimation>
Next, it was verified whether or not it is possible to estimate the target image recalled by the subject using the recalled image estimation device 10 to which the created decoder 2 is applied.

In FIG. 8, time 0 indicates the timing when the subject is instructed to recall the image (“character”, “landscape”, etc.). The black line in FIG. 8 indicates the trial average of the correlation coefficient normalized with respect to the decoding information associated with the image including the content instructed to the subject and the decoding information estimated from the cortical brain wave of the brain B of the subject. (* P <0.05, Student's t-test). On the other hand, the gray line in FIG. 8 shows the trial average of the decoded information associated with the image that does not include the recalled image, the decoded information estimated from the cortical EEG of the subject's brain B, and the normalized correlation coefficient. Show. According to FIG. 8, it was demonstrated that the image recalled by the subject can be estimated with significantly high accuracy.

DESCRIPTION OF SYMBOLS 1 Multipoint electric potential measurement part 2 Decoder 3 Image determination part 3a Image search part (image determination part)
4 Display Control Unit 5 Display Unit 6

Storage Unit

10, 10a Recall Image Estimation Device 60a Website A
60b Website B
S1 decoder generation step S2 candidate image display step S3 estimation step S4 image determination step S11 learning image preparation step S13 learning step

Claims

A multipoint potential measurement unit that measures the electrical characteristics of the subject's brain at a plurality of measurement points in the brain region including the visual association area;
From the electrical characteristics measured while the subject is viewing the candidate image, a decoder for estimating decoding information indicating the content of the target image recalled by the subject;
An recall image estimation apparatus comprising: an image determination unit that determines a candidate image to be visually recognized by the subject based on the decoding information estimated by the decoder.
The image determination unit determines an image associated with the decoding information that is the same as or similar to the decoding information estimated by the decoder as a candidate image to be viewed after the candidate image. The recall image estimation apparatus according to claim 1.
The image determination unit
Generating a search query using the decoded information that is the same as or similar to the decoded information estimated by the decoder;
Using the generated search query, search for an image associated with the decoded information that is the same as or similar to the decoded information from the information group to be searched,
The recall image estimation apparatus according to claim 1, wherein an image acquired as a search result is determined as the candidate image.
The said image determination part determines the image acquired as the said search result as a candidate image made to visually recognize following the said candidate image, The recall image estimation apparatus of Claim 3 characterized by the above-mentioned.
Teacher decoding information generated in advance using a word vector corresponding to one or more words included in one or more explanatory texts describing the contents of the predetermined candidate image is associated with the predetermined candidate image. ,
The decoder learns to output the teacher decoding information associated with the predetermined candidate image when an electrical characteristic of the brain measured while the predetermined candidate image is viewed is input. The recall image estimation device according to any one of claims 1 to 4, wherein the recall image estimation device is provided.
The decoder uses at least one of the cortical potential of the brain and the magnetic field generated by the electrical activity of the brain, which is measured while viewing the candidate image, and indicates decoding information indicating the content of the candidate image 6. The recall image estimation device according to claim 1, wherein the recall image estimation device according to claim 1 is estimated.
Decoding that shows the contents of the target image recalled by the subject from the electrical characteristics of the brain measured at a plurality of measurement points in the brain area including the visual association area while the subject visually recognizes the candidate image An estimation step for estimating information;
An image determining step of determining a candidate image to be visually recognized by the subject based on the decoding information estimated in the estimating step.
A control program for causing a computer to function as the recall image estimation apparatus according to any one of claims 1 to 6, wherein the control program causes the computer to function as the decoder and the image determination unit.
A computer-readable recording medium on which the control program according to claim 8 is recorded.