JP4704952B2 - Mental state determination device and program - Google Patents

Description

  The present invention relates to a mental state determination device and a program.

Conventionally, a technique for estimating the mental state of a subject based on biological information such as a heartbeat is known. For example, Patent Document 1 describes that the higher the heart rate, the higher the degree of excitement of the subject.
In addition, in this patent document, this inventor is disclosing the technique which estimates a test subject's mental state based on a test subject's audio | voice signal.
JP 2002-34936 A Japanese Patent No. 3676969

In general, when psychological stress such as anxiety is applied, the subject's physiological excitement increases and the heart rate increases. Also in a situation where a strong anger is felt, the physiological excitement level of the subject similarly increases and the heart rate increases. For this reason, it is difficult to accurately distinguish mental states such as anxiety / anger by using only biological information such as heartbeats.
Therefore, the present invention provides a technique for accurately distinguishing the mental state of the subject.

The mental state determination device of the present invention includes a biological excitement level determination unit, a voice excitement level determination unit, a correspondence storage unit, and an estimation unit.
The biological excitement level determination unit acquires biological information reflecting the level of physiological excitement and tension, and determines the physiological excitement level of the subject (hereinafter referred to as “biological excitement level”) based on the biological information.
The voice excitement level determination unit determines the voice excitement level of the subject (hereinafter referred to as “voice excitement level”) based on the voice signal of the subject.
The correspondence storage unit holds a correspondence relationship between a predetermined “biological excitement level and voice excitement level” and “mental state”.
An estimation part estimates a test subject's mental state from a test subject's biological excitement degree and audio | voice excitement degree based on a correspondence.
Further , the estimation unit estimates that the subject is in an anxiety state when it is determined that the biological excitement level is high and the voice excitement level is low.
Is or Preferably, the estimation unit is determined to have a high biological excitement level, and if it is determined that the high sound level of excitement, the subject is estimated that the state of anger or extreme anxiety.
It is a contact Preferably, the estimation unit is determined to be low biological excitement level, and if it is determined that the low sound excitement level, estimates that the subject is in a state of depression or fear.
The program of the present invention is a program for causing a computer to function as the mental state determination device of the present invention .

In the present invention described above mentioned, the excitement of the subject is determined from each of the two cut biometric information and the audio signal.
Among these, biometric information is involuntary information that is difficult to control by will, such as blood pressure and heart rate. From this biological information, it is possible to determine the excitement and tension that appear involuntarily in the human body, that is, the degree of biological excitement.
On the other hand, the audio signal is information that changes reflecting the emotion and will of the subject. From this voice signal, it is possible to determine the degree of excitement that reflects the subject's emotions and intentions, that is, the degree of voice excitement.
Normally, it is considered that these two types of biological excitement and voice excitement show the same tendency and strongly show a positive correlation. However, the present inventor has noticed that the degree of biological excitement and the degree of voice excitement show different tendencies due to stress and will.
From this, when the degree of biological excitement and the degree of voice excitement show the same tendency, it can be determined that the subject is in an honest mental state that can express emotions. On the other hand, when the biological excitement level and the voice excitement level show different tendencies, it can be determined that the subject is in an unstable mental state (strong emotion, stress, inspiration, acting, lie, etc.).
The present invention has succeeded in classifying mental states in more detail by comparing and contrasting the degree of biological excitement and the degree of voice excitement by utilizing such a phenomenon.

And while the involuntary human excitement and tension are high, the voice excitement level reflecting emotions and intentions may be low. Such a case is considered to be in a mental state in which the subject has anxiety (anxiety). That is, the sympathetic nerve is activated to counter the life crisis due to anxiety, and the biological excitement level is increased. On the other hand, the degree of voice excitement decreases because the anxiety causes atrophy.
Therefore, when it is determined that the degree of biological excitement is high and the degree of voice excitement is low, it is preferable to estimate that the subject is in an anxiety state.

Also, high involuntary human body of excitement and tension, and is higher voice excitement level that reflects the feelings and the will case may occur. If the biological excitement level and the voice excitement level are in the normal range, this case is considered to be in a mental state in which excitement emotions such as anger are obediently expressed. Therefore, if the two types of excitement are high within a normal range, it is preferable to estimate an angry state.
In this case, when the degree of voice excitement is remarkably high, there is a possibility that the anxiety emotion is so strong that the emotional atrophy is exceeded and the state is shifted to the ataxia (split) state. Therefore, when it is determined that the two types of excitement levels are high and the voice excitement level is higher than a predetermined normal range, it is preferable to estimate the state of extreme anxiety.
Also, if it is not necessary to distinguish between the two possibilities described above, it is preferable to estimate that the subject is in an angry state or an extreme anxiety state.

Name your, involuntary human body of excitement and tension is low and is below the sound level of excitement to reflect the feelings and the will case may occur. Such a case is a state in which the subject is frozen from the bottom of the heart, and is considered to be a state of depression or fear.
Therefore, when it is determined that the degree of biological excitement is low and the degree of voice excitement is low, it is preferable to estimate that the subject is in a state of depression or fear.

<Description of configuration>
FIG. 1 is a block diagram illustrating a configuration of the mental state determination device 11.
As shown in FIG. 1, the mental state determination device 11 includes the following configuration requirements.
(1) Living body excitement level determination unit 12... The level of living body excitement level is determined based on the biological information of the subject.
(2) Speech analysis unit 14... Analyzes a speech signal and detects speech parameters including at least one of inflection and pitch frequency.
(3) Voice excitement level determination unit 15... Determines the level of voice excitement level of the subject based on the voice parameters.
(4) Corresponding storage unit 18... “Mental state estimation result” is preset for each combination of “biological excitement level” and “voice excitement level”. This correspondence relationship is preferably created experimentally by associating self-reports of mental states by a plurality of persons with determination results of biological excitement level and voice excitement level.
(5) Estimating unit 17: The “biological excitement level” and “voice excitement level” of the subject are applied to the correspondence in the correspondence storage unit 18 to estimate the mental state of the subject.

  Note that some or all of the above-described configuration requirements may be configured in hardware. Moreover, you may implement | achieve part or all of these structural requirements like software by running a program in a computer.

<Operation description>
FIG. 2 is a flowchart for explaining the operation of the mental state determination device 11.
The mental state estimation operation will be specifically described below with reference to FIG.

Step S1: The biological excitement level determination unit 12 acquires at least one of the following biological information from a heart rate monitor, an electrocardiograph, a sphygmomanometer, a thermometer, a skin resistance meter, a camera, or the like.

(1) Heart rate (pulse rate) ··· Increased adrenaline secretion in the body due to excitement and tension increases the heart beat and increases the heart rate. It can be measured with a heart rate monitor.

(2) Obtained by frequency analysis of heartbeat variability / electrocardiogram waveform. The low frequency component LF (for example, 0.04 to 0.14 Hz) mainly increases with the activity of the sympathetic nerve, and the high frequency component HF (for example, 0.14 to 0.5 Hz) increases with the activity of the parasympathetic nerve. By comparing the amount of the low-frequency component LF and the high-frequency component HF, it is possible to determine the level of excitement or tension, and the electrocardiogram waveform can be measured by an electrocardiograph. It can be analyzed by an FFT device or a frequency filter.

(3) Blood pressure: When blood vessels contract with excitement or tension, blood flow resistance increases and blood pressure increases. It can be measured with a sphygmomanometer.

(4) Body temperature: Increased heart rate, blood sugar level, muscle tension, etc. occur due to excitement and tension, and heat is generated in the body to increase body temperature. It can be measured with a thermometer.

(5) Sweating volume (skin resistance, skin potential)-Sweat is promoted by excitement and tension, and skin resistance decreases. It can be measured with a skin resistance meter.

(6) Eye movements: The eye movements become intense due to excitement and tension. It can be measured by an electrooculometer or camera photography.

(7) Pupil diameter: The pupil expands due to excitement and tension. It can be measured by camera photography.

(8) Number of blinks ... The number of blinks increases due to excitement and tension. It can be measured by an electrooculometer or camera photography.

(9) Exhalation: The number, speed, and amount of exhalation increase due to excitement and tension. It can be measured by breathing volume and breathing sound.

(10) Endocrine secretions of hormones, biomolecules, etc .. The excretion or tension changes the amount or quality of hormones or biomolecules secreted in the body. It can be measured by chemical analysis such as saliva, blood, lymph, sweat, digestive fluid, or urine. Analysis and measurement are also possible from peripheral blood vessels, digestive system, myoelectric potential, peripheral skin temperature, blood flow, immune system, and the like.

(11) Brain wave: The brain wave changes due to excitement and tension. It can be measured using an electroencephalograph or an optical brain activity meter.

(12) fMRI (functional MRI) information: The active site in the brain changes due to excitement and tension. In particular, emotional excitement and tension appear as changes in blood flow in the limbic system (amygdala), hypothalamus, cerebellum, brain stem, hippocampus, and the like. Such a change in blood flow can be measured using an MRI (magnetic resonance imaging apparatus) or the like in order to change the brain distribution of oxyhemoglobin.

  Not only the above-described biological information but also body measurement information that changes according to physiological actions such as sympathetic nerves and parasympathetic nerves can be used as biological information.

Step S2: The biological excitement level determination unit 12 determines the threshold value of the biological information and determines the level of the biological excitement level.
In addition, since biometric information has a large individual difference, it is preferable to determine threshold values Th1 and Th2 for each subject as follows.
First, biological information is accumulated for a subject over a predetermined time, and a frequency distribution of biological information as shown in FIG. 3 is created. The frequency distribution is statistically analyzed to obtain a standard deviation σ. By multiplying this standard deviation σ by a proportionality constant A, a range A · σ where the degree of biological excitement is normal is determined. Note that the value of the proportionality constant A may be experimentally determined in advance so as to match the self-report of the degree of biological excitement by a plurality of persons.
The upper limit of such a range A · σ is set to the threshold value Th1, and the lower limit is set to the threshold value Th2. When the biometric information exceeds the threshold Th1, it is determined that the bioexcitability is high. Conversely, when the biological information falls below the threshold Th2, it is determined that the degree of biological excitement is low.

Step S3: The voice analysis unit 14 analyzes the voice signal of the subject and obtains a voice parameter including at least an inflection or pitch frequency.
This inflection calculation procedure is disclosed in Japanese Patent No. 3676969 of the present inventor. That is, the region of the same frequency component is detected from the intensity change pattern in the word of the speech signal, and the time interval in which the region of the same frequency component appears is detected as inflection.
On the other hand, the pitch frequency calculation procedure is performed as follows. First, an audio signal is converted into a frequency spectrum. An autocorrelation waveform is obtained while shifting the frequency spectrum on the frequency axis. A pitch frequency is obtained based on the local peak-to-peak or valley-to-valley interval of the obtained autocorrelation waveform.
Note that frequency analysis of an audio signal may be performed by combining known cepstrum techniques.

Step S4: Generally, the voice excitement level of the subject is higher as the “appearance interval of the same frequency component” indicated by the intonation becomes denser. Further, the higher the pitch frequency, the higher the voice excitement level of the subject.
Therefore, as shown in FIG. 4, the voice excitement level determination unit 15 determines whether the voice excitement level is high or low by determining the threshold of the inflection or pitch frequency. The threshold values Th3 and Th4 in this case are preferably set using the same statistical analysis as in step S2.

Step S5: The estimation unit 17 takes in the “biological excitement level” and the “voice excitement level” determined in steps S2 and S4. The estimation unit 17 applies these to the correspondence relationship in the correspondence storage unit 18 to obtain a mental state estimation result.
FIG. 5 is a diagram showing an example of this correspondence. Hereinafter, the mental state estimation operation will be described in detail for each combination shown in this figure.

[Part 1] A case in which the degree of biological excitement is high and the degree of voice excitement is low. In this case, the inner surface of the subject shows an excitement state or a tension state, but on the contrary, the sound that appears on the outer surface is atrophied Show. This distorted contradiction state is a kind of stress state and is considered to be a mental state in which the subject has anxiety (anxiety).

[Part 2] A case in which the degree of biological excitement is high and the degree of voice excitement is judged to be high. In this case, the excitement of the human body and the tension are increasing, and the voice is excited together. It is thought that it is in the mental state to let you. Such a state is often in a relatively primitive emotional state such as anger. In addition, when the voice excitement level is exceptionally high, it can be considered that a kind of ataxia (split) state is caused by extreme anxiety.

[No. 3] Case where it is determined that the degree of biological excitement is low and the degree of voice excitement is low. This case is a state in which the subject is frozen from the bottom of the heart, and is considered to be a state of depression or fear.

[Part 4] Case in which the degree of biological excitement is low and the degree of voice excitement is high. In this case, the excitement of the human body is calm, but the voice is unilaterally excited. . Such a state is a state in which the subject is intentionally excited by voice, and is considered to be a state in which acting (lie) is forcibly performed.

Step S6: The mental state determination device 11 outputs the estimation result of the mental state obtained in Step S5 to the outside.

<< Effects of the embodiment >>
Conventionally, emotion detection based on heart rate cannot distinguish between an excitement state due to an anger emotion and an excitement state due to anxiety emotion.
On the other hand, in the present embodiment, the state of excitement of the subject is determined by distinguishing between a physiological (involuntary) biological excitement determined from blood pressure and heartbeat and a voice excitement that well reflects the subject's emotions and intentions. . As a result, as shown in FIG. 5, it is possible to more accurately determine the mental state such as anxiety / anger.

<< Additional items of embodiment >>
In the above-described embodiment, the biological excitement level and the voice excitement level are divided into two levels, high and low, and the mental states are distinguished into 2 × 2 = 4 types by combining these levels. However, the embodiment is not limited to this.
For example, by setting the normal range of excitement level (see FIGS. 3 and 4) to a new level, 3 × 3 = 9 types of mental states can be distinguished. Further, by dividing the biological excitement level and the voice excitement level into a plurality of levels, it becomes possible to further distinguish the mental state of the subject. Further, by adding parameters other than the degree of excitement, it becomes possible to further distinguish the mental state of the subject.
By subdividing the subject's mental state in this way, an intermediate mental state in which anger and anxiety are mixed, and a new mental state such as frustration and sadness that would naturally shift from there is estimated. It is also possible to do.

  In the above-described embodiment, a specific estimation rule (see FIG. 5) has been described in order to simplify the description. However, the present embodiment is not limited to this estimation rule. For example, this kind of estimation rule can be specifically determined by statistically combining the mental state distinguished by the excitement level and the mental state by self-reporting of a plurality of subjects, for example. Good.

  Further, in the above-described embodiment, an inflection or pitch frequency having a strong correlation with the voice excitement level is used as the voice parameter. However, the embodiment is not limited to this. In addition to the inflection and pitch frequency, the audio parameters may include waveform characteristics of the audio signal (rise time Attack, duration Keep, decay time Decay), volume power, and the like. By increasing the voice parameters in this way, it is possible to determine a more accurate voice excitement level.

  In the above-described embodiment, the thresholds th1 to th4 for determining the level of biological excitement and voice excitement are determined by statistical analysis. However, the embodiment is not limited to this. For example, these threshold values are arbitrarily initialized, and an appropriate threshold value is determined on a trial and error basis by adjusting the threshold value according to the result of the mental state estimation result by the mental state determination device 11. Is also possible.

《Examples of mental state application fields》
(1) In general, the mental state estimation result can be used for all products that change processing in response to the mental state. For example, it is possible to construct a virtual personality (agent, character, etc.) on the computer that changes the response (personality, conversational characteristics, psychological characteristics, sensitivity, emotional pattern, conversational branching pattern, etc.) according to the mental state Is possible. In addition, for example, depending on the customer's mental state, product search, product complaint handling, call center operation, reception system, customer sensitivity analysis, customer management, game, pachinko, pachislot, content distribution, content creation, net search, mobile phone It can also be applied to a system that realizes telephone service, product description, presentation, or educational support.

(2) The mental state estimation result can also be used for all products that improve the processing accuracy by using the mental state as calibration information about the user. For example, in a speech recognition system, it is possible to improve speech recognition accuracy by selecting a vocabulary having a high affinity for the mental state of a speaker from among recognized vocabulary candidates.

(3) Furthermore, the estimation result of the mental state can be used for all products that enhance security by inferring the user's illegal intention from the mental state. For example, in a user authentication system, it is possible to increase security by rejecting authentication or requesting additional authentication for a user who shows a mental state such as anxiety or performance. Furthermore, it is possible to construct a ubiquitous system based on such high security authentication technology.

(4) The mental state estimation result can also be used for all products that handle the mental state as an operation input. For example, it is possible to realize a system that executes processing (control, voice processing, image processing, text processing, or the like) using a mental state as an operation input. Further, for example, it is possible to realize a story creation support system that develops a story by controlling a character action using a mental state as an operation input. Further, for example, it is possible to realize a music creation support system that performs music creation and arrangement according to the mental state by changing the temperament, the key, or the musical instrument configuration using the mental state as an operation input. In addition, for example, it is possible to realize an effect device that controls the surrounding environment such as lighting and BGM using the mental state as an operation input.

(5) Furthermore, the estimation result of the mental state can be used for all apparatuses for the purpose of psychoanalysis, emotion analysis, sensitivity analysis, personality analysis, or psychological analysis.

(6) In addition, the estimation result of the mental state is also applied to all devices that externally output the mental state using expression means such as sound, voice, music, fragrance, color, video, text, vibration, or light. Can be used. By using such a device, it becomes possible to support emotional communication with humans.

(7) Furthermore, the mental state estimation result can be used for all communication systems that communicate information on the mental state. For example, it can be applied to emotional communication or emotional emotion resonance communication.

(8) The mental state estimation result can also be used for all devices that determine (evaluate) the psychological effects of content such as video and music on humans. Furthermore, by classifying content using this psychological effect as an item, it is possible to construct a database system that enables content search from the aspect of psychological effect.
In addition, by analyzing the content itself such as video and music in the same manner as the audio signal, it is also possible to detect the degree of voice excitement and the emotion tendency of the content performer and the musical instrument player. It is also possible to detect the feature of the content by recognizing the voice of the content or recognizing the phoneme. By classifying content according to such detection results, it becomes possible to search for content based on the features of the content.

(9) Furthermore, the mental state estimation result can be used for all devices that objectively determine the user satisfaction level and the like when using a product according to the mental state. By using such an apparatus, product development and specification creation that are familiar to the user are facilitated.

(10) Further, the mental state estimation result can be applied to the following fields.
Nursing care support system, counseling system, car navigation, car control, driver status monitoring, user interface, operation system, robot, avatar, online shopping mall, distance learning system, e-learning, learning system, manner training, know-how learning system, Psychological element input to ability judgment, semantic information judgment, artificial intelligence field, application to neural network (including neurons), judgment criteria and branching criteria for simulations and systems that require a probabilistic model, market simulation such as economy and finance , Questionnaire collection, analysis of artist's feelings and sensibility, financial credit survey, credit management system, content such as fortune telling, wearable computer, ubiquitous network products, support for human perception judgment, advertising business, building Management of halls, filtering, user judgment support, control of kitchens, baths, toilets, etc., human devices, clothing in conjunction with fibers that change softness and breathability, virtual pets and robots for healing and communication purposes , Planning system, coordinator system, traffic support control system, cooking support system, performance support, DJ video effects, karaoke equipment, video control system, personal authentication, design, design simulator, system to stimulate purchasing, personnel management system, audition , Virtual customer group market research, jury / judgment simulation system, sports / art / sales / strategy image training, creation of memorial content for deceased and ancestors, system for storing patterns of emotions and sensibilities before life Services, navigation and concierge services, blog creation support, messenger services, alarm clocks, health appliances, massage appliances, toothbrushes, medical appliances, biological devices, switching technologies, control technologies, hubs, branch systems, capacitor systems, molecular computers, quantum Computer, Neumann computer, bio-element computer, Boltzmann system, AI control, fuzzy control.

<< Remarks: Acquisition of audio signal in fMRI information measurement environment >>
Since the noise level of an MRI gantry reaches a noise level similar to that of a jet aircraft, it is difficult to obtain a sound signal from a subject in the gantry. For this reason, it is impossible to simultaneously measure brain activity and acquire voice signals.
In order to solve this problem, the present inventor has constructed the following measurement environment.
First, the subject's occipital region is applied to a fixture such as polystyrene foam shaped into the shape of the occipital region by thermoplasticity. In this state, the subject's head (above the eyebrows) is fixed to the gantry bed with a nylon belt or the like.
In this state, a soundproof mask having the following structure is attached to the subject's mouth.

First, a gas mask (TOYO SAFETY No1880-1) will be procured as the base material for the sound mask. This gas mask is made of rubber at the portion covering the mouth. Since this rubber vibrates due to ambient noise, the ambient noise enters the mask. Therefore, silicon (Nisshin Resin Co., Ltd., quick silicone, light gray liquid, specific gravity 1.3) is injected into the rubber part to make it heavy. In addition, the gas filter ventilation filter enhances airtightness by stacking five or more kitchen papers and sponges in multiple layers. A small microphone is fitted in the central portion of the mask chamber in this state. The soundproof mask prepared as described above can effectively attenuate the vibration of the surrounding noise by the laminated structure of the weight of the silicon and the foreign material. As a result, a small soundproof room in the form of a mask is successfully provided around the subject's mouth, and the subject's voice can be collected well while suppressing the influence of ambient noise.
Furthermore, by wearing headphones with similar soundproofing measures on the subject's ears, it is possible to have a conversation with the subject without much influence from ambient noise.

  The above-described soundproof mask is effective for detecting the pitch frequency. However, since the sealed space of the soundproof mask is narrow, there is a tendency that the sound tends to be trapped. Therefore, it is not suitable for frequency analysis other than pitch frequency or tone color analysis. For such applications, it is preferable to pass a soundproofing pipeline similar to the mask through the soundproofing mask to vent the outside environment (air chamber) of the soundproofing environment. In this case, since there is no trouble in breathing, not only the mouth but also the nose can be masked. By adding this ventilation equipment, it is possible to reduce the volume of sound in the soundproof mask. Furthermore, since there is little discomfort such as breathlessness for the subject, it is possible to collect more natural sound.

  With the above measurement equipment, the detection of the degree of biological excitement based on the fMRI information and the detection of the degree of voice excitement based on the audio signal can be performed simultaneously for the first time, and the mental state of the subject can be estimated.

  The present invention can be implemented in various other forms without departing from the spirit or main features thereof. For this reason, the above-described embodiments are merely examples in all respects and should not be interpreted in a limited manner. The scope of the present invention is indicated by the scope of claims, and is not restricted by the text of the specification. Further, all modifications and changes belonging to the equivalent scope of the claims are within the scope of the present invention.

  As described above, the present invention is a technique that can be used for a mental state determination device, a program, and the like.

2 is a block diagram illustrating a configuration of a mental state determination device 11. FIG. 3 is a flowchart illustrating the operation of the mental state determination device 11. It is a figure explaining judgment of a living body excitement degree. It is a figure explaining judgment of voice excitement degree. It is a figure which shows an example of a correspondence.

Explanation of symbols

DESCRIPTION OF SYMBOLS 11 ... Mental state determination apparatus, 12 ... Living body excitement degree determination part, 14 ... Speech analysis part, 15 ... Voice excitement degree determination part, 17 ... Estimation part, 18 ... Corresponding memory | storage part

Claims (4)

A biological excitement determination unit that acquires biological information that reflects the level of physiological excitement and tension, and determines the physiological excitement level of the subject (hereinafter referred to as “biological excitement level”) based on the biometric information;
A voice excitement level determination unit that determines a voice excitement level of the subject (hereinafter referred to as “voice excitement level”) based on the voice signal of the subject;
A correspondence storage unit that holds a correspondence relationship between the predetermined “excited degree of biological excitement and voice excitement” and “mental state”;
An estimation unit that estimates a mental state of the subject from the biological excitement level and the voice excitement level of the subject based on the correspondence relationship ;
The estimation unit includes
When it is determined that the biological excitement level is high and the voice excitement level is low, it is estimated that the subject is in an anxiety state.
Mood determination apparatus characterized by and this. The mental state determination device according to claim 1,
The estimation unit includes
Wherein the biological excitement level is determined to be high, and if the audio excitement level is high the Most determined, the subject mentality determination device and estimating the state of anger or extreme anxiety. In the mental state determination device according to claim 1 or 2,
The estimation unit includes
Wherein the biological excitement level is low contemplated determined, and if the audio excitement level is low contemplated determined, the subject mentality determination apparatus characterized by estimating that the state of depression or fear. A program for a computer to function as mood determination apparatus according to any one of claims 1 to 3.

Images