JP2006071936A - Dialogue agent - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a dialogue agent capable of performing anatural dialogue making a user feel neither incompatible nor unpleasant by changing a response to the user according to the ego-state. <P>SOLUTION: A face feeling estimation section 13 estimates a feeling from the expression of the user imaged by a camera 41. From a user's speech inputted from a microphone 42, a speech feeling estimation section 14 estimates a feeling, a tone estimation section 15 estimates a tone, and a text extraction section 16 extracts a text. An ego-state estimation section 20 combines four kinds of pieces of information of the feeling obtained from the expression of the user, the feeling obtained from the speech, the tone, and the text to estimate an ego-state vector for the utterance of the user. A dialogue control section 30 determines an ego-state vector and a text for responding from the ego-state vector estimated by the utterance of the user to respond with a synthesized speech through a speaker 43. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a dialog agent that interacts with a user in response to the user's voice.

Conventionally, various interactive systems that interact with a user in response to the user's voice have been proposed. In this type of dialogue system, which is realized using a computer, it is desired to perform a natural dialogue similar to a dialogue between humans. For example, the user's voice recognition information is divided into static information and dynamic information, and the dynamic information is managed according to the items to be recognized. A technique for enabling this is proposed (for example, see Patent Document 1).
JP-A-6-208389 (paragraphs 0023-0046, FIG. 3)

  The technique described in Patent Document 1 described above is a technique for suppressing a sense of incongruity due to a delay in response by shortening a response time in dialogue, and the same response is obtained if the user's utterance content is the same. Only a uniform response is possible. For example, regardless of whether the user is an adult or a child, there is a problem that only a dialogue that the machine is responding can be performed.

  The present invention has been made in view of the above-described reasons, and its purpose is to recognize a ego state at the time of dialogue with a user, thereby enabling a dialogue in an ego state according to the situation, and use. It is an object to provide a dialog agent that makes it possible for a user to perform a natural conversation that is easy to accept and does not feel uncomfortable by appropriately changing the response to the user according to the ego state.

  According to the first aspect of the present invention, there is provided voice input means for inputting a user's voice, dialog processing means for generating text responding to the contents of the voice input from the voice input means, and text generated by the dialog processing means. And a text output means for outputting to the user, and the dialogue processing means classifies the user's emotions into a plurality of types using the prosodic features of the voice input from the voice input means, as voice emotion data Voice emotion estimation unit for output, tone estimation unit for classifying user's tone into multiple types using the prosodic features of speech input from speech input means, and output as tone data, input from speech input means A text extraction unit that extracts a sound string from speech and outputs it as text data, an image input unit that captures the user's face, and each part of the user's face that is captured by the image input unit A facial expression estimator that classifies facial expressions based on changes in position over time, and facial expressions extracted by the facial expression estimator are used to input multiple types of user emotions using facial expression change patterns with time. Emotion summary data, voice emotion data, and tone as the ego state vector, including the orientation from the speaker to the listener, the combination of the ego state, which is a model of the emotions of the face that classifies and outputs as emotion summary data. An ego state estimation unit that estimates an ego state vector based on a user's utterance from a set of data and text data, and responds to the user according to a pre-set correspondence rule from the ego state vector estimated by the ego state estimation unit A dialogue control unit that determines the ego state vector and automatically determines the text to respond to the user from the contents of the text data And wherein the Rukoto.

  According to this configuration, by using four types of information, emotion obtained from the user's facial expression and emotion, tone, and text obtained from the user's voice, the user's ego state and the dialogue agent stimulated The user's combination with the ego state is estimated as the ego state vector along with the direction of the stimulus, and the ego state vector when responding to the user is determined using this ego state vector and the response text is determined. The response to the user is changed in accordance with the ego state vector estimated by the utterance of the utterance, so that it is easy for the user to accept and natural conversation without causing discomfort and discomfort is possible.

  In the invention of claim 2, in the invention of claim 1, the ego state estimator indicates the likelihood ego for each ego state vector candidate estimated from a combination of the emotion summary data and the voice emotion data. An emotion score assigning unit for obtaining a state score, a tone score assigning unit for obtaining a tone ego state score indicating likelihood for each candidate of ego state vector estimated from a combination of the voice emotion data and the tone data, and the text The ego obtained by the text score assigning unit that obtains the text ego state score indicating the likelihood for each candidate of the ego state vector estimated from the content of the data, the emotion score assigning unit, the tone score assigning unit, and the text score assigning unit Classify the ego state included in the candidate state vector for each person who interacts, and for each ego state of each person, the candidate's emotional ego A weighted sum obtained by multiplying each of the state score, the tone ego state score, and the text ego state score by multiplying each weight coefficient is obtained as an integrated score that is an evaluation value of the likelihood, and each person included in the ego state vector candidate is obtained. It is characterized by comprising a score integration calculation unit that estimates the ego state having the maximum likelihood among the integrated scores for each ego state as the ego state of each person in the ego state vector based on the user's utterance.

  According to this configuration, by estimating the ego state vector by the emotion obtained from the facial expression and voice, the tone obtained from the voice, and the text obtained from the voice, respectively, and by integrating the likelihood of each ego state vector, Since an appropriate ego state vector is obtained, the accuracy of determining the ego state vector for the user's utterance is increased. The term “likelihood” is used in the sense of plausibility.

  In the invention of claim 3, in the invention of claim 2, the emotion ego state score gives a perfect score to the candidate of ego state vector when the emotions indicated by the emotion summary data and the voice emotion data match. , A numerical value that is distributed to the same point for candidates of the ego state vector obtained when emotions do not match, and the tone ego state score is an ego state when there is no contradiction between the voice emotion data and the tone data This is a numerical value that gives a full score to a vector candidate and is distributed to the ego state vector candidates when there is a contradiction, and the text ego state score includes a specific incidental phrase in the text data A numerical value in which a large value is assigned in descending order of likelihood with respect to the ego state vector candidate corresponding to the supplementary phrase, and corresponds to one supplementary phrase The sum of the text ego state score was devoted to the candidates of our state vector is characterized to be a perfect score.

  According to this configuration, the emotional ego state score, the tone ego state score, and the text ego state score can be set relatively easily and appropriately.

  In the invention of claim 4, in the invention of claim 2 or claim 3, the weighting coefficient for obtaining the integrated score is the sum of the weighting coefficient for the emotional ego state score and the weighting coefficient for the tone tone ego state score. The weight coefficient for the text ego state score is larger for the stimulated ego state than the weight coefficient for the user ego state.

  According to this configuration, in determining the ego state vector for the user's utterance, the non-linguistic information is more important than the linguistic information, and the emotion ego state score and the tone ego state score are superior to the text ego state score. Therefore, the emotional ego state score and the tone ego state score are mainly used and the text ego state score is used as an auxiliary to estimate the ego state, and the ego state for the user's utterance is appropriately determined. Can do. For language information, it is assumed that the listener's ego state is intended rather than the speaker's ego state, and the text ego state score weighting factor is set as described above. is doing.

  According to a fifth aspect of the present invention, in the first to fourth aspects of the invention, the dialogue control unit has a function of generating a prosodic parameter of speech from the determined response ego state vector and the determined response text. The text output means includes: a speech synthesis processing unit that generates a synthesized speech in which prosodic parameters are applied to the response text determined by the dialogue control unit; and a voice that outputs the synthesized speech generated by the speech synthesis processing unit Output means.

  According to this configuration, it is possible to respond to the user by voice, and it is possible for the user to interact while performing other work. In addition, it is possible to interact with visually impaired people.

  According to a sixth aspect of the present invention, in the first to fifth aspects of the present invention, the facial expression estimation unit includes “no expression”, “surprise”, “fear”, “disgust”, “anger”, “happiness”, “ It is characterized by classifying seven kinds of expressions of “sadness”.

  According to this configuration, seven types of facial expressions classified by the facial expression estimation unit are used: “no facial expression”, “surprise”, “fear”, “disgust”, “anger”, “happiness”, and “sadness”. If these seven types of facial expressions are used, the association with the ego state is relatively easy.

  According to a seventh aspect of the present invention, in the first to sixth aspects of the present invention, the voice input means includes a user database in which the voice characteristics of the user and the facial characteristics of the user are registered in association with the user. A user recognizing unit for identifying a user by comparing a feature of a user's voice input from the user and a feature of a user's face image captured by the image input unit with a user database, and the dialog control unit The user attribute specified by the user recognition unit is registered in advance, and the ego state vector estimated by the ego state estimation unit when determining the ego state vector and text when responding to the user and In addition to the text data, user attributes are also used.

  According to this configuration, since the ego state is determined for a specific number of users registered in the user database in advance, it is possible to use known information about the user when determining the ego state vector at the time of response. Compared with the case where the ego state is determined for an unspecified number of users, the possibility of a response that does not cause discomfort or discomfort can be increased. It is also possible to prohibit a dialog with an unauthorized user by specifying the user with whom the dialog agent interacts.

  In the invention of claim 8, in the invention of claim 7, the ego state history storage unit that stores and stores the ego state vector estimated by the ego state estimation unit in association with the user specified by the user recognition unit; The ego state feature extraction unit that estimates the user's personality from the distribution pattern of the appearance frequency of the user's ego state vector stored in the ego state history storage unit, and the personality estimated by the ego state feature extraction unit to the user An ego state feature storage unit is stored in association with each other, and the dialog control unit determines an ego state vector and text when responding to the user using the user's personality stored in the ego state feature storage unit. It is characterized by doing.

  According to this configuration, since the user's personality can be estimated from the history of the appearance frequency of the user's ego state, the user's personality can be estimated without performing a counselor diagnosis or self-diagnosis test. Become. Since the personality estimation result is stored in the ego state feature storage unit and the user's personality estimation result is used to determine the ego state vector and text when responding to the user, a smooth response according to the user can be obtained. It becomes possible. Since the user's ego state history and personality estimation results are stored, the counselor can also use the user's counseling.

  According to a ninth aspect of the present invention, in the first to eighth aspects of the invention, in the correspondence rule of the dialogue control unit, the ego state stimulated by the user's utterance is set as the ego state at the time of response, and the use is stimulated at the time of response. The user's ego state is the ego state at the time of the user's previous utterance.

  According to this configuration, since the ego state vector when the user utters and the ego state vector when the dialogue agent responds, the user can have a smooth dialogue without causing discomfort or discomfort to the user. . In addition, if the ego state vector when the user speaks is determined, the ego state vector when the dialog agent responds can be uniquely determined, so that the correspondence rule becomes simple.

  According to a tenth aspect of the present invention, in the first to ninth aspects of the present invention, the body model expression unit that performs expression accompanied by a body motion, and the ego state vector and the text determined by the dialogue control unit are stored in the body model expression unit. A body expression control unit for converting into body motion is added.

  According to this configuration, since the text and the ego state vector determined by the dialog control unit are reflected in the body movement of the body model expression unit, the user can respond by giving gestures or imitations to the user during the dialog. Message transmission to users.

  In the invention of claim 11, in the inventions of claims 1 to 10, the ego state is a model of a mind based on an alternating current analysis, which is “critical parent”, “protective parent”, “adult”, It is classified into five types: “free children” and “adapting children”.

  According to this configuration, as the ego state, the models of the mind based on the exchange analysis, “critical parents”, “protective parents”, “adults”, “free children”, “adapted children” are used. Thus, the ego state vector at the time of response can be set relatively easily according to the AC analysis.

  According to the configuration of the present invention, the user's ego state is stimulated by using four types of information, emotion obtained from the user's facial expression and emotion, tone, and text obtained from the user's voice. Estimate the ego state vector, which is a combination with the ego state, and determine the ego state vector when responding to the user using this ego state vector and the response text. The response to the user is changed in accordance with the ego state vector to be performed, and there is an advantage that a natural dialogue that is easy for the user to accept and does not cause discomfort or discomfort is possible.

(basic action)
The dialogue agent described below is configured using a computer, and generates a response voice by focusing on the ego state in order to realize a natural dialogue with the user. The ego state is a model of the mind based on the exchange analysis (for example, Mineyasu Sugita: “exchange analysis”, Nihon Bunka Sakusha, 1985). Furthermore, parents are classified as critical parents (CP) and protective parents (NP), and children are classified as free children (FC) and adapting children (AC). That is, the ego state is classified into five types. In the following, the CP, NP, A, FC, and AC codes are used when describing the ego state. In the embodiment described below, a “user” is assumed to be a human and is called a “user”, and in principle, the user speaks first.

  In the exchange analysis, a smooth dialogue is established when the ego states of the two parties having the dialogue are in an appropriate relationship. In other words, in the relationship between the speaker and the responder when talking, if the speaker speaks in a certain ego state, the responder's ego state is stimulated by the speaker's speech, and the speaker's If the relationship between the ego state and the responder's stimulated ego state is appropriate, the responder's response will not cause discomfort or discomfort to the speaker, and the content of the response can be transmitted efficiently. It becomes possible. On the other hand, when the relationship between the ego state of the speaker and the stimulated ego state of the responder is not appropriate, the responder's response gives the speaker a sense of incongruity and discomfort, and gives a familiar and unfamiliar impression. . That is, it is known that in order to establish a smooth dialogue, it is necessary to make the relationship between the ego state of the speaker and the responder appropriate.

  In order to consider the ego state during the dialogue, the dialogue agent basically operates according to the procedure shown in FIG. First, when the user's voice is input from the microphone (S1), the user's voice and the user's facial expression are acquired using the microphone and the TV camera (S2), and stimulated by the user's ego state and the user. The combination with the ego state of the dialogue agent is estimated (S3). Here, the combination of the ego state of the user and the dialogue agent is called the “ego state vector” including the direction from the speaker to the listener, and if the user stimulates the ego state of the dialogue agent, it is stimulated. If the dialogue agent stimulates the user's ego state, it will be called the ego state vector of the stimulus. Once the ego state vector of the stimulus is estimated, after determining the ego state vector of the stimulus that is a combination of the ego state of the dialog agent and the user's ego state stimulated by the dialog agent (S4), A response text suitable for the ego state vector is generated (S5), and the text is output to the user (S6). Further, in the present invention, when estimating the ego state vector of the stimulus, a configuration is adopted in which the combination of information acquired from the user is replaced with a score that is an evaluation value, and the ego state vector of the stimulus is determined using the score. ing. An integer value of 0 to 100 is used for the score.

  The ego state vector describes the user's ego state and the dialogue agent's ego state in parentheses side by side so that the left side is the user's ego state. The two are coupled with a right-pointing arrow so that they are directed, and in the case of stimulation, the two are coupled with a left-pointing arrow. For example, if the user's ego state is CP and the dialogue agent's ego state is AC, it is described as (CP → AC), and if it is a stimulus, it is described as (CP ← AC). For the score corresponding to the ego state, the user's score and the dialogue agent's score are written side by side in parentheses so that the left side is the user's score, and the two are separated by commas. For example, if the score of the user is 50 and the score of the dialog agent is 50, (50, 50) is described. When the ego state vector of the stimulus is (CP → AC) and the score of the ego state vector is (50, 50), (CP → AC) = (50, 50) is described.

(Embodiment 1)
The configuration of the dialog agent of this embodiment is shown in FIG. The dialogue agent includes a camera 41 including a TV camera as an image input unit for acquiring the facial expression of the user, and a microphone 42 as an audio input unit for acquiring the user's voice. The field of view of the camera 41 is set so as to capture the vicinity of the user's face. The dialogue agent outputs text in response to the user's utterance. In the present embodiment, an example is shown in which text is output by voice and characters. Accordingly, as text output means for outputting text, there are a speaker 43 which is voice output means for outputting synthesized speech generated by the voice synthesis processing unit 17 according to the text, and an image output means for displaying the text on the screen. And a display 44 such as a CRT or a liquid crystal display. The camera 41, the microphone 42, the speaker 43, and the display 44 are connected to the dialogue processing means 10 realized by a computer that executes an appropriate program, and the dialogue processing means 10 is input from the microphone 42 by the processing described below. Generate text that responds to the voice of the user.

  The dialogue processing means 10 basically estimates a ego state vector of the stimulus from the image captured by the camera 41 and the sound input from the microphone, and the ego of the stimulus from the ego state vector of the stimulus. A response text is generated from the means for determining the state vector, the text included in the voice input from the microphone 42, and the ego state vector of the stimulus, and the response text is output through at least one of the speaker 43 and the display 44. Means for presenting text to the user.

  In this embodiment, the means for estimating the ego state vector of the stimulus is the expression estimation unit 11, the expression database 12, the face emotion estimation unit 13, the voice emotion estimation unit 14, the tone estimation unit 15, the text extraction unit 16, the ego state. The estimation unit 20 is configured by the dialogue control unit 30 and the voice synthesis processing unit 17 to determine the ego state vector of the stimulus and to present the response text to the user.

  The image captured by the camera 41 is a moving image, and the moving image acquired by the camera 41 is input to the facial expression estimation unit 11. The facial expression estimation unit 11 sets feature points for each part such as the eyebrows, eyes, and mouth of the user's face image captured by the camera 41 (for example, the image is captured at 30 frames / second). Classify facial expressions based on changes in position over time. As described in Ekman: “Introduction to facial expression analysis”, the types of facial expressions are “no expression”, “surprise”, “fear”, “disgust”, “anger”, “happiness”, “sadness”. 7 types are used.

  The facial expression estimation unit 11 extracts a temporal change pattern of each part of the face and collates the extracted temporal change pattern with the facial expression database 12. The facial expression database 12 stores temporal change patterns of feature points in association with the seven types of facial expressions described above, and the facial expression estimation unit 11 collates the temporal change patterns of each part of the face with the facial expression database 12. Thus, the facial expressions are classified from the facial expression database 12 by the pattern matching technique.

  Note that the image handled by the facial expression estimation unit 11 is a digital image. Although the digital image can be output from the camera 41, the facial expression estimation unit 11 may perform analog-digital conversion on the analog video signal output from the camera 41.

  The facial expression extracted by the facial expression estimation unit 11 is input to the facial emotion estimation unit 13. The face emotion estimation unit 13 estimates a user's emotion using a facial expression change pattern for each frame over time. When estimating emotions from facial expressions, it detects the start and end points of the user's utterance, evaluates the consistency of facial expressions in the interval from start to end, and estimates in both intervals If there is no obvious contradiction in the sentiment, adopt the result. When classifying the user's emotion based on the face image, the types of emotion output by the face emotion estimation unit 13 are “seduce or anger”, “joy”, “estimation” so that classification in real time is possible. It is limited to three types of “impossible”.

  On the other hand, the voice signal output from the microphone 42 is input to the voice emotion estimation unit 14, the tone estimation unit 15, and the text extraction unit 16. The voice emotion estimation unit 14, the tone estimation unit 15, and the text extraction unit 16 perform segmentation of a voice signal and extract a frequency component as a feature amount using a technique such as FFT. Since the process for extracting the feature amount is the same for the voice emotion estimation unit 14, the tone estimation unit 15, and the text extraction unit 16, a processing unit for performing the process for extracting the feature amount is provided, and the voice emotion estimation unit is provided. 14, the tone estimation unit 15, and the text extraction unit 16 may be shared, and the voice emotion estimation unit 14, the tone estimation unit 15, and the text extraction unit 16 may omit the process of extracting the feature amount.

  The voice emotion estimation unit 14 classifies emotions based on the prosodic features of the speech uttered by the user, and uses a Gaussian mixture model (GMM) for classification of the prosodic features. There are three types of emotions to be classified: “anger”, “joy” and “calm”.

  The tone estimation unit 15 classifies the tone of the voice spoken by the user using a Gaussian mixture model. The tone for the five types of ego states is shown in the above-mentioned document as 16 tone for each ego state. In this embodiment, as shown in Table 1, four tone values are associated with each ego state. Are classified into 20 types of tone. The number of mixed Gaussian distributions in the GMM used for classifying the tone is 64. Table 2 shows the feature extraction conditions from the audio signal. The learning method of the voice emotion estimation unit 14 and the tone estimation unit 15 will be described later.

  The text extraction unit 16 extracts text (sound string) from the user's voice. A known technique using a GMM or a Hidden Markov Model (HMM) can be employed for speech recognition in the text extraction unit 16.

  As described above, the face emotion estimation unit 13 extracts three types of emotions from the user image captured by the camera 41. Further, the voice emotion estimation unit 14 extracts three types of emotions “anger”, “joy”, and “calmness” from the user's voice, and the tone estimation unit 15 extracts the user's voice from the 20 types of tone shown in Table 1. The text extraction unit 16 extracts text from the user's voice. In the following, the facial emotion estimation unit 13 outputs the facial expression summary data d1, the voice emotion estimation unit 14 outputs the voice emotion data d2, the tone estimation unit 15 outputs the tone data d3, and the text extraction unit 16 outputs the text data d4. I will call it. The facial expression summary data d1, the voice emotion data d2, the tone data d3, and the text data d4 are input to the ego state estimation unit 20 having a function of estimating the ego state vector of the stimulus.

  The ego state estimation unit 20 converts the combination of the facial expression summary data d1 and the voice emotion data d2 with the emotion ego state score S1, and the combination of the voice emotion data d2 and the tone data d3 with the tone ego state score. The tone score assigning unit 22 to be replaced with Sb and the text score assigning unit 23 for obtaining the text ego state score Sc from the contents of the text data d4 are provided. The emotion ego state score Sa, the tone ego state score Sb, and the text ego state score Sc are obtained by assigning 0 to 100 points to the respective ego states of the user and the dialogue agent for the stimulated ego state vector. The closer to 100 points, the higher the possibility of being in the ego state.

  By the way, since there are five ego states, there are 25 possible ego state vectors. However, it is assumed that the user makes a request or question to the dialog agent and responds so that the user does not feel uncomfortable or uncomfortable, like a dialog agent that presents a recipe for cooking. Then, there is no NP or AC as the ego state when the user speaks, and there is no CP as the ego state when the dialog agent responds. That is, three types of CP, A, and FC can be selected as the user's ego state, and four types of NP, A, FC, and AC can be selected as the ego state of the conversation agent. Further, consider an ego state vector in which a complementary AC pattern (an AC pattern in which both of the interacting users have the same ego state) is established, and an ego state vector in which an AC pattern that is likely to occur as described in the above-mentioned literature is established. Then, there are four types of stimulated ego state vectors: (A → A), (CP → AC), (FC → FC), and (FC → NP).

  In order for a user to interact with a dialog agent, the dialog agent must first estimate whether the user's ego state is CP, A, or FC. On the other hand, when the ego state is FC, there are characteristics such as “instinctive”, “free expression of emotion”, “excitement”, and when the ego state is CP, “high pressure”, “evacuation”, “reprimand”, “anger” It is known to have a feature such as “Poi Stubborn Father”. Further, the emotion ego state score Sa and the tone ego state score Sb include the user's emotion obtained from the voice emotion data d2, and the “anger” which is the three kinds of emotions of the voice emotion data d2 for the FC ego state. From the viewpoint of whether or not it has the characteristics of “joy” and “calmness”, the ego state of FC has characteristics similar to “joy” or “joy”, and cases of having characteristics of “anger”. There are two types of cases.

  Therefore, when the emotional ego state score Sa and the tone ego state score Sb are obtained, the FC is considered to be divided into an FCa having a “joy” feature and an FCb having an “anger” feature. In other words, five types of (A → A), (CP → AC), (FCa → FCa), (FCa → NP), and (FCb → FCb) are considered as stimulated ego state vectors. Further, if the user's emotion obtained from the voice emotion data d2 is “peaceful”, the ego state is A, if it is “joy”, the ego state is FCa, and if it is “anger”, the ego state is CP or FCb. .

  The emotion ego state score Sa for the combination of the facial expression summary data d1 and the voice emotion data d2 includes three types of expression summary data d1 of “seduce or anger”, “joy” and “cannot be estimated”, and voice emotion data d2 of “anger”. Since there are three types of “joy” and “seduce”, there are nine types of combinations as shown in Table 3. That is, the ego state vector is associated with nine types of combinations of the facial expression summary data d1 and the voice emotion data d2, and the emotion ego state score Sa is defined for each ego state vector. In Table 3, when the emotions indicated by the facial expression summary data d1 and the voice emotion data d2 match, the emotion ego state score Sa is set to a perfect score (100 in this embodiment), and the facial expression summary data d1 and the voice emotion data d2 If the emotions indicated by are different from each other, the emotional ego state score Sa is set in the order of the possibility of the ego state vector according to the contents of the facial expression summary data d1 and the voice emotion data d2.

  When a plurality of types of ego state vectors can be estimated for one combination of the expression summary data d1 and the voice emotion data d2, the emotion ego state score Sa in one combination of the expression summary data d1 and the voice emotion data d2. The emotion ego state score Sa is assigned so that the sum of When it is considered that none of the above-described five kinds of ego state vectors is applicable, the ego state vector is set to “unknown”, and the emotion ego state score Sa is defined in the unknown ego state vector. That is, when the facial expression summary data d1 is “cannot be estimated”, the case where the ego state vector is “unknown” regardless of the content of the voice emotion data d2 is defined, and the emotion ego state score Sa is (20, 20). ) And the remaining (80, 80) are allocated by other possible ego state vectors. In addition, when the facial expression summary data d1 is “joy” and the voice emotion data d2 is “anger”, the emotions are contradictory, so that the emotional ego state score Sa is (100, 100).

  In the present embodiment, the tone data d3 used when obtaining the tone ego state score Sb is obtained by associating four types of tone with respect to the five types of ego states. Although it is possible, as described above, since the user's ego state at the time of utterance is limited to three types of A, CP, and FC (FCa, FCb), the types of tone are as shown in Table 4. There are 12 types. Further, in the present embodiment, as to the tone for obtaining the tone ego state score Sb, as shown in Table 4, the ego state corresponds to CP, the case corresponding to A, the case corresponding to FC1, FC3, These are classified into five types, one corresponding to FC2 and one corresponding to FC4. Therefore, the tone ego state score Sb for the combination of the voice emotion data d2 and the tone data d3 is 15 types of combinations as shown in Table 4. Here, the combinations of alphabets and numbers in Table 4 are combinations of vertical alphabets and rows in Table 1, and the meshes where the alphabets and numbers intersect are the type of tone. Further, N in Table 4 indicates “no tone”. Ego state vectors are associated with 15 types of combinations of voice emotion data d2 and tone data d3, and an emotional ego state score Sa is defined for each ego state vector. In the estimation of the ego state vector corresponding to the tone ego state score Sb, if it is considered that none of the five kinds of ego state vectors corresponds, the ego state vector is set to “unknown” and an appropriate tone ego state score is set. Sb is given.

  In the present embodiment, it is assumed that the user makes a request or question to the dialog agent. Therefore, in order to obtain the text ego state score Sc, a phrase representing the request or question is used as an additional phrase from the text data d4. Extract, associate an ego state vector with an accompanying phrase, and define a text ego state score Sc for each ego state vector. As supplementary phrases extracted from the text data d4, 14 types are assumed as shown in Table 5. (FCb → FCb) is also conceivable as an ego state vector obtained by an auxiliary phrase extracted from the text data d4, but FCb is not included as an ego state in the example of Table 5.

  Tables 3 to 5 are registered in the emotion score assigning unit 21 and the tone score assigning unit 22 and the text score assigning unit 23, respectively. Tables 3 to 5 are determined on the basis of the experimental results, but are appropriately determined depending on the configuration for extracting the facial expression summary data d1, the voice emotion data d2, the tone data d3, and the text data d4, the purpose of use of the dialogue agent, and the like. Be changed.

The ego state estimation unit 20 uses the emotion ego state score Sa obtained by the emotion score assigning unit 21, the tone ego state score Sb obtained by the tone score assigning unit 22, and the text ego state score Sc obtained by the text score assigning unit 23. A score integration calculation unit 24 for obtaining a weighted sum obtained by multiplying and adding the respective weighting factors is provided. That is, the score integration calculation unit 24 calculates the integrated score SI by the calculation of the following equation.
SI = w1, Sa + w2, Sb + w3, Sc
However, w1, w2, and w3 are weighting factors. As shown in Tables 3 to 5, since the emotion ego state score Sa, the tone ego state score Sb, and the text ego state score Sc are scored for both the user and the dialogue agent, the integrated score SI Asks for both users and dialog agents. That is, the emotion ego state score Sa, the tone ego state score Sb, and the text ego state score Sc are all a set of a user score and a dialogue agent score. The calculation of the expression is performed individually for the user score and the interactive agent score, and the calculation result is used as the user score and the interactive agent score in the integrated score SI. The integrated score SI is an evaluation value of the stimulated ego state vector, and the stimulated ego state vector can be estimated by the integrated score SI.

In Tables 3 to 5, the emotional ego state score Sa, the tone ego state score Sb, and the text ego state score Sc have the same values for the user score and the dialogue agent score for any ego state vector. By setting the following conditions for the weighting factors w1, w2, and w3, the integrated score SI may have different values for the user and the interactive agent. Here, the integrated score SI and the weighting factors w1, w2, and w3 are described by adding (u) to the user, and the dialog agent is described by adding (a). That is, the integrated score SI can be expressed as (SI (u), SI (a)). Further, the conditions of the weighting factors w1 (u), w2 (u), w3 (u), w1 (a), w2 (a), and w3 (a) are to satisfy the following three conditions.
w1 (u) + w2 (u)> w3 (u)
w1 (a) + w2 (a)> w3 (a)
w3 (u) <w3 (a)
As weighting factors w1 (u), w2 (u), w3 (u), w1 (a), w2 (a), w3 (a) satisfying the above conditions, for example, w1 (u) = w2 (u) = w2 (A) = w3 (a) = 0.4 and w3 (u) = w1 (a) = 0.2 can be set. That is, the following relationship is established.
w1 (u) + w2 (u) = 0.8> 0.2 = w3 (u)
w1 (a) + w2 (a) = 0.6> 0.4 = w3 (a)
In the above example, the reason why w3 (u) is relatively small is that the incidental phrase extracted from the text data d4 reflects the ego state as compared to the emotion, although it reflects the user's ego state. This is because w1 (a) is relatively small in the dialogue agent, and the ego state stimulated by the user in the dialogue agent is largely reflected in the tone data d3 and the text data d4. Because.

  By using data as shown in Tables 3 to 5, one or more types of ego state vectors are obtained for one combination of facial expression summary data d1, voice emotion data d2, tone data d3, and text data d4. Extracted. Further, for one combination of facial expression summary data d1, voice emotion data d2, tone data d3, and text data d4, at least one of emotion ego state score Sa, tone ego state score Sb, and text ego state score Sc is provided. There may be a plurality of rules. As described above, when a plurality of types of ego state vectors are obtained, the ego states included in the ego state vector are classified for the user and the dialogue agent, and the weight coefficients w1 (u) and w2 are classified for each classified ego state. The weighted sum obtained by multiplying (u), w3 (u), w1 (a), w2 (a), and w3 (a) is obtained, and the obtained weighted sum is integrated for each ego state in the user and the dialogue agent. The score is SI. Since the integrated score SI is obtained for each type of ego state for each of the user's ego state and the ego state stimulated by the dialogue agent, the maximum of the integrated score SI (u) obtained for each ego state of the user. The ego state where the value is obtained is estimated as the user's ego state, and the ego state where the maximum value is obtained from the integrated scores SI (a) obtained for each ego state of the dialogue agent is estimated as the ego state of the dialogue agent.

  Below, an example of the procedure which estimates the ego state vector of the stimulus by a user's utterance is shown. Here, it is assumed that the facial expression summary data d1 is “peaceful”, the voice emotion data d2 is “peaceful”, the tone data d3 is “no tone”, and the incidental phrase of the text data d4 is “to”. According to Table 3, in a combination in which the facial expression summary data d1 is “serious” and the voice emotion data d2 is “serious”, (A → A) = (100, 100). Further, according to Table 4, in a combination in which the tone data d3 is “no tone” and the voice emotion data d2 is “calm”, (A → A) = (100, 100). Further, when the incidental phrase extracted from the text data d4 according to Table 5 is “to”, a plurality of types of ego state vectors are obtained, and (A → A) = (50, 50), (CP → AC) = (30, 30), (FCa → FCa) = (10, 10), (FCa → NP) = (10, 10).

By the way, there are five types of ego states, CP, NP, A, FC, and AC. Further, in this embodiment, since FC is divided into FCa and FCb, there are a total of six types of ego states. The weighting factors are respectively w1 (u) = 0.4, w2 (u) = 0.4, w3 (u) = 0.2, w1 (a) = 0.2, w2 (a) = 0.4. , W3 (a) = 0.4, and in order to distinguish the ego state, each integrated score SI (u), SI (a), emotion ego state score Sa, tone ego state score Sb, text ego state score Sc, respectively If <X> is added and X is an ego state, the integrated scores SI (u) <X>, SI (a) <X> for each ego state of the user and the dialogue agent are as follows: .
SI (u) <X> = 0.4 × Sa <X> + 0.4 × Sb <X> + 0.2 × Sc <X>
SI (a) <X> = 0.2 × Sa <X> + 0.4 × Sb <X> + 0.4 × Sc <X>
In the above example, for the user, Sa <A> = 100, Sb <A> = 100, Sc <A> = 50, Sc <CP> = 30, Sc <FCa> = 10 + 10 (ego state is FCa For the dialogue agent, Sa <A> = 100, Sb <A> = 100, Sc <A> = 50, Sc <AC> = 30, Sc <FCa> = 10, Sc <NP> = 10, and other values of the emotion ego state score Sa, the tone ego state score Sb, and the text ego state score Sc are 0.

When these values are used to categorize the user and the dialogue agent and obtain the total scores SI (u) and SI (a) for each of the six types of ego states, they are as follows.
SI (u) <CP> = 0.4 × 0 + 0.4 × 0 + 0.2 × 30 = 6
SI (u) <NP> = 0
SI (u) <A> = 0.4 × 100 + 0.4 × 100 + 0.2 × 50 = 90
SI (u) <FCa> = 0.4 × 0 + 0.4 × 0 + 0.2 × (10 + 10) = 4
SI (u) <AC> = 0
SI (u) <FCb> = 0
SI (a) <CP> = 0
SI (a) <NP> = 0.2 × 0 + 0.4 × 0 + 0.4 × 10 = 4
SI (a) <A> = 0.2 × 100 + 0.4 × 100 + 0.4 × 50 = 80
SI (a) <FCa> = 0.2 × 0 + 0.4 × 0 + 0.4 × 10 = 4
SI (a) <AC> = 0.2 × 0 + 0.4 × 0 + 0.4 × 30 = 12
SI (a) <FCb> = 0
When the maximum values of the integrated scores SI (u) and SI (a) are obtained for the user and the dialogue agent, respectively, SI (u) <A> = 90 and SI (a) <A> = 80. Assume that the user's ego state is A, the dialogue agent's ego state is A, and the stimulated ego state vector is (A → A) = (90, 80).

  When the ego state vector is determined by estimating the ego state of the user and the ego state of the stimulated dialogue agent in the ego state estimation unit 20, the ego state vector is given to the dialogue control unit 30 together with the text data d4. The dialogue control unit 30 determines the ego state vector of the stimulus from the ego state vector of the stimulus estimated by the ego state estimation unit 20. The ego state vector of the stimulation when responding to the user from the dialogue agent is determined by using a correspondence rule set in advance in association with the ego state vector of the stimulus in the response strategy determination unit 31. The dialog control unit 30 is also provided with a response text determination unit 32 that determines text to be responded to the user using the ego state vector determined by the response strategy determination unit 31 and the text data d4.

  As shown in Table 6, the dialogue control unit 30 includes a scenario database 33 in which correspondence rules between the stimulated ego state vector and the stimulated ego state vector are collected and registered. When the ego state vector is given from the ego state estimation unit 20, the ego state vector of the stimulus is extracted using the correspondence relationship of Table 6 stored in the scenario database 33. In Table 6, arrows connecting the ego state vectors indicate a causal relationship. That is, when the parenthesis (ego state vector of the stimulus) on the left side of the arrow is given, it means that the (ego state vector of the stimulus) on the right side of the arrow is used.

  In the dialogue agent that proposes a recipe for cooking, since there is no NP or AC as the user's ego state, No. 6 and No. 7 in Table 6 are not used. In addition, since there is a case where the ego state vector is “unknown”, No8 is used when it is not applicable to No1 to No5. No. 3 corresponds to a case where the user “pampers” and No. 5 corresponds to a case where the user “gets angry”.

  The response text determination unit 32 automatically determines the text for response by the dialogue agent using the ego state vector of the stimulus obtained by the response strategy determination unit 31 and the keyword included in the text data d4. The text that the dialog agent responds to the user has a response keyword determined so as to correspond to the keyword in the text data d4 generated by the user's utterance, and the text depends on the ego state between the dialog agent and the user. By adding the determined incidental phrase to the response keyword and assembling the text, the text to which the interactive agent responds is generated. The incidental phrases determined according to the response keyword and the ego state vector are stored in the scenario database 33. That is, in the scenario database 33, the ego state vector of the stimulus is registered in association with the ego state vector of the stimulus, the incidental phrase is registered in association with the ego state vector of the stimulus, and is included in the text data d4. Is added to the predicted keyword, and a response keyword is registered. Further, among the keywords included in the text data d4, a keyword to be interpreted as a command word from the user is associated with a command for performing other work such as a recipe search. Further, since the response prosody is determined from the response text and the ego state vector, the response text determination unit 32 also generates a prosody parameter for prosody control.

  The response text and prosodic parameters determined by the response text determination unit 32 are provided to the speech synthesis processing unit 17 that performs text synthesis. The speech synthesis processing unit 17 generates a synthesized speech for response using the text and the prosodic parameters, and outputs it as a response speech to the user through the speaker 43.

  In the following, a specific example will be described in which a dialogue agent is searched for a cooking recipe. In this example, when the user verbally conveys the material to the dialog agent, the search is made for a plurality of candidate names that can be made from the material, and when the user selects a desired name from the displayed candidates. Assume that the dialogue system is constructed so that the dialogue agent presents the recipe of the dish. The dish name and recipe are presented on the display 44. In addition, an instruction from the user to the dialogue agent may be configured to use an operation unit such as a switch, but here, the instruction from the user is only by voice. The recipe data to be searched by the dialog agent is registered in the dialog agent, or the dialog agent is provided with a Web search function and is obtained by Web search. Alternatively, what is registered in the dialogue agent may be preferentially presented, and the recipe may be obtained by web search when the user requests a recipe other than the registered recipe.

  Table 7 shows a specific example in a case where the dialogue agent is requested to search for a cooking recipe. In Table 7, “ego state (interest)” represents the user's ego state, and “ego state (d)” represents the ego state of the dialog agent. In the configuration example described above, the ego state vector of the stimulus is determined using the facial expression summary data d1, the voice emotion data d2, the tone data d3, and the text data d4. Using the facial expression summary data d1, the voice emotion data d2, and the tone data d3, it is found that the user's emotion is one of “seduce”, “anger”, “joy”, and “unknown”. It is assumed that incidental phrases are obtained. Since the ego state estimation unit 20 determines the ego state vector of the stimulus by using the emotion and the accompanying phrase, the dialogue control unit 30 determines the ego state vector of the stimulus using the corresponding rule of the response strategy determination unit 31. . When the ego state vector of the stimulus is determined, the response text determination unit 32 determines the text to respond to the user and the prosody for response using the keyword of the text data d4 and the auxiliary words registered in advance.

  By the way, by recognizing whether or not the user is registered in advance, the dialogue agent can use the characteristics of a specific user to increase the recognition rate or perform user authentication. A user recognition unit 50 is provided. As shown in FIG. 2, the user recognition unit 50 collates the feature amount of the user's voice input from the microphone 42 with the user database 51 and extracts a user candidate, A face image recognition processing unit 54 that extracts a user candidate by collating the feature amount of the user's face image captured by the camera 41 with the face image database 53; Here, for convenience, the user database 51 and the face image database 53 are separately described in the figure, but both can be collectively used as a user database. The user candidates extracted by the user recognition processing unit 52 and the user candidates extracted by the face image recognition processing unit 54 are input to the user determination unit 55, and the user determination unit 55 uses fuzzy logic or the like. By combining the user candidates, the user is determined and the identification information given to each user is output. As described above, the user's voice feature value and the user's face image feature value are collated with the user database to extract the user candidate, and the extracted user candidate is used to select the user. Since it is determined, the recognition rate of the user is increased.

  In order to employ the identification information output from the user recognition unit 50 in the dialogue control unit 30, the scenario database 33 stores information such as user preferences and user personalities in the dialogue in association with the identification information. When the identification information is obtained from the user recognition unit 50, the dialogue control unit 30 determines a corresponding rule according to the user, and determines the ego state vector using the corresponding rule. In order to enable this operation, in the scenario database 33, a correspondence rule corresponding to the user's preference and personality is associated with identification information, and in the dialog control unit 30, the identification information is obtained by the user recognition unit 50. Sometimes the correspondence rule associated with the identification information is used, and when the identification information cannot be obtained, the default correspondence rule is used. Further, by obtaining the user identification information, the dialog agent can use the user name for calling.

  The dialogue agent can express with a body motion using a virtual body displayed on the display 44, and a friendly response is possible by performing the body expression. In order to perform such body representation by a virtual body, the dialogue agent of the present embodiment includes a body model representation unit 18 in which a data group for representing a virtual body is stored in a storage device, and dialogue control. There is provided a body expression control unit 19 that converts response contents (text, agent's ego state, stimulating user's ego state) determined by the unit 30 into body movements of the body model expression unit 18. Here, the conversion to the body movement of the body model expression unit 18 is to extract a data group necessary for expression from the data group of the body expression stored in the body model expression unit 18 and display the extracted data group on the display. This means that it is applied to the virtual body displayed at 44. By this processing, it becomes possible to control the gesture / hand gesture of the agent displayed on the display 44 (lifting hands, reaching out, shaking head, whispering, etc.) according to the response content of the dialogue agent.

  By the way, since the voice emotion estimation part 14 and the tone estimation part 15 use a Gaussian mixture model, it is necessary to learn beforehand. Below, the learning method of the tone estimation part 15 is demonstrated first, and the learning method of the audio | voice emotion estimation part 14 is demonstrated next. In learning by the tone estimation unit 15, it is necessary to collect 20 kinds of tone voices shown in Table 1 as data. Therefore, the inventors first collected text from the Web using a search engine. In order to collect the text, a search sentence combining []] and a utilization form was used. For example, in the case of a “command” tone, when “[” is commanded] is used as a search sentence, a text such as “(command)“ go ahead ”” is extracted. In this case, “go ahead” can be used as the text for the “command” tone. The text extracted in this way was automatically shaped, and sentences containing inappropriate expressions were removed manually. Also, depending on the tone, there were cases where a sufficient number of sentence examples could not be collected on the Web, so the tone that was able to collect more than 50 sentences was used for learning. The tone in Table 1 is a tone that can be collected over 50 sentences. In addition, in order to identify each tone, the tone data d3 used the combination of the ego state and the numerical value in Table 1 (see Table 4). For example, in the “command” tone, the tone data d3 is “CP1”, and in the “comfort” tone, the tone data d3 is “NP2”.

  After collecting the texts of each tone, the actors read the texts in the simple soundproof room, and they also acted on the facial expressions. There were 6 actors in total, including a father role (2 men), a mother role (2 women), and a child role (2 women). In addition, 5 texts, which are different from 30 sentences with tone, were read aloud without tone. The parameters at the time of acoustic analysis are as shown in Table 2.

  After learning the tone estimation unit 15 by the above-described method, the ability to identify the tone by the evaluation voice was tested. Six actors each generated five sentences, and the average of all the actors that were correctly identified by the tone estimation unit 15 was averaged. As a result of the experiment, the rate of correctly recognizing the tone with respect to the voice with tone was 49.5%, and the rate of recognizing that there was no tone with respect to the speech without tone was 90%. Note that the tone with the tone data d3 of CP2, FC1, AC1, AC3, and AC4 could be identified at a high rate, and the tone with the tone of CP4 and NP3 was slightly low.

  In the learning of the voice emotion estimation unit 14, as with the learning of the tone estimation unit 15, six actors read out the text. As described above, the user's emotions identified by the voice emotion estimation unit 14 are the three types of “joy”, “anger”, and “seduce”, “joy” is “FC1” in Table 1, and “anger” is Table 1. “FC2” and “Silence” in FIG. 1 are considered to correspond to A1 to A4 in Table 1, and the speech emotion estimation unit 14 is trained using the same text as the tone estimation unit 15.

  The voice emotion estimation unit 14 after learning was experimented on the ability to identify emotions using evaluation voices. For the evaluation speech for identifying “peace”, the speech that read the text corresponding to “joy” without tone and the speech that read the text corresponding to “anger” without tone were used. The number of texts evaluated was 5 sentences for each of the 6 actors for “joy” and “anger”, a total of 30 sentences, and 10 sentences for each of the 6 actors for “seduce”. A total of 60 sentences were used. Table 8 shows the identification results obtained by the voice emotion estimation unit 14. In any case, it was possible to discriminate with high accuracy, and an average discriminating performance of 85.6% was obtained.

(Embodiment 2)
This embodiment records the ego state history for each user using the fact that the user recognition unit 50 can identify the user, and estimates the user's personality using the user's ego state history. It is possible to respond to the personality of the user.

  In the present embodiment, as shown in FIG. 4, an ego state history storage unit 61 that stores and stores the user's ego state estimated by the ego state estimation unit 20 in association with the user specified by the user recognition unit 50. The ego state feature extracting unit 62 for estimating the user's personality from the distribution pattern of the appearance frequency of the user's ego state stored in the ego state history storage unit 61, and the personality estimated by the ego state feature extracting unit 62 are used. An ego state feature storage unit 63 that stores the information in association with the person is added. The ego state history storage unit 61 stores the user's ego state as a history in chronological order and stores the appearance frequency of each ego state.

  The ego state feature extraction unit 62 collates the ego state appearance frequency distribution pattern and the registered ego state feature database 64 in association with each other, and estimates the personality having a high degree of matching as the personality of the user. . That is, in the ego state feature database 64, distribution patterns of appearance frequencies of five kinds of ego states in the AC analysis (distribution patterns obtained by normalizing the appearance frequency) are registered in association with the personality, and the ego state history storage unit The distribution pattern of the appearance frequency of the ego state for each user stored in 61 is collated with the distribution pattern registered in the ego state feature database 64. This matching is a pattern matching and a high similarity is selected, and the selected personality is estimated as the personality of the user.

  In the ego state characteristic database 64, for example, when the NP and FC are low and the CP and AC are high as the ego state, “a personality that makes it difficult to express yourself and is prone to depression” “a personality that does not attend school” Characters such as “a sense of responsibility, reality-examination ability, and cooperativity, but lacking compassion” are associated. The personality estimation result in the ego state feature extraction unit 62 is stored in the ego state feature storage unit 63 in association with the user.

  By the way, when the personality of the user is stored in the ego state feature storage unit 63, the dialogue control unit 30 acquires the personality of the user from the ego state feature storage unit 63 when performing a dialogue with the user. . Since the correspondence rule is registered in the scenario database 33 in association with the identification information of the user, the interaction control unit 30 can select the correspondence rule corresponding to the character of the user from the scenario database 33, and as a result In addition, a smooth conversation according to the personality of the user becomes possible.

Further, when the dialog agent of the present embodiment is used for a nursing robot or the like, it is possible to make the dialog agent function like a therapist by estimating the personality of the user. For example, when the user's personality tends to be pessimistic or masochistic, it becomes possible to make a response that improves the user's psychological state. Now, it is known that when the NP and FC are low and the CP and AC are high in the appearance frequency of the ego state, as described above, there is a depression tendency. Therefore, if a general user performs the dialogue shown in [1] below between the dialogue agent and the user, the NP and FC are low in the frequency of appearance of the ego state, and the CP and AC are high. Then, as shown in [2] below, a positive expression is used to encourage the user and a dialogue that improves the user's psychological state becomes possible.
[1]
Dialogue agent: It's time for medicine.
User: I already know.
Dialogue Agent: Well thank you.
[2]
Dialogue agent: It's time for medicine.
User: I already know.
Dialogue Agent: This will help you, so please do well. Because it is a little more.

  As described above, the ego state history storage unit 61 stores the ego state history (including the appearance frequency of the ego state) for each user, and the ego state feature storage unit 63 stores each user. Since the personality estimation result is stored, when the user receives counseling, the counselor can provide these data for counseling. Other configurations and functions are the same as those of the first embodiment.

(Embodiment 3)
In each of the above-described embodiments, the case where there is one user is exemplified. However, in the present embodiment, there are two users and a configuration that enables a three-party dialogue including a dialogue agent is possible. explain. In the present embodiment, as shown in FIG. 5, a conversation partner recognition unit 65 that monitors the direction of the line of sight of a person included in the image captured by the camera 41 and recognizes the conversation partner based on the direction of the line of sight, a user, A dialogue recording unit 66 for accumulating dialogue data including dialogue time, dialogue partner, text, and ego state vector is added. The user and the utterance can be acquired by the user recognition unit 50, the conversation partner can be acquired from the conversation partner recognition unit 65, and the utterance text and the ego state between the user and the conversation partner can be acquired from the dialog control unit 30. . In the present embodiment, it is necessary to use a camera 41 having a resolution that can monitor the position of the user's eyes and the position of the pupil.

  Moreover, in the user recognition part 50, when the face image recognition process part 54 recognizes two users from the image imaged with the camera 41, the dialogue control part 30, the emotion recognition part 13, the ego state estimation part 20, By notifying the dialogue partner recognizing unit 65, the operation is switched to an operation in which two users are present. Thereafter, the user recognition unit 50 identifies the user who spoke in the user recognition processing unit 52 using the user's voice input from the microphone 42, and the dialogue control unit 30, the emotion recognition unit 13, the ego state estimation unit. 20. Notify the recognized user to the dialogue partner recognition unit 65. In short, in the present embodiment, the image captured by the camera 41 is used to identify the number of users and the conversation partner of the user, the emotion of the user uttered by the voice input from the microphone 42 is estimated, and the utterance Estimate the ego state vector between the selected user and the conversation partner. The ego state vector estimated by the ego state estimation unit 20 and the text data d4 obtained by the text extraction unit 16 are given to the dialogue control unit 30, and the dialogue control unit 30 uses the dialogue data described above as the dialogue recording unit 66. To record.

  In the scenario database 33 in this embodiment, a dialogue scenario between three parties is stored separately from the dialogue scenario between two parties as in the above-described embodiments, and two users are received from the dialogue partner recognition unit 65. Is notified to the dialogue control unit 30, the dialogue control unit 30 selects a dialogue scenario between the three parties from the scenario database 33. In the three-party dialogue scenario, the dialogue agent is set to speak only when the dialogue partner includes the dialogue agent, and the dialogue agent is set not to speak from the dialogue agent. That is, the dialogue control unit 30 uses the utterance text extracted by the text extraction unit 16, the dialogue partner recognized by the dialogue partner recognition unit 65, and the ego state estimated by the ego state estimation unit 20 (dialog recording unit 66). The dialogue agent determines whether or not to speak. When the conversation agent does not speak, the process returns to the process of confirming the number of users from the image captured by the camera 41, and when the conversation agent speaks, through the speech synthesis processing unit 17 and the body expression control unit 19 as in the above-described embodiments. Speak with physical expression.

An example of a dialogue scenario using this embodiment is shown below. In the following example, a dialogue agent is provided in a nursing robot, and there are two users, a nursing person and a visitor. [1] is a case where the dialog agent does not speak, and [2] is a case where the dialog agent speaks.
[1]
Nurse → Visitor (FCa → FCa): Have you ever changed in school?
Visitors → Nurses (FCa → FCa): As usual.
[2]
Nurse → visitor (FCa → NP): It ’s hard.
Visitors → Nurses (NP → FCa): It will soon improve.
Nursing robot-> nurse (NP-> FCa): Yes, please do your best.

  As described above, according to the configuration of the present embodiment, even when there are a plurality of users, a conversation is possible. For example, when a dialogue agent is provided in a nursing robot, A dialogue between the customer and the nursing robot is possible. Here, the dialogue control unit 30 refers to the dialogue data in the dialogue recording unit 66 in order to determine whether or not a response from the dialogue agent is requested, and the dialogue partner includes the dialogue agent. The utterance by the dialogue agent will be performed only during the period.

  In this embodiment, a three-way dialogue between two users and a dialogue agent is illustrated. However, by extending the technology of this embodiment to three or more users, a dialogue with a larger number of people can be performed. Is possible. Other configurations and operations are the same as those of the first embodiment.

(Embodiment 4)
In this embodiment, the speed of the synthesized speech when the dialogue agent responds is adjusted by judging the user's arousal level. In other words, when the user's arousal level is low, if the user responds at a fast tempo, the response may not be heard by the user. Therefore, the user's arousal level is determined, and the speed of the synthesized voice output from the speaker 43 is adjusted according to the user's arousal level.

  In the present embodiment, the skin potential level (SPL) is used to determine the user's arousal level. In order to obtain the skin potential level, as shown in FIG. 6, an electrode 71 (a pair of electrodes 71) can be contacted with a site where the amount of sweating is likely to change due to mental state, such as the palm of a user or the sole of a foot. The electrode 71 is connected to the potential measuring unit 72, and the potential measuring unit 72 measures the potential difference between the pair of electrodes 71. When the electrode 71 is attached to the user, the user may feel annoyed. For example, when the dialogue agent is incorporated into the nursing robot, the user can touch the electrode 71 by an action such as shaking hands with the user. It is desirable to touch. The potential difference measured by the potential measuring unit 72 is input to the arousal level determination unit 70, which converts the potential difference into the arousal level. In general, the greater the potential difference (SPL) is, the higher the arousal level is. Therefore, if the potential difference is large with respect to an appropriately set threshold, the arousal level determination unit 70 determines that the user's arousal level is high. In the present embodiment, the user's arousal level is determined in two levels, but the arousal level may be determined in multiple levels. By the way, since the potential difference measured by the potential measuring unit 72 has individual differences depending on the user, the detection result in the potential measuring unit 72 is stored and stored in association with the user recognized by the user recognition unit 50. The average value of the accumulation results for each user may be used as the above threshold value.

  The user's arousal level obtained by the arousal level determination unit 70 as described above is given to the dialogue control unit 30. When the user's arousal level is low (SPL is lower than the threshold value), the synthesized voice The output speed of the synthesized speech generated by the speech synthesis processing unit 17 is adjusted so that the response speed according to is a relatively slow tempo. Thus, by responding to a user with a low arousal level at a slow tempo, the contents of the text can be easily heard. On the other hand, when the user's arousal level is high (SPL is higher than the threshold), the user's arousal level is maintained by responding at a slightly faster tempo so that the user is not frustrated by the slow response. In other words, by adjusting the speed of the synthesized speech according to the user's arousal level, if the user's arousal level is low, the response is made easier to hear the user by responding at a slow tempo, and conversely the user's arousal level If it is high, it becomes possible to respond without making the user wait by responding at a fast tempo. Other configurations and operations are the same as those of the first embodiment.

(Embodiment 5)
In the present embodiment, it is determined whether or not the ego state estimated by the dialogue agent is appropriate for the dialogue by detecting the biological information of the user. That is, when the user and the dialogue agent interact, the ego state estimation unit 20 estimates the user's ego state and the ego state stimulated by the dialogue agent based on the user's utterance. In general, smooth interaction is possible by performing complementary exchange using the estimated ego state. Here, the smooth dialogue means that the user talks without harming emotions. On the other hand, in an exchange where an intersection of ego states occurs instead of a complementary exchange, the user may harm emotions and the dialogue may not proceed smoothly.

  Therefore, in the present embodiment, the user's instantaneous heart rate (1 minute heart rate) is used to determine the user's emotions. In general, it is known that the instantaneous heart rate increases when a defensive or aggressive emotion such as anger, stress, and fear occurs, and decreases when it is calm. In this embodiment, by using such a physiological phenomenon, the instantaneous heart rate of the user during the conversation is obtained, and the fluctuation of the instantaneous heart rate is monitored, so that the user has a defensive or aggressive feeling. Judging whether or not there is. ECG is used to detect the instantaneous heart rate, and electrodes 73 (a pair of electrodes 73 are provided) as in the fourth embodiment. The electrode 73 is attached to the chest or limbs of the user.

  As shown in FIG. 7, the electrode 73 is connected to the ECG measurement unit 74, and a potential difference detected by the pair of electrodes 73 is detected by the ECG measurement unit 74. In the ECG measurement unit 74, for example, a potential change as shown in FIG. 8 is detected. This potential change includes a P wave indicating atrial excitation in the heart, a QRS indicating ventricular depolarization, and a T wave indicating ventricular fine polarization. Here, the R-R interval is subject to antagonistic control between the heart exchange nerve and the cardiac parasympathetic nerve, and the instantaneous heart rate can be obtained by converting the R-R interval to the heart rate per minute. Therefore, the output of the ECG measurement unit 74 as shown in FIG. 8 is input to the instantaneous heart rate detection unit 75, and the instantaneous heart rate is obtained by obtaining the RR interval. That is, the electrode 73, the ECG measurement unit 74, and the instantaneous heart rate detection unit 75 constitute a biological information measurement unit. If the user has a defensive or aggressive emotion, the instantaneous heart rate rises more than usual. Therefore, the suitability determination unit 76 compares the instantaneous heart rate obtained by the instantaneous heart rate detection unit 75 with an appropriate threshold value. Thus, it is determined whether the user's emotion is defensive or aggressive. That is, the suitability determination unit 76 determines whether or not the ego state estimated by the ego state estimation unit 20 is appropriate. When the instantaneous heart rate is lower than the threshold value, the suitability determination unit 76 determines that the user's ego state estimated by the ego state estimation unit 20 and the stimulated ego state of the dialogue agent are appropriate, and the instantaneous heart rate is the threshold value. When it is above, it is determined that the estimation result of the ego state estimation unit 20 is not appropriate.

  Note that, as indicated by a broken line in FIG. 7, if the determination result of the suitability determination unit 76 is fed back to the ego state estimation unit 20 and used for learning of the ego state estimation unit 20, the likelihood of the estimated ego state can be increased. As a result, the dialog agent's response reduces the possibility of harming the user's emotions and enables a smooth conversation. That is, since the suitability determination unit 76 can determine whether or not the ego state vector estimated by the ego state estimation unit 20 does not harm the user's emotions, the determination result of the suitability determination unit 76 is used as the determination result of the ego state vector. By feeding back to the estimation result, an appropriate ego state vector can be estimated without harming the user's emotion. An optical sensor that monitors the blood flow instead of the electrode 73 may be used to detect the instantaneous heart rate (this type of sensor is used in heart rate monitors in various exercise machines). Other configurations and operations are the same as those of the first embodiment.

1 is a block diagram illustrating a first embodiment. It is a block diagram of the principal part same as the above. It is operation | movement explanatory drawing of a basic composition. FIG. 6 is a block diagram illustrating a second embodiment. FIG. 6 is a block diagram illustrating a third embodiment. FIG. 10 is a block diagram illustrating a fourth embodiment. FIG. 10 is a block diagram illustrating a fifth embodiment. It is a figure which shows an example of ECG used for the same as the above.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Dialogue processing means 11 Facial expression estimation part 13 Face emotion estimation part 14 Speech emotion estimation part 15 Tone estimation part 16 Text extraction part 17 Speech synthesis process part 18 Body model expression part 19 Body expression control part 20 Ego state estimation part 21 Emotion score allocation Unit 22 Tonal Score Allocation Unit 23 Text Score Allocation Unit 24 Score Integration Calculation Unit 30 Dialog Control Unit 41 Camera (Image Input Means)
42 Microphone (voice input means)
43 Speaker (Audio output means)
44 Display (image output means)
DESCRIPTION OF SYMBOLS 50 User recognition part 51 User database 52 User recognition process part 53 Face image database 54 Face image recognition process part 55 User judgment part 61 Ego state history memory | storage part 62 Ego state feature extraction part 63 Ego state feature memory part 65 Dialog partner Recognition unit 66 Dialog recording unit 70 Arousal level determination unit 73 Electrode 74 ECG measurement unit 75 Instantaneous heart rate detection unit 76 Suitability determination unit

Claims (11)

  Voice input means for inputting the user's voice, dialog processing means for generating text responding to the contents of the voice input from the voice input means, and outputting the text generated by the dialog processing means to the user A speech output estimating unit for classifying a user's emotions into a plurality of types using the prosodic features of the speech input from the speech input unit and outputting as speech emotion data; A tone estimation unit that classifies the user's tone into a plurality of types using the prosodic features of the voice input from the voice input means and outputs as tone data, and extracts a sound string from the voice input from the voice input means A text extraction unit that outputs text data, an image input unit that captures the face of the user, and a time point of feature points set for each part of the user's face captured by the image input unit A facial expression estimator that classifies facial expressions based on changes in facial position, and facial expressions extracted by the facial expression estimator are used to classify the user's emotions into multiple types using the facial expression change patterns that accompany temporal changes. The combination of the ego state, which is the model of the emotion of both the face and emotion to be output, and the ego state vector including the direction from the speaker to the listener, is used as the emotion summary data, voice emotion data, tone data, and text data. The ego state estimation unit that estimates the ego state vector based on the user's utterance from the pair, and the ego state vector when responding to the user according to the preset correspondence rule from the ego state vector estimated by the ego state estimation unit And a dialog control unit that automatically determines text to be answered to the user from the contents of the text data. Dialogue agent.   The ego state estimation unit includes an emotion score assigning unit that obtains an emotion ego state score indicating likelihood for each candidate of ego state vector estimated from a combination of the emotion summary data and the voice emotion data, and the voice emotion data For each ego state vector estimated from the content of the text data, and a tone score assigning unit for obtaining a tone ego state score indicating the likelihood for each candidate ego state vector estimated from a combination of the tone data and the tone data Dialogue of ego states included in candidates of ego state vectors obtained by a text score assigning unit for obtaining a text ego state score indicating likelihood, an emotion score assigning unit, a tone score assigning unit, and a text score assigning unit Classified for each person, and for each person's ego state, the candidate's emotional ego state score, tone ego state score, and text ego The weighted sum obtained by multiplying the state score by each weighting factor is obtained as the integrated score, which is the likelihood evaluation value, and the likelihood of the combined score for each ego state of each person included in the candidate for the ego state vector The interactive agent according to claim 1, further comprising: a score integration calculation unit that estimates an ego state having the maximum degree as an ego state of each person in an ego state vector based on a user's utterance.   The emotion ego state score is given to the candidate of ego state vector when the emotions indicated by the emotion summary data and the voice emotion data match, and the candidate of the ego state vector obtained when the emotion does not match The tone ego state score is given to the candidate of ego state vector when there is no contradiction between the voice emotion data and the tone data, and there is a contradiction The text ego state score is a numerical value that is allocated to the ego state vector candidates when the text data includes a specific incidental phrase. A text ego letter assigned to a candidate for an ego state vector corresponding to one supplementary phrase, in which numerical values are assigned in descending order of likelihood. Dialogue agent of claim 2, wherein the sum of the scores is characterized by comprising a scale.   The weighting factor for obtaining the integrated score is a sum of a weighting factor for the emotional ego state score and a weighting factor for the tone ego state score is greater than the weighting factor for the text ego state score, and the weighting factor for the text ego state score is the user 4. The dialogue agent according to claim 2, wherein a weighting factor for the stimulated ego state is larger than a weighting factor for the ego state.   The dialog control unit has a function of generating a prosodic parameter of speech from the determined ego state vector for response and the determined response text, and the text output means is for the response determined by the dialog control unit. 5. The speech synthesis processing unit for generating a synthesized speech in which prosodic parameters are applied to text, and speech output means for outputting the synthesized speech generated by the speech synthesis processing unit. The dialogue agent according to any one of the above items.   2. The facial expression estimation unit classifies seven types of facial expressions of “no expression”, “surprise”, “fear”, “disgust”, “anger”, “happiness”, and “sadness”. The dialogue agent according to claim 5.   It has a user database in which user voice features and user face features are registered in association with users, and is picked up by the user voice features input from the voice input means and the image input means. A user recognizing unit that identifies a user by comparing the features of the user's face image with a user database, and the dialog control unit is pre-registered with user attributes specified by the user recognizing unit. In addition to the ego state vector estimated by the ego state estimation unit and the text data when determining the ego state vector and text when responding to the user, the user attribute is also used. The dialogue agent according to any one of claims 1 to 6.   The ego state history storage unit that stores and stores the ego state vector estimated by the ego state estimation unit in association with the user specified by the user recognition unit, and the user's ego state stored in the ego state history storage unit An ego state feature extraction unit that estimates the user's personality from the distribution pattern of the appearance frequency of the vector and an ego state feature storage unit that stores the personality estimated by the ego state feature extraction unit in association with the user are added. 8. The dialogue agent according to claim 7, wherein the dialogue control unit determines an ego state vector and text when responding to the user by using the personality of the user stored in the ego state feature storage unit.   In the correspondence rule of the dialog control unit, the ego state stimulated by the user's utterance is set as the ego state at the time of response, and the user's ego state stimulated at the time of response is set as the ego state at the time of the user's previous utterance. The dialogue agent according to any one of claims 1 to 8, characterized in that:   A body model expression unit that performs expression accompanied by a body motion and a body expression control unit that converts the ego state vector and text determined by the dialog control unit into the body motion of the body model expression unit are added. The interactive agent according to any one of claims 1 to 9.   The ego state is classified into five types of “critical parents”, “protective parents”, “adults”, “free children”, and “adapting children”, which are models of mind based on the exchange analysis. The dialogue agent according to any one of claims 1 to 10, wherein the dialogue agent is characterized.

Images