JP2011081445A - Facial expression recognition device, inter-personal feeling estimation device, facial expression recognizing method, inter-personal feeling estimating method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize the head posture of a person and the facial expression of the person with high accuracy, even if the person changes the head posture. <P>SOLUTION: An input unit 1 inputs an input moving image capturing the plurality of persons. Facial expression estimation units 3-1 to 3-N estimate the head posture of each person based on the likelihood of each person's head posture for the input moving image and estimate the display category of each person based on the likelihood of each person's facial expression category for the input image. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a human facial expression recognition technique based on image processing, and an emotion estimation technique between persons, in particular, a time series filtering technique, a facial expression recognition apparatus based on a statistical model, and an interpersonal emotion estimation apparatus. The present invention relates to a facial expression recognition method, an interpersonal emotion estimation method, and a program.

  Face-to-face dialogue is a basic form of daily communication for daily information sharing and decision making. In face-to-face conversations, not only linguistic information such as utterance contents but also non-linguistic information such as gestures, gaze, and prosody are expressed, and both are important psychologically (reference 1: M.M. Argyle: “Bodily Communication-2nd ed.”, Routledge, London and New York, 1988.).

  Among emotional expressions in dialogue, smiles are considered to be particularly important. Because smiles facilitate the flow of dialogue, express positive emotions, form and maintain intimacy (Reference 2: JN Cappella: Behavioral and judged coordination in adult informal social interactions: vocal and kinesic indicators , J. Personality and Social Psychology, 72 (1): 119--131, 1997).

  Generally, there is ridicule in something visually similar to smile. It is thought that smile is more important than ridicule because it does not involve utterances and changes in facial expressions are much smaller, so it can be easily co-occurred with the utterances of the speaker and the smiles of others. But in dialogue, there are more important differences between the two. That's because smiles have a clear direction, that is, they are basically aimed at a specific audience, while ridicule has little such function.

  So far, several methods for automatic recognition of smiles have been proposed. Some have attempted to improve smile detection accuracy using information on the head posture (see, for example, Non-Patent Document 1). There are also studies such as recognition of ridicule (for example, see Non-Patent Document 2) and recognition of whether or not they are interested (for example, see Non-Patent Document 3).

  For recognition of facial expressions, it is desirable to use a camera that is a non-contact / non-invasive device, that is, to be able to recognize from an image. However, recognizing smiles from images has various technical challenges. First, in smiles, the amount of movement of facial parts is small, that is, the change in appearance in the image is small.

  Furthermore, in a three-or-more-person dialogue, participants in the dialogue are positioned in various directions with respect to the subject person, so that the person to be watched will not be attached unless the camera is attached to the head of the subject person. The direction of the face changes with the change. Therefore, as assumed in many existing facial expression recognition methods (for example, see Non-Patent Document 4), the assumption that the human face in the input image is a front face, that is, the front / upright with respect to the camera is Basically not true. For this reason, it is necessary to simultaneously estimate the head posture in order to correctly recognize the facial expression.

  Furthermore, facial expressions vary greatly between individuals. For this reason, when trying to recognize smiles, it is highly possible that smiles will be confused with visually similar facial expressions such as ridicule unless individual differences in smiles are addressed. Many existing facial expression recognition methods take an approach of preparing a single model from learning data for facial expressions of a large number of people and using it as a model for an unspecified number of people (for example, non-patented) Reference 5).

  Patent Document 1 discloses a method aiming at achieving a high recognition rate by preparing a simple model for each individual. In this method, a small number of points of interest are fixed to the periphery of a facial part on the face, and the facial expression is recognized by utilizing the fact that the luminance of the image at those points changes as the facial expression changes.

JP 2009-110426 A

J. Cohn, L. Reed, T. Moriyama, J. Xiao, K. Schmidt, and Z. Ambadar: Multimodal coordination of facial action, head rotation, and eye motion during spontaneous smiles, In Proc.IEEE Int'l Conf. Automatic Face and Gesture Recognition, pages 129--135, 2004. S. Petridis and M. Pantic: Audiovisual laughter detection based on temporal features, In Proc.ICMI, pages 37--44, 2008. B. Schuller, R. Muller, B. Hornler, A. Hothker, H. Konosu, and G. Rigoll: Audiovisual recognition of spontaneous interest within conversations, In Proc. ICMI, pages 30--37, 2007. Y. L. Tian, T. Kanade, and J. Cohn: Facial expression analysis, Springer, 2005. M. S. Bartlett, G. Littlewort, M. G. Frank, C. Lainscsek, I. R. Fasel, and J. R. Movellan: Automatic recognition of facial actions in spontaneous expressions, J. Multimedia, 1 (6): 22--35, 2006.

  However, the techniques of Non-Patent Document 1 to Non-Patent Document 3 described above have a problem that it is impossible to automatically recognize to whom the smile is directed.

  Most of the image-based facial expression recognition methods introduced in Non-Patent Document 4 assume that the facial expression in the input image is a front face, that is, front / upright with respect to the camera. However, in a scene where the target person changes the head posture including dialogue, a camera must be attached to the head of each target person in order to obtain such an image, which is not practical.

  In the method of Non-Patent Document 5, it is only necessary to create a model that can achieve a high recognition rate for an unlearned person, but it has been reported that the current technique causes an overfitting problem.

  In the method of Patent Document 1, the position of the point of interest is determined without considering the change in facial expression. Therefore, a point suitable for recognition of the target facial expression is not always selected. As a result, visually similar expressions such as smiles and ridicule are still likely to be confused. However, any of the facial expression recognition methods described in Patent Document 1 and Non-Patent Document 1 to Non-Patent Document 5 basically recognizes what a person's facial expression is. There has never been a method for recognizing to whom each facial expression expressed in is directed.

  The present invention has been made in view of such circumstances, and its purpose is to recognize a person's facial expression together with the person's head posture with high accuracy even when the person changes the head posture. An expression recognition device, an interpersonal emotion estimation device, an expression recognition method, an interpersonal emotion estimation method, and a program are provided.

  In order to solve the above-described problem, the present invention provides an input unit that captures a person and inputs the input moving image as an input moving image, and the human head posture likelihood for the input moving image input by the input unit. The head posture is estimated and output as a head posture estimated value, and the person's display category is estimated based on the likelihood of the facial expression category with respect to the input image input by the input means. A facial expression recognition device comprising facial expression estimation means for outputting.

  According to the present invention, in the above invention, the facial expression estimating means is a set of attention points that are predetermined for recognition of the expression category, and the attention point set arranged in a predetermined area of the input moving image A first likelihood is calculated based on the luminance distribution model and the luminance distribution model, and is a set of attention points that are predetermined for estimating the head posture, and is arranged in a predetermined region of the input moving image. A second likelihood is calculated based on the set of attention points and its luminance distribution model, and the product of the first likelihood and the second likelihood is calculated simultaneously with the facial expression category and the head posture. It is calculated as a likelihood, and the expression category estimated value and the head posture estimated value are estimated and output based on the calculated simultaneous likelihood.

  According to the present invention, in the above invention, the facial expression estimating means is a region in which a change in facial expression of a person is remarkable in the input moving image, wherein a set of points of interest predetermined for recognition of the facial expression category is defined. It is characterized in that it is arranged in a predetermined area for a person.

  In order to solve the above-described problem, the present invention provides an input unit that inputs an input moving image obtained by photographing a plurality of persons, and a likelihood of the head posture of each person with respect to the input moving image input by the input unit. Based on the likelihood of each person's facial expression category with respect to the input image input by the input means And the facial expression category and head posture estimation means for outputting each person's display category estimation value, and based on the head posture estimation value of each person, It is an emotion estimation apparatus between persons characterized by including the estimation object estimation means which estimates and outputs as an expression object estimation value.

  According to the present invention, in the above-described invention, there is further provided an interpersonal emotion network creating means for graphically outputting an emotional relationship between the plurality of persons based on the expression category estimated value and the expression target estimated value. It is characterized by providing.

  According to the present invention, in the above invention, the type of dialogue, the degree of excitement of dialogue, the human relationship, An estimation unit is provided for estimating at least one of the composition of subgroups and the personality of the relationship.

  According to the present invention, in the above invention, the input unit captures a person and inputs it as an input moving image, and the facial expression estimation unit is based on the likelihood of the head posture with respect to the input moving image input by the input unit. The head posture of the person is estimated and output as a head posture estimated value, and the display category of the person is estimated based on the likelihood of the facial expression category for the input image input by the input means. And a step of outputting as an estimated value.

  According to the present invention, in the above invention, the input unit inputs an input moving image obtained by photographing a plurality of persons, and the facial expression estimating unit determines the head posture of each person with respect to the input moving image input by the input unit. Based on the likelihood, the head posture of each person is estimated and output as a head posture estimated value of each person, and based on the likelihood of each person's facial expression category for the input image input by the input means, Estimating the display category of each person and outputting the estimated display category value of each person; and a representation target estimating means for expressing the facial expression of each person based on the estimated head posture value of each person. And a step of estimating a target and outputting it as a target estimation value.

  According to the present invention, in the above invention, an input function for photographing a person and inputting it as an input moving image on a computer of a facial expression recognition device that recognizes the expression of the person based on image processing, and a head posture with respect to the input moving image The head posture of the person is estimated based on the likelihood, and is output as a head posture estimated value, and the display category of the person is estimated based on the likelihood of the facial expression category for the input image. A program that executes a facial expression estimation function that is output as an estimated value.

  According to the present invention, in the above invention, an input moving image in which a plurality of persons are photographed on a computer of an interpersonal emotion estimation apparatus that recognizes facial expressions of a plurality of persons based on image processing and estimates emotions between the plurality of persons. An input function for inputting an image, estimating the head posture of each person based on the likelihood of the head posture of each person with respect to the input moving image, and outputting the estimated head posture of each person as well as the input Estimating the display category of each person based on the likelihood of each person's facial expression category with respect to the image, and outputting the facial expression category and head posture estimation function as each person's display category estimation value, estimating the head posture of each person According to the present invention, there is provided a program for executing an expression target estimation function for estimating an expression target of each person's facial expression based on a value and outputting the estimated expression target value.

  According to the present invention, even when the person changes the head posture, the facial expression of the person can be recognized with high accuracy together with the head posture of the person.

It is a block diagram which shows the whole structure of the facial expression recognition apparatus in this embodiment. It is a figure which shows an example of an imaging | photography scene and an example of the input image which image | photographed it with the omnidirectional camera. 3 is a block diagram showing a specific configuration of a facial expression estimation unit 3. FIG. It is a block diagram which shows the specific structure of the expression category and the expression target estimation part 32. FIG. 6 is a block diagram showing a specific configuration of a facial expression category and head posture estimation unit 321. FIG. It is a conceptual diagram which shows the relationship between the expression category e _t , the head posture h _t , and the image z _t . It is a block diagram which shows the process of the shape model creation part 5. FIG. It is a schematic diagram which shows an example of the attention point set registered about each face part. It is a block diagram which shows the flow of a series of processes of a brightness | luminance average map, a brightness | luminance dispersion map, attention point selection, and preparation of a brightness distribution model. It is a schematic diagram which shows an example of the brightness | luminance average map about smile, ridicule, etc. It is a schematic diagram which shows an example of the brightness | luminance dispersion map about smile, ridicule, etc. It is a schematic diagram which shows the expression saliency map about smile, ridicule, and others. It is a conceptual diagram which shows an example which expressed the emotion network between persons as a graph structure.

  The present invention correctly identifies facial expressions such as minute, ridicule, bitter laughter, and anger that are expressed among the participants from a moving image obtained by capturing a conversation for a multi-person conversation, and also expresses emotions between persons based on them. presume. That is, it recognizes who expresses what expression to whom and when from the dialogue moving image, and estimates the feelings between the dialogue participants such as favorable / hostile from there. In the facial expression recognition part, facial expressions that are visually similar but have different properties with respect to emotions between persons are correctly identified.

The facial expression category is robust against the subject's head posture by estimating the subject's facial expressions such as micro, ridicule, bitter laughter, surprise, anger, sadness and thinking at the same time as the person's head posture. Recognize In this embodiment, “estimating the facial expression simultaneously with the head posture” means “estimating the facial expression together with the head posture” or “estimating the facial expression in parallel with the head posture”.
Furthermore, it recognizes the person of the visual gaze target based on the recognized head posture, and assumes that the person of the visual gaze target and the person of the facial expression expression are the same. Identify people. Finally, based on the facial expression category in each frame in the input image and the expression target, it is estimated how the emotions between persons are.

  First, for the method of simultaneously estimating the facial expression and head posture in each frame in each moving image for each subject, the attention point suitable for recognizing the target facial expression of the target person is automatically selected from the learning image This framework is newly introduced in the present invention (the other parts are in accordance with Patent Document 1).

  Next, the expression target of the expression, that is, the target person to whom the expression is directed is estimated. Existing knowledge that there is a high correlation between visual gaze target and head posture during dialogue (Reference 3: R. Stiefelhagen: Tracking focus of attention in meetings, In Proc. ICMI, pages 273--280 , 2002.), it is estimated from the estimated head posture of the target person to whom the facial expression was directed by the framework of maximum likelihood method.

  Finally, all the test subject's facial expressions and the time-series data of the person to be expressed are integrated, and various indexes that suggest emotions between the persons in the conversation are calculated. Specifically, the inter-personal emotion is estimated based on how friendly expressions, hostile expressions are expressed, or how many facial expressions co-occur. Furthermore, the estimated emotions between persons are displayed in a graph structure that is easy to understand visually.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 1 is a block diagram showing the overall configuration of the facial expression recognition device in the present embodiment. In the figure, the facial expression recognition device includes an input unit 1, learning image creation units 2-1 to 2-N (hereinafter collectively referred to as a learning image creation unit 2), and facial expression estimation units 3-1 to 3-N ( Hereinafter, it is referred to as a facial expression estimation unit 3 when collectively referred to) and an emotion network creation unit 4 between persons. The input unit 1 captures a moving image of the face of each participant using a CCD camera or the like as an imaging unit. The learning image creation units 2-1 to 2-N respectively include learning images of the respective conversation participants P _{1 to} P _N and moving images 10-1 to 10-N to be estimated (hereinafter collectively referred to as learning images 10a, The input moving image 10b) is output.

Expression estimating unit 3-1 to 3-N, respectively, enter the moving image from each interaction participants _P 1 to P _N of the learning image and the estimated target moving image 10-1 to 10-N, and the input unit 1 and then, the conversation participants _P 1 to P _N, and outputs the estimated value 11-1 to 11-N of the estimated expression category and exposed object in real time. The person-to-person emotion network creation unit 4 generates facial expression categories and expression target person estimation values 11-1 to 11-N (hereinafter collectively referred to as expression categories and expression objects) of all the conversation participants P _{1 to} P _N. Estimate the network structure of emotions between participants, such as “who smiled at how often” and “who and who ’s facial expressions often co-occurred” Output. The functions of the learning image creation units 2-1 to 2-N, the facial expression estimation units 3-1 to 3-N, and the inter-person emotion network creation unit 4 are realized by software processing.

  It is assumed that the number N of participants in the dialogue is already known, and the internal parameters of the camera of the input unit 1 have already been obtained by prior calibration. However, for the number N of participants in the dialogue, an existing face detector (for example, a cascade type AdaBoost detector based on Haar-like features (Reference 4: P. Viola and M. Jones: Rapid object detection using a boosted cascade of simple features, In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 511--518, 2001).

Here, symbols used mainly below are summarized here. In the description of FIG. 1, although the number of dialogue participants was N, in the following, denoted as N _i, e∈ facial expression category {1, ..., N _e}, also the exposed target expressed in g _i. Here, i represents one of the N _i conversation participants. That is, iε {1,..., N _i }. It is assumed that the expression object is directed to one of the other participants in the dialogue or not directed to anyone. That is, it is discretely expressed as gε {1,..., N _i }. The expression object g _{i, t} = j (≠ i) indicates that the facial expression of the conversation participant P _{i at} the time t is directed to the conversation participant P _j , while g _{i, t} = i is , Indicating that the gaze target does not belong to anyone else, that is, the facial expression is not directed to a specific person.

In this embodiment, the number N of participants in the dialogue is 4 (i = 1 to 4), and the facial expression to be recognized is e∈ {smiling, laughing, laughing, etc.}. That is, the number N _e of the recognized facial expression is 4. In summary, the state e _{i, t} = “smile” and g _{i, t} = j (≠ i) means that the conversation participant P _i is smiling with respect to the conversation participant P _{j at} time t. To do. However, the framework of the present invention can target any number N of two or more dialog participants, any facial expression category, and any number of categories.

  Next, the input unit 1 will be described. As described above, the input unit 1 uses the CCD camera or the like as an image capturing unit to capture the face images of each participant. As a main camera arrangement method, there are a method in which one omnidirectional camera is arranged in the vicinity of the center between dialogue participants, and a method in which one camera is arranged in the front direction of each dialogue participant.

FIGS. 2A and 2B are diagrams illustrating an example of a shooting scene and an example of an input image obtained by shooting the shooting scene with an omnidirectional camera. In the framework of the present invention, any arrangement method can be used as long as it can capture the faces of all the participants, but in this embodiment, as shown in FIG. and placing an omnidirectional camera in the center of ₁ to P _4. This is because, when an omnidirectional camera is used, it is possible to participate in the dialogue for specifying the expression target person of the facial expression required in the present invention without obtaining external parameters (translation and rotation with respect to the world coordinate system) of the camera. This is because the positional relationship between the persons can be easily grasped from the positional relationship in the image of the target person regardless of the number of participants in the dialogue, which is advantageous. If you place an omnidirectional camera in the center of conversation participants P ₁ to P _4, as shown in FIG. 2 (b), each conversation participant P ₁ to P ₄ are photographed.

  On the other hand, with the method using multiple cameras, it is easy to increase the resolution of each participant's face, but in addition to the complexity of the hardware configuration, the number of participants and the seat arrangement differ for each dialogue. Therefore, it is necessary to perform calibration for obtaining external parameters of each camera each time.

Next, the facial expression estimation unit 3 will be described.
FIG. 3 is a block diagram showing a specific configuration of the facial expression estimation unit 3. In the figure, the facial expression estimation unit 3 includes a template creation unit 31 and a facial expression category and expression target estimation unit 32. The template creation unit 31 specializes in a target person from a data set (learning image) 10a in which a face-upright face image of the facial expression category to be recognized by the target person and a label of the facial expression category given in advance are paired. A face template 12 is created. The expression category and expression target estimation unit 32 estimates the expression category and the expression target in each frame in the input moving image 10b of the object person based on the template 12 of the object person, and the expression category and the expression object thereof. The estimated value 11 is output.

Next, the expression category and expression target estimation unit 32 will be described.
FIG. 4 is a block diagram illustrating a specific configuration of the facial expression category and expression target estimation unit 32. In the figure, a facial expression category and expression target estimation unit 32 includes a facial expression category and head posture estimation unit 321 and a facial expression expression target estimation unit 322. The facial expression category and head posture estimation unit 321 estimates the facial expression category in each frame in the input moving image for one conversation participant and outputs it as the facial expression category estimate 11a, and estimates the head posture. Output as head posture estimated value 13. The facial expression expression target estimation unit 322 estimates a facial expression expression target from the head posture estimation value 13 estimated by the facial expression category and head posture estimation unit 321 and outputs the facial expression expression target estimation value 11b.

Next, the facial expression category and head posture estimation unit 321 will be described.
FIG. 5 is a block diagram illustrating a specific configuration of the facial expression category and head posture estimation unit 321. In the figure, a facial expression category and head posture estimation unit 321 estimates the facial expression category and head posture of the target person in each frame of the input moving image 10b based on their likelihoods.

Here, FIG. 6 is a conceptual diagram illustrating the relationship between the expression category e _t , the head posture h _t , and the image z _t . It is assumed that the facial expression category and head posture at each time are conditionally independent under the given image, and that the facial expression category and head posture follow independent Markov processes. ing. The head posture h has 6 degrees of freedom. In the present embodiment, the face center position (three-dimensional) and a three-dimensional rotation angle (roll, pitch, yaw) are used.

The head posture estimated value ^ h _t and the expression category estimated value ^ e _t at time _t are calculated based on the simultaneous posterior probability density distribution p (h _t , e _t | z _{1: t} ) at the same time. The head pose estimate is calculated as the peripheral probability for the head pose of the simultaneous posterior probability density distribution, and the facial expression category estimate is calculated as the category that maximizes the peripheral probability for each facial expression category in the simultaneous posterior probability density distribution. Is done. The head posture estimated value ^ h _t and the expression category estimated value ^ e _t are expressed by Equations (1) and (2).

  Further, the simultaneous posterior probability density distribution can be decomposed as shown in the following equation (3) using Bayes rule.

Here, p (z | h, e) is a simultaneous likelihood for the observed image z of the facial expression category e and the head posture h. Further, the probability density distribution p (h _t , e _t | z _{1: t−1} ) represents a predicted distribution at time t. Furthermore, the simultaneous likelihood is expressed as the product of the likelihood of the head posture with respect to the observed image and the likelihood with respect to the observation image of the facial expression category given the head posture, that is, the following equation (4): The

  These likelihoods are calculated based on the template. Details of the likelihood will be described in detail in a template described later. The posterior probability density distribution is normalized so as to satisfy the following equation (5).

  When the predicted distribution is conditioned on the state of the previous facial expression and head posture, it is converted into a sequential expression such as the following expression (6).

Here, p (h _t | z _{1: t} ) represents the posterior probability density distribution of the head posture when all the observations up to the current time t are given. The probability matrices p (h _t | h _t−1 ) and P (e _t | e _t−1 ) are the transition probabilities for each of the facial expression and the head posture, that is, the facial expression and the head between two consecutive times. This shows how the posture changes with probability.

  In this embodiment, it is assumed that the probabilities of transition between times are not obtained before for both facial expressions and head postures, and for facial expression categories, the probabilities of transition between all two facial expressions are equal, and for head postures Let's follow the random walk model. However, any transition probability can be used in the framework of the present invention. For example, with respect to the head posture, it is possible to use a human head posture motion model acquired in advance by using a constant velocity motion, a constant acceleration motion, or a magnetic sensor. For facial expression categories, the probability that the same facial expression lasts is assumed to be another α (α> 1), or the probability of transition of facial expression categories during actual dialogue obtained from labels attached manually. Can be considered.

  The predicted distribution of Equation (6) cannot be calculated strictly as a closed form. This is because the effect of the head posture on the image is due to non-linear transformation called camera projection (in this case, central projection), and a part of the face that occurs when the face is turned sideways largely. This is because an effect such as shielding is also added. Therefore, it is necessary to use an approximate solution.

  In the present embodiment, a method of performing sequential sampling in the time direction, which is called a particle filter or a sequential Monte Carlo method, is used. More specifically, as shown in FIG. 5, the posterior probability density distribution is estimated by the particle filter 3211, and the facial expression category estimated value and the head are estimated by the estimated value calculation unit 3212 based on the posterior probability density distribution. The estimated posture value 11 is calculated. At this time, the state held by each sample (particle) is a total of seven variables of a six-dimensional head posture h and a facial expression category e. For details on particle filters, see Reference 5 (M. Isard, and A. Blake: Condensation-conditional density propagation for visual tracking, International Journal of Computer Vision}, 29 (1): 5--28, 1998.) It is written in.

Next, the template creation unit 31 will be described.
The template M is for defining the likelihood p (z | h, e) in the mathematical formula (3). The template M is composed of five elements of three types: M = {S, Q ^E , Q ^H , I ^E , I ^H }. Here, S, Q, and I are a rigid face shape model, an attention point set, and an attention point luminance distribution model, respectively. Subscripts E and H represent facial expressions and head postures, respectively. Hereinafter, the model for the expression or the head posture is referred to as a target state, and is represented by * ∈ {E, H}.

  First, the face shape model S is for giving depth information of the attention point defined on the learning image. Thereby, the three-dimensional coordinates (image coordinates + depth coordinates) of the attention point are aligned, and the three-dimensional position of each attention point in various three-dimensional head postures can be calculated.

Next, the attention point set Q ^* indicates the attention position on the face to be considered in the likelihood calculation. These attention points are sparsely arranged on the learning image. In particular, attention point Q ^E of facial expression is constituted by a set of point of interest that specializes in the detection of a specific facial expression category. The attention point set is expressed by the following equation (7).

Here, q _k is the position on the learning image of the kth point of interest. This will be described later. N ^* _k is the number of points of interest for the target state *.

Finally, the luminance distribution model I ^* models how the luminance of each attention point k changes in each facial expression, and is expressed by the following equation (8).

In this embodiment, the attention point set Q ^* and the luminance distribution model I ^* are personalized. That is, these models are created from learning images of the target person. Therefore, even if the target person changes the head posture greatly and the appearance of the entire face changes greatly, it is possible to accurately identify facial expressions with small differences in appearance such as smile and ridicule. On the other hand, the face shape model S is a general-purpose model.

The learning image set Y includes a plurality of images for each target facial expression category to which facial expression categories are assigned. Here, it is expressed as Y = {Y ₀ , Y _{e = 1} ,..., Y _{e = Ne} }. Here, Y ₀ is a single expressionless facial image, and Y _e is a plurality of images to which a label of expression category e (> 0) is assigned. Learning image Y ₀ of expressionless is a model created for the head pose estimation, other learning image Y _e, is for facial expression recognition. Note that each learning image includes a front-right face of the target person, and the positions of those faces are the same between the images. Such a learning image creation method will be described in a learning image creation unit 2 described later.

Note that in the framework of the present invention, when it is difficult to prepare learning data for each individual in advance, instead of creating a personalized model, a general-purpose attention point set Q ^* and luminance A distribution model I ^* may be prepared. At this time, it is necessary to prepare learning data for a large number of people in advance. Then, an average face image for each facial expression and a dispersion image thereof are created therefrom, and this is regarded as a learning image set Y, and a model may be created by a method similar to a model for personalization. However, when such a general-purpose model is used, there is a high possibility that the recognition accuracy will be lower than in the case of a personalized model. Conversely, as for the face shape model S, if there is data obtained by measuring the face shape of the target person in advance with a range scanner or the like, it may be used.

<Creation of face shape model S>
FIG. 7 is a block diagram showing the configuration of the shape model creation unit 5.
In the present embodiment, the average shape of the plurality of persons face shape, a material obtained by fitting against learning image Y ₀ of one expressionless of the target person, for use as the face shape model S. Specifically, the shape model creation unit 5 shown in FIG. 7 scales the average shape vertically and horizontally so as to fit the face size of the learning image (for shape model creation) 10c. The shape model 14 is created by scaling in the depth direction using the square root of the sum of squares as a coefficient. However, in the framework of the present invention, any rigid body shape can be used as a shape model. For example, if there is data obtained by measuring the face shape of the target person with a range scanner or the like in advance, it may be used.

<Creation of facial expression recognition models (Q ^E , I ^E )>
Target point set Q ^E for facial expression recognition, in particular facial expression category, brightness is defined in a region characteristic. Point of interest is, for expressionless each facial part of the learning image Y ₀ of, is registered by 1 point for each of the target facial expression category. In the present embodiment, the facial parts are four regions and six regions in total: eyebrows (left / right), eyes (left / right), nose, and mouth. The number of attention points to be selected for each facial expression is arbitrary, but affects the processing speed and the recognition accuracy. In this embodiment, there are a total of 68 points (eyebrow: 12 × 2, eyes: 6 × 2, nose: 8, and mouth: 24). The total number of attention points for facial expression recognition at this time is N ^E _k = 68 × 4 = 272.

  FIG. 8 is a schematic diagram showing an example of a set of attention points registered for each facial part. The attention point set here is different for each individual. This is because the movement and deformation due to the face part arrangement, the change in its appearance, and the change in facial expression are different for each individual. In addition, although the detection method of a facial component is also arbitrary, in this embodiment, a cascade type AdaBoost detector (P. Viola and M. Jones: Rapid object detection using a boosted cascade of simple features, In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 511--518, 2001).

The attention point luminance distribution model ^IE for facial expression recognition is a model of how the luminance of each attention point changes for each expression. The present invention uses this characteristic to recognize the facial expression category at that time from the luminance of the point of interest in the observed image. However, due to errors in the face shape model, it is difficult to accurately calculate the position of the point of interest, and even if the lighting in the shooting environment is constant in time, As the direction of illumination with respect to the face changes, the brightness of the observed point of interest varies. These variations may be modeled in any way, but in this embodiment, they are modeled by a normal distribution. The luminance distribution model I ^E _k for the _kth attention point is expressed as the following equation (9).

Here, μ _k (e) and σ ² _k (e) represent an average and a variance in the facial expression category e of the k-th attention point, respectively. The average and variance are values at the position q ^E _k of the luminance average map and luminance variance map calculated from the learning image Y _e .

FIG. 9 is a block diagram showing a flow of a series of processes of luminance average map, luminance dispersion map, attention point selection, and luminance distribution model creation. The luminance average map and luminance variance map creating unit 6 creates a luminance variance map 15 and a luminance average map 16 from the learning image 10d (Y _e ). The saliency calculation unit 7 calculates a saliency map 17 from the luminance dispersion map 15. The point-of-interest selection unit 8 selects a point-of-interest set 18 that is useful for identifying the target facial expression category from within the face part region, according to the saliency map 17. Further, the luminance distribution model creation unit 9 creates a luminance distribution model 19 indicating the luminance change of the attention point set 18 with respect to the facial expression and the head posture from the luminance average map 16 and the attention point set 18.

  FIGS. 10A to 10C are schematic diagrams showing an example of the luminance average map 16 for smile, ridicule and others. FIGS. 11A to 11C are schematic diagrams showing an example of the luminance dispersion map 15 for smile, ridicule and others. The luminance average map 16 and the luminance variance map 15 are an image obtained by averaging the learning images of each expression category and an image of the variance. This will be described in detail below.

<Selecting the point of interest>
As shown in the following expression (10), the saliency of the luminance of the image coordinate x with respect to the target facial expression e is the ratio of the variance in the facial expression category e of luminance (intra-category variance) and the variance between categories of the luminance.

Here, σ ² _{x, W (e)} represents the variance in luminance at the position x of the facial expression category e, and σ ² _{x, B (e)} is the luminance between the facial expression category e and the other facial expression categories. Represents the variance of. An example of facial expression saliency is shown in FIG. FIGS. 12A to 12C show facial expression saliency maps for smile, ridicule and others, respectively.

The attention point for each target facial expression e is selected as follows.
(1) First, the saliency S _{x, e} for the expression category e for all the positions x on the face is calculated from the learning image set Y _e for the expression category e.
(2) Next, a position having the highest saliency S _{x, e} (referred to as x ′) is selected as a point of interest.
(3) The saliency of the selected position x ′ is set to 0, and the saliency of the surrounding points is also reduced based on the distance d from the position x ′. Here, S _{x, e} ← S _{x, e} −α · exp (−d ² ).
(4) If the number of selected attention points has not reached the specified number, the process returns to the above (2), and further attention points are selected. Note that, due to the above (3), attention points are arranged as widely as possible in consideration of saliency in each facial expression.

<Likelihood of facial expression category>
The likelihood when the facial expression category and head posture estimation unit 321 estimates the facial expression category of the target person in each frame of the input moving image will be described. The likelihood p (h, z | e _t ) of the facial expression category e for the observed image z given the head posture h is calculated using the template created by the template creation unit 31.

  Assuming that the brightness of the sparsely arranged attention points is independent of each other, the likelihood for the facial expression category is decomposed into products of likelihoods for each attention point as shown in the following equation (11). The

Here, z _k (h) represents the luminance at the image coordinates of the kth feature point in the head posture h, as shown by the following equations (12) and (13).

Here, the function ρ (•) represents a robust function, and the likelihood p is determined by observation noise included in a small number of attention points (image noise, shift in the calculation position of the attention point due to the shape model error, etc.). (z _k | e) is intended to alleviate the decrease in the overall size. In this embodiment, the Geman-McClure function ρ (x) = c · x ² / (1 + x ² ) is used as the robust function ρ, and the scaling coefficient is c (= 9).

The luminance z _{i, t} is the luminance in the image z _t at the image coordinates in the head posture h _t of the k-th attention point. The image coordinates are calculated as follows.

(1) First, the target point defined on the learning image is projected onto the shape model S by parallel projection in a state in which the front erect direction of the shape model S is aligned with the front erect direction in the image (paste). wear).
(2) Next, the shape model is isotropically scaled so that the product of the height and width of the face of the shape model S is equal to that of the original original face shape (here, the average face shape). .
(3) Subsequently, the translation of the shape model S accordance head posture h _t, and rotate.
(4) Finally, the i-th target point on the shape model S is projected onto the observation image by the central projection method. Note that if the processes (1) and (2) are performed once at the time when the template is created, the results may be used as needed.

<Model and likelihood for head posture estimation>
Next, the likelihood when the facial expression category and head posture estimation unit 321 estimates the head posture of the target person in each frame of the input moving image will be described.

  The luminance distribution model 19 for estimating the head posture models how the luminance of the point of interest changes according to the head posture. Two methods are conceivable as a method of learning this in advance. One is a learning image with various head postures, and can be created if the exact position of the point of interest in each image is given.

For example, the head posture may be measured using a magnetic sensor or the like in combination with the face shape measured by a range sensor or the like and the image. Alternatively, if the lighting environment, the face shape model of the target person, and the reflection characteristics thereof are known, by performing rendering using computer graphics technology, the brightness of the attention point in an arbitrary head posture is obtained, Based on this, it is possible to calculate a luminance distribution model at the point of interest. As for the shape and the reflection characteristic, an approximate model can be created using the average shape of the person and the reflection characteristic. If any of these methods can be selected, the luminance distribution model I ^H _k for head posture estimation can be expressed by using a normal distribution or the like in the same manner as the luminance distribution model I ^E _k for facial expression recognition. That's fine.

However, there are many cases where the above method cannot be selected. In this case, the calculating the likelihood based on the distance between the luminance that is observed as luminance in the learning image Y ₀ expressionless point of interest Q ^H. This method is not much different from the framework of a head tracking method based on general template matching. Therefore, in this embodiment, Reference 6 (D. Mikami, K. Otsuka, and J. Yamato: Memory-based particle filter for face pose tracking robust under complex dynamics, In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 999--1006, 2009.), define a template for head posture estimation.

As target point Q ^H for head pose estimation, dipole across the luminance of the edge of the face image (pairs of two points) is disposed sparsely on the face. As a selection method, in order from the largest difference in luminance between two points in the dipole, the target dipole centers are selected one by one away from the already selected dipole center by a threshold or more. The dipole selected in this way is arranged over a wide range on the face. In the present embodiment, the number of attention points N ^H _k for head position estimation (a value twice the number of dipoles) is empirically set to 256.

For the luminance distribution I ^H of these attention points, the average μ _{k, 0} is the luminance at the coordinate q _k of the expressionless learning image Y ₀ , and its standard deviation is proportional to the average, that is, σ _{k, 0} ∝ It is assumed that μ _{k, 0} . Note that the proportionality constant at this time need not be explicitly defined. This is because it can be considered to be included in the normalization constant in Equation (3).

  About the likelihood about a head posture, it is set as the following formula (14) according to the above-mentioned reference 6.

  Here, the scaling coefficient c in the robust function ρ is set to 1.

<Responding to changes in facial brightness due to external factors>
In general, the brightness on the face may change due to factors other than the change in facial expression. Two typical factors are non-uniformity of the lighting environment with respect to the face direction, and changes in the brightness of the lighting itself. As described above, in the present embodiment, changes in facial brightness due to these external factors are not dealt with explicitly, but in the framework of the present invention, these can be dealt with explicitly.

  Explaining these two method examples, first, the change in luminance on the face is uniform over the entire face (for example, Reference 6), or uniform within a small area of the face (for example, patent document). 1) The assumption is made. Then, in the simultaneous estimation of the facial expression and the head posture, the luminance change coefficient is regarded as the same random variable as that and is added to the state held by the sample in the particle filter (for example, the head posture is estimated). However, the estimation is performed only in the reference literature 6) or deterministically in the framework of the maximum likelihood method (for example, Patent Document 1).

Next, the learning image creation unit 2 will be described.
The learning image includes a single expressionless face image Y ₀ of the target person and a plurality of images Y _e for each target expression category e having the same face position and orientation as the face image Y ₀ . Become. The learning image creation unit 2 prepares learning images as follows.

First, the face image Y ₀ expressionless, among moving images of facial expression category to be recognized, the target person is expressionless and shall chose any one frame is a front erect. Even if such a frame is not included in the input moving image, it is sufficient if there is an image of the target person photographed at the same time in the same lighting environment as the target moving image. Next, a shape model and a head posture estimation model, that is, {S, Q ^H , I ^H } are created from the expressionless learning image Y ₀ .

Subsequently, with respect to the target moving image, the head posture is estimated using only those models (not including the expression recognition model). This method is basically the same as the method of Reference 6 which is an existing head posture estimation method. Then, a plurality of frames for which the facial expression of the target person is a recognition target category are selected for each category from the input moving image. Finally, learning about the facial expression category, using the estimated head posture and shape model S for those frames, and back-projecting the target person's face in each frame into a face-upright face image _Let it be image Ye. Note that the size of a face-up face image created by back projection can be freely determined. It may be about 150 × 200 [pixel].

Next, the expression expression target estimation unit 322 will be described.
It is assumed that the expression object g in the expression category is equal to the gaze object. There are two main methods for estimating the gaze target. One is a method for directly estimating the gaze direction (for example, Reference 7: Daijo Yamazoe, Akira Utsumi, Shinji Abe, Monocular gaze estimation by tracking facial feature points, Journal of the Institute of Image Information and Television Engineers, Vol.61, No.12. , pp.1750-1755, 2007.). The other is a method in which the gaze direction is not estimated directly, but is estimated based on the head posture that has a high correlation with the gaze direction. In the framework of the present invention, both methods can be applied, but in this embodiment, the head posture of each conversation participant is obtained at the same time as the facial expression category in the facial expression category and head posture estimation unit 321. The gaze target is estimated based on the estimated value of the head posture. In addition, as a conventional study on the relationship between gaze direction and head posture during conversation, Reference 8 (R. Stiefelhagen: Tracking focus of attention in meetings, In Proc. ICMI, pages 273--280, 2002.).

  In the following, a method for specifying the expression target person from the head posture will be described. In this embodiment, it is assumed that the distribution of the head posture of the target person with respect to the conversation participant to be watched follows a normal distribution, and the head posture output from the facial expression category and head posture estimation unit 321 is used as an input. Estimate the facial expression subject using the likelihood estimation framework. This is expressed by Equation (15).

Here, {circumflex over (g ₎ } _i = j means that the facial expression of the conversation participant P _i is estimated to be directed toward the conversation participant P _j . The angle h ^HOR _i is a horizontal angle component of the head posture of the conversation participant P _i output from the facial expression category and head posture estimation unit 321, and φ _{i, j} is viewed from the conversation participant P _i. and it represents the relative azimuth of the conversation participants P _j. This can also be calculated from the dimension positions of the dialogue participant P _i and the dialogue participant P _j output from the expression category and head posture estimation unit 321. σ ² and κ represent the variance and the scaling factor, respectively.

It should be noted that the head posture when the expression of each person's facial expression does not belong to any of the other participants, that is, when the gaze target is deviated from any other participant, is uniform. Expressed by distribution. That is, if the probability of the estimated ^ g _i is equal to or less than the threshold value in Equation (15), fixation target at that time shall not belong to anyone other interactive participants.

Next, the inter-person emotion network creation unit 4 will be described.
Emotions between persons (interpersonal feelings) are estimated based on how many facial expressions each participant has expressed with respect to other participants.

The inter-person emotion network represents emotions among participants in the dialogue. There are various measures of emotion such as favorable-hostile and the degree of cooperation. As a criterion for estimating a favorable-hostile emotion network, there may be a favorable expression between dialogue participants, an amount of expression of a hostile expression, a degree of co-occurrence of expressions, and the like. First, a method for defining an emotion based on the amount of expression of a favorable-hostile expression will be described using favorable-hostile as a measure of emotion between persons. The expression amount of expression relative to interact participant P _j conversation participant P _i, if the expression of the expression time from normalized conversation participant P _i in an interactive length to interact participant P _j, conversation participants emotion r _{i → j} for dialogue participant _{P j} of P _i is represented by the following equation (16).

Here, w (e) is a signed weight function, and is 1 when the expression category e is a favorable category, -1 when it is a hostile category, and a neutral category. Returns 0. 1 (•) is a function that returns 1 if the condition in parentheses is satisfied, and returns 0 otherwise. T represents the total number of frames of the target moving image. A structure in which the estimated interpersonal emotions r _{i → j} are gathered for the entire conversation participants is called an interpersonal emotion network.

  It should be noted that here, the output magnitude (absolute value) of w (•) is 0 or 1, that is, the contributions of all facial expressions to the emotion between persons are treated with equal weight. However, the output size of w (•) may be changed for each category. For example, the weight of a category may be increased if it is considered that even a short time expression such as anger greatly affects the feelings between persons. In addition, even in the same favorable category such as smile and ridicule, it is conceivable that the weight of smile is increased and the weight of laughter is decreased for a category having a different expression directivity.

  A scale other than the amount of expression of facial expressions can be used as a scale of emotions between other persons. For example, when co-occurrence of a facial expression that is considered to be an index with good closeness is used, the following equation (17) is used.

Here, E ⁺ indicates a set of facial expression categories that contribute positively to closeness such as smiles. The output of the function w (e) at this time is positive for facial expressions belonging to E ⁺ , negative for facial expression categories that contribute negatively to intimacy such as laughter and anger, and zero for other facial expressions. In this case, when the dialogue participants P _i is exposed to a positively contribute facial expression category facial expressions that belong to E ⁺ to the intimacy degree to the conversation participant P _j, interactive participant P _j also interactive participant P _i The longer the time for expressing the facial expression belonging to E ⁺ , the larger the value of r _{i → j} , and the dialogue participant P _j expresses the facial expression that does not belong to E ⁺ to the dialogue participant P _i . The value of r _{i → j} becomes smaller as the conversation time is longer and as the conversation participant P _j does not point the gaze target in the direction of the conversation participant P _i .

<Graphic representation of emotion network between people>
FIG. 13 is a conceptual diagram showing an example in which an inter-person emotion network is expressed as a graph structure. In this graph representation, the participant is represented by a circular node, and the directional link (arrow) between the two represents the emotion r _{i → j} between the persons. Here, the emotion r between persons is treated in a one-dimensional manner of neutral / favorable / hostile, and this is expressed by the density (or color) of the link. When neutral (r≈ ± 0), it is gray, as friendly (r >> 0) is closer to white, and as hostile (r << 0) is closer to black Yes. The link thickness (line width) is proportional to the amount of expression of the facial expression in the target person direction. If the dimension number of emotion r is 3 or less, the emotional state can be similarly expressed by assigning each component of r to each component of RGB.

The size of the node of the dialogue participant P _i is proportional to the amount of expression (number of frames) of the friendly and hostile facial expression category, and the shade (or color) of the node represents the emotion from other dialogue participants. In accordance with the sum of r, that is, ri _{→ j} , it is expressed in the same manner as the shade (or color) of the link. However, there are other indicators used as a reference for the size and shading (or color) of this node, such as the distribution among the participants in the facial expression expression and the dialogue between the participants in the facial expression expression described later. Various things can be considered, such as dispersion of.

  Although this interpersonal emotion network is a compact expression, it suggests various states of dialogue and relationships among participants in the dialogue. An example is given below.

(1) Type of dialogue It can be estimated to some extent whether the target dialogue is a discussion on an agenda with different opinions or a dialogue with no particularly big opinion conflict such as daily conversation. In the former case, favorable emotions and hostile emotions are mixed. In other words, the distribution of r _{i → j} with respect to i and j is large with the feeling r as a favorable-hostile feeling as a scale. In the graph display method shown in FIG. 13, links of various colors appear from near black to near white. In the latter case, mainly positive feelings appear. That is, the variance of r _{i → j} for i and j is small and the average is large. In the graph display method shown in FIG. 13, a color link mainly between gray and white appears.

(2) Exciting degree of dialogue When the emotion r is the degree of co-occurrence of facial expressions, if the variance for many combinations of i and j in r _{i → j} is small and the average is large, the dialogue is exciting. It is suggested. In addition, with respect to the dialogue participant P _i and the dialogue participant P _j , if both r _{i → j} and r _{j → i} are large, it is suggested that at least the two parties were excited.

(3) Human relationships You can see who and who are in a positive relationship, and conversely who and who are in a conflicting relationship. For example, if the relationship between the dialogue participant P _i and the dialogue participant P _j is r _{i → j} > τ∩r _{j → i} > τ (τ> 0), it is favorable, and r _{i → j} <−τ∩r. It is hostile if _{j → i} <−τ, neutral if | r _{i → j} | <τ∩ | r _{j → i} | <τ, otherwise one-sided favor. Can be considered.

(4) Grouping It is possible to divide the participants in the dialogue into groups in which the people who have a favorable relationship are put together. For example, it can be considered that a group connected by a link whose emotions between the two parties are min {r _{i → j} , r _{j → i} }> τ ′ is regarded as one group. In the example illustrated in FIG. 13, for example, there are a group composed of the dialogue participant P ₁ and the dialogue participant P ₂ and a group composed of the dialogue participant P ₃ and the dialogue participant P ₄ .

(5) interactive participant except for the character dialogue participants _{P i} of individuals, the sum of the emotions you are holding to the conversation participant _{P i r IN (i) =} Σ j (≠ i) and r _{j → i} And the sum of emotions of all other dialogue participants held by the dialogue participant P _i is represented as r _OUT (i) = Σ _{j (≠ i)} r _{i → j} . If the value of r _OUT (i) is large (small) and the variance of r _{i → j} with respect to j is small, is the dialogue participant P _i trying to keep the flow of dialogue well as a whole? Do you think it is boring), or do you like other dialogue participants favorably (hostile) and vice versa? It is implied. If the distribution of r _{i → j} with respect to j is large, it implies that the dialogue participant P _{i considers} some dialogue participants particularly friendly (hostile). When r _OUT (i) is near 0 and the variance of r _{i → j} with respect to j is small, the dialogue participant P _i is not interested in all other dialogue participants or the topic itself. Imply.

  In addition, as described above as (1) to (5), the relationship between emotions between a plurality of persons that is graphically output with an emotion network between persons, that is, a graph structure as shown in FIG. An estimation unit that estimates at least one of the type of dialogue, the degree of excitement of dialogue, the relationship between humans, the structure of subgroups in a favorable relationship, and the personality of an individual may be provided.

According to the above-described embodiment, the following remarkable effects that cannot be expected can be expected.
(1) Visually similar facial expression categories can be recognized with high accuracy in real time for a person who is interacting with changing head posture over a wide range.
(2) A personalized template effective for identifying visually similar facial expression categories can be created from a small number of learning images.
(3) It is possible to recognize which dialogue participant has displayed what kind of facial expression to whom and when.
(4) It is possible to estimate feelings from one dialogue participant to another dialogue participant, such as favorable-hostile or cooperative.
(5) The structure of the emotion network between persons can be immediately displayed as a graph that can be intuitively grasped.
(6) Based on the emotional network between people, various types of dialogues such as the type of dialogue, the degree of excitement of dialogue, subgroups in human relations and favorable relationships, and personality, and relationships between people are estimated. can do.

1 Input unit 2-1 to 2-N Learning image creation unit 3-1 to 3-N Facial expression estimation unit 4 Interpersonal emotion network creation unit 31 Template creation unit 32 Facial expression category and expression target estimation unit 321 Facial expression category and head Posture estimation unit 322 Expression expression target estimation unit 3211 Particle filter 3212 Estimated value calculation unit 5 Shape model creation unit 6 Luminance average map and luminance variance map creation unit 7 Saliency calculation unit 8 Attention point selection unit 9 Luminance distribution model creation unit

Claims (10)

An input means for photographing a person and inputting it as an input moving image;
The head posture of the person is estimated based on the likelihood of the head posture with respect to the input moving image input by the input means, and is output as a head posture estimated value, and the facial expression for the input image input by the input means A facial expression recognition device comprising: facial expression estimation means for estimating the display category of the person based on the likelihood of the category and outputting the estimated display category value. The facial expression estimation means includes
A first likelihood set based on a set of attention points that are predetermined for the recognition of the facial expression category and arranged in a predetermined region of the input moving image and its luminance distribution model. Calculate
A second likelihood based on a set of attention points that are set in advance for estimating the head posture and are arranged in a predetermined region of the input moving image and its luminance distribution model To calculate
A product of the first likelihood and the second likelihood is calculated as a simultaneous likelihood of the facial expression category and the head posture, and based on the calculated simultaneous likelihood, the facial expression category estimated value and The facial expression recognition device according to claim 1, wherein the head posture estimation value is estimated and output. The facial expression estimation means includes
The point of interest set predetermined for recognizing the facial expression category is placed in a region in the input moving image where a change in the facial expression of the person is significant and predetermined for the person. The facial expression recognition device according to claim 2. An input means for inputting an input moving image obtained by photographing a plurality of persons;
Based on the likelihood of the head posture of each person with respect to the input moving image input by the input means, the head posture of each person is estimated and output as an estimated value of the head posture of each person. Based on the likelihood of each person's facial expression category with respect to the input image, the facial expression category and the head posture estimation means for estimating each person's display category and outputting each person's display category estimation value;
The human emotion is characterized by comprising: an expression target estimation means for estimating the expression object of each person's facial expression based on the estimated head posture value of each person and outputting the expression object estimation value. Estimating device.   5. The inter-person emotion network creating means for graphically outputting an emotion relationship between the plurality of persons based on the expression category estimated value and the expression target estimated value. The human emotion estimation apparatus described.   Based on the emotional relationship between the plurality of persons that is graphically output by the interpersonal emotion network creation means, the type of dialogue, the degree of excitement of the dialogue, the relationship between humans, the subgroups that are favorably related, the individual 6. The inter-person emotion estimation apparatus according to claim 5, further comprising an estimation unit that estimates at least one of the personality of the person. A step in which an input means photographs a person and inputs it as an input moving image;
The facial expression estimating means estimates the head posture of the person based on the likelihood of the head posture with respect to the input moving image input by the input means, and outputs the estimated head posture as a head posture estimated value. And a step of estimating the display category of the person based on the likelihood of the facial expression category with respect to the input image and outputting the estimated display category as a display category estimation value. An input means for inputting an input moving image obtained by photographing a plurality of persons;
The facial expression estimation means estimates the head posture of each person based on the likelihood of the head posture of each person with respect to the input moving image input by the input means, and outputs the estimated head posture of each person. Estimating each person's display category based on the likelihood of each person's facial expression category with respect to the input image input by the input means, and outputting each person's display category estimated value;
And a step of estimating the expression target of the facial expression of each person based on the estimated head posture value of each person, and outputting the estimation target estimation value. Emotion estimation method between people. In the computer of the facial expression recognition device that recognizes the facial expression of a person based on image processing,
An input function to capture a person and input it as an input video
Based on the likelihood of the head posture with respect to the input moving image, the head posture of the person is estimated and output as a head posture estimated value, and based on the likelihood of the facial expression category with respect to the input image, the person This program executes a facial expression estimation function that estimates a display category and outputs it as a display category estimate. Recognizing facial expressions of a plurality of people based on image processing, and estimating a feeling between the plurality of people in a computer of the emotion estimation device between people,
An input function for inputting input moving images obtained by shooting multiple people,
Based on the likelihood of the head posture of each person with respect to the input moving image, the head posture of each person is estimated and output as an estimated value of the head posture of each person, and the facial expression category of each person with respect to the input image A facial expression category and a head posture estimation function to estimate the display category of each person based on the likelihood of and output as a display category estimation value of each person,
A program for executing an expression target estimation function for estimating an expression target of each person's facial expression based on the estimated head posture value of each person and outputting the estimated expression target value.