WO2020175969A1

WO2020175969A1 - Emotion recognition apparatus and emotion recognition method

Info

Publication number: WO2020175969A1
Application number: PCT/KR2020/002928
Authority: WO
Inventors: 류숙희; 이재익; 이찬수
Original assignee: 주식회사 하가
Priority date: 2019-02-28
Filing date: 2020-02-28
Publication date: 2020-09-03

Abstract

A recognition apparatus is provided. The recognition apparatus may comprise: a capturing unit for capturing an image of a counterpart facing a user; an emotion recognition unit for recognizing an external emotion of the counterpart through analysis of the counterpart's image captured by the capturing unit; and a display unit for displaying the external emotion by voice or vibration.

Description

2020/175969 1»（：1/10公020/002928 Specification

Title of Invention: Emotion Recognition Device and Emotion Recognition Method Technical Field

[1] The present invention relates to a device and a method for recognizing the emotions of the other party facing the user.

Background

[2] The number of people who do not have emotional communication skills in human relationships, including those with low vision, is increasing. Many people do not have emotional communication skills due to congenital disabilities, acquired diseases, aging, or lack of communication education. Causes pain in friends, relatives, and colleagues.

[3] The inability to read emotions can lead to isolation in human relations. Congenital and acquired problems are increasing the number of people who have difficulty reading, understanding, and concentrating on the other person's emotions. The elderly and suffering from depression There is a growing trend of people with whom they are isolated from human relations due to the problem of not being able to read the other's emotions in their expressions.

[4] Expensive devices that assist with vision impairment do not include technology to support communication skills. Expensive products amount to 3,500,000 won. The actual supply rate is less than 5%.

[5] Attempts to use face recognition, expression recognition, and motion recognition technologies in various fields at home and abroad are increasing, and related markets are also expanding.

[6] Specifically, research on biometrics technology began in the early 2000s, centering on T-company and university research centers related to facial recognition and expression recognition technology. As a leading research institute, the research team at Carnegie Mellon University, MIT in the US There are Media Lab, INRIA in Europe, ATR Lab in Japan, etc.

Visionics, Viisage, Miros, etc. are leading the market. These are not only face detection technology, lighting recognition technology, pose recognition technology, expression recognition technology, which are the core technologies of face recognition technology, but also aging recognition technology, large-scale DB construction technology, and 3D. Active research is in progress focusing on facial shape restoration technology.

Detailed description of the invention

Technical task

[7] The present invention is intended to provide a recognition device and recognition method capable of accurately recognizing the emotion of a user facing a user.

Problem solving means

[8] The recognition device of the present invention includes a photographing unit for photographing the other party facing the user; an emotion recognition unit for recognizing the external emotion of the other party through the analysis of the image of the other party photographed by the photographing unit; the appearance of the above by voice or vibration Expressing emotion 2020/175969 1»（：1^1{2020/002928 display part; can be included.

[9] The emotion recognition unit analyzes the image of the other party to

At least one of the gestures can be recognized.

[1] The emotion recognition unit may recognize the external emotion by using the expression or the gesture.

[11] The recognition method of the present invention includes the steps of photographing the surrounding area; storing the photographed image; Analyzing the photographed image and extracting the other party to be conversed with the user; When the other party is extracted, recognizing and correcting the external feeling of the other party; Displaying the corrected external feeling. It may include.

[12] The step of recognizing and correcting the appearance emotion,

[13] Recognizing the external emotion by using the opponent's expression or gesture projected on the photographed image, and by analyzing the previously stored photographed image to recognize a nearby object or recognize a character, corresponding to the recognition result of the object. The appearance emotion can be corrected by using the object information or content information corresponding to the recognition result of the character.

Effects of the Invention

[14] The recognition device and recognition method of the present invention analyzes the other party photographed by the camera,

You can recognize the other's external feelings.

[15] External emotions such as facial expressions and gestures may show different strengths or different types of real emotions depending on the surrounding environment.

[16] The recognition device and recognition method of the present invention recognizes the external feeling of the other party, and can correct the external external feeling of the other party, which is attributed to the object or text, by using the peripheral lens. Considering the Eunju conversion

Correct emotion recognition based on complex emotion recognition technology Can correct the external emotion of the other party.

[17] According to the present invention, the emotion of the other party can be analyzed and recognized in consideration of the surrounding environment. Through the presentation of the complex emotion recognition technology considering the surrounding environment, the present invention can accurately recognize the emotion of the other party.

[18] The emotion of the other party recognized through complex emotion recognition technology can be provided to the user. A user with low vision can accurately recognize the other party's emotions, and thus can communicate in the process of the other party.

[19] According to an embodiment of the present invention, the communication ability of a user with low vision

It can be improved by using complex emotion recognition technology.

[2 This invention is for people with low vision based on complex emotion recognition technology to improve communication skills.

The present invention can aid vision by allowing low vision people to read characters, objects, and expressions that are difficult to distinguish. In addition, the present invention enables users with low vision to understand the surrounding situation and , Be aware of the risk, 2020/175969 1»（：1^1{2020/002928 You can guide your everyday life, such as shopping. This invention can guide users to communicate their emotions normally with the other party.

Brief description of the drawing

1 is a schematic diagram showing the recognition device of the present invention.

2 is a schematic diagram showing an image photographed by a photographing unit.

[23] Fig. 3 is a flow chart showing the recognition method of the present invention.

4 is a diagram showing a computing device according to an embodiment of the present invention.

Modes for the implementation of the invention

[25] Hereinafter, with reference to the accompanying drawings, an embodiment of the present invention will be described in detail so that those with ordinary knowledge in the technical field to which the present invention belongs can easily implement. However, the present invention has several different forms. It may be implemented and is not limited to the embodiments described herein. In order to clearly describe the present invention in the drawings, portions irrelevant to the description are omitted, and similar reference numerals are attached to similar portions throughout the specification.

[26] In this specification, redundant explanations for the same component are omitted.

[27] In addition, in this specification, when an element is referred to as being'connected' or'connected' to another element, it may be directly connected or connected to that other element, but in the middle It should be understood that other components may exist, whereas in this specification, when a component is referred to as being'directly connected' or'directly connected' to another component, no other component is present in the middle. It should be understood as not.

[28] In addition, terms used in the present specification are only used to describe specific embodiments, and are not intended to limit the present invention.

[29] Also, in this specification, a singular expression may include a plural expression, unless the context clearly indicates otherwise.

[3] In this specification, terms such as,'include' or'have' are only intended to designate the existence of features, numbers, steps, actions, components, parts, or combinations of them described in the specification. It should be understood that the presence or addition of any further features, numbers, steps, actions, components, parts, or combinations thereof, does not preclude in advance.

[31] Also in this specification, the terms'and/or' include a combination of a plurality of listed items or any of a plurality of listed items. In this specification, ^ or

have. 2020/175969 1»（：1^1{2020/002928

[34] Emotion recognition technology is a technology that aims to improve the quality of human life by measuring human emotions and analyzing them, applying them to product development or environmental design. This is the comfort of external physical stimulation obtained through personal experiences, It belongs to the field of technology that changes the product or environment by scientifically measuring and analyzing complex emotions such as discomfort, comfort, and discomfort, and applying them engineeringly.

[35] These emotion recognition technologies can be divided into fields such as biometric technology to understand human characteristics, human five sense sensors and emotion processing technology, emotion design technology, micro processing technology, and usability evaluation and virtual reality technology. Recognition technology is being used to develop electronic products and software interfaces that consider human biometric and psychological suitability.

[36] In addition, emotion recognition technology accurately recognizes the emotions of users and

For example, emotion recognition technology can provide emotion-based services to users by using the emotions of users in entertainment, education, and medical fields, and respond immediately to users when using the service. The quality of service can be improved by checking and providing feedback according to the response.

[37] In this emotion recognition technology, as a method of measuring a user's biological response

Autonomic nerve bran | (ANS, Autonomic Nervous System) Immediate method, quartet nerve freckles | (CNS, Central Nervous System) There are EEG methods for immediate improvisation, or methods for photographing a face image (Behavior Response). In addition, in order to increase the accuracy of measuring the biological response, the user's emotions are extracted using only one biological response. Rather than that, there is a multi-modal emotion recognition technology that integrates and analyzes a number of biological responses. In other words, the initial emotion recognition technology is a fragmentary approach through autonomic nervous system measurement, central nervous system measurement, and facial image measurement. Was used, but since then, it has developed into a multimodal system that integrates and analyzes multiple biological reactions.

[38] However, the conventional emotion recognition technology as described above only improves the accuracy of the measured value itself for the biological response, and there is a problem in that it does not reflect the difference in the individual's emotion recognition and surroundings. That is, the emotion recognition technology. In technology, a standardized emotion recognition rule-base that can be applied statistically to all people has been established, and this standardized rule-base has been applied to all individuals.

[39] However, human body reactions and the accompanying emotions do not appear uniformly.

It is different for each individual, so the standardized

There is a problem in that it is not possible to accurately discriminate the difference in individual's emotional perception when determining the biological reaction and the corresponding emotion according to the rule-base.

[4] The recognition device and recognition method of the present invention can be used to accurately grasp the emotion of the other party faced by this user.

[41]

[42] * Fig. 1 is a schematic diagram showing the recognition device of the present invention. Fig. 2 is a photographing unit (1W) 2020/175969 1»（：1^1{2020/002928 This is a schematic diagram showing the captured image.

The recognition device shown in FIG. 1 includes a photographing unit 110, an emotion recognition unit 130, and a display unit 150

Can include

[44] The photographing unit 110 may photograph the user and the other party who speaks. For example,

The photographing unit (0) can photograph the other person's face, arms, legs, and body.

[45] The opponent's face photographed in the photographing unit 110 can be used to analyze and recognize the other's expression. The opponent's arms, legs, and body photographed in the photographing unit (0) analyze and recognize the other's gestures. The photographing unit (0) can contain various types of cameras that generate video images.

[46] The emotion recognition unit 130 may recognize the external emotion of the other party through the analysis of the image of the other party photographed by the photographing unit 0.

[47] The other party's image may refer to an image that includes the other party among the images captured by the photographing unit 110. The emotion recognition unit 130 may extract the other party included in the other party image. The emotion recognition unit 130 Can apply various expression recognition technologies and various gesture recognition technologies to the image of the other party to the extracted information of the other party, and at least one of the other party’s expressions and gestures) can be grasped or recognized.

[48] The emotion recognition unit 130 recognizes the emotion of the identified opponent's expression or gesture

It is possible to apply an algorithm and recognize the other party's emotions. The emotion recognition algorithm can be generated based on deep learning. Using the external appearance of the other party, such as expressions or gestures), the external emotion corresponding to the external emotion expression of the other party can be recognized. have.

[49] Appearance may be a difficult part for low vision users to recognize.

The appearance feeling recognized by the recognition unit 130 may be provided to the user by the display unit 150.

[5 This display unit 150 can display the external feeling of the other party by voice or vibration. The user can sense the voice or vibration of the display unit 150 by holding or wearing the display unit 150. The photographing unit 110 and the emotion recognition unit 130 may also be formed to be possessed or worn by the user. .

[51] As an example, the display unit 150 may include an audio output means such as a speaker that displays a voice such as “the other party is in a joyful state” of the other party’s emotions. For example, the display unit 150 makes an appointment with the user. Vibration means for generating a vibration signal can be included. The vibration means can indicate that the emotion of the other party is in a joyful state through vibrations tailored to the setting routine.

[52] The user can receive the external feeling through the display unit 150 while acquiring the voice of the other party by himself. The user can accurately grasp the other party's feelings through the detection of the external feeling. In addition, the user himself/herself Emotional learning can be performed while matching the external emotion provided by the display unit 150 to the voice of the other party being heard. Through this, the user can learn what voice the other party is talking with. 2020/175969 1»（：1^1{2020/002928

[53] On the other hand, the external sentiment of the other party through expressions or gestures may vary depending on the surrounding environment. For example, in a library where quietness is required, expressions/gestures in which the other party expresses characteristic emotions is a noisy market. It may be different from what is expressed in. For this reason, it may be difficult to accurately recognize the other party's emotions only with the user's external emotions.

[54] Various means for correcting the external feeling can be provided so that the external feeling that closely follows the actual feeling of the other party is derived.

For example, the recognition device of the present invention may be provided with a collection unit 120 for collecting the voice of the other party.

[56] The emotion recognition unit 130 can analyze the voice by using a voice emotion recognition model such as deep learning or a voice characteristic analysis algorithm, and recognize the voice emotion of the other party. The sound emotion is recognized through the voice. It may contain emotions that can be or perceived.

[57] The emotion recognition unit 130 may correct the external emotion by using the recognized sound emotion. The display unit 150 may display the corrected appearance emotion. The display unit 150 may display the corrected appearance emotion using sound emotions.

[58] Correction of appearance emotion can mean correction of intensity for the same kind of emotion.

[59] In order to display the intensity of emotion, a plurality of display information may be provided on the display unit 150 by dividing the intensity of emotion for the same type of external emotion. For example, even if it is a feeling of joy, a little joy can be provided. It is advantageous that the intensity is included as, normal joy, very joy, etc. The emotion recognition unit 130 can recognize the type of external emotion through the analysis of the image of the other party. The emotion recognition unit 130 can recognize the type of external emotion according to the strength of the voice. It is possible to discriminate the strength and weakness of the type of external feeling that is attributed. The display unit 150 can display specific display information indicating the type of external feeling and the determination result of the strength and weakness. The century can be strong.

[6 This emotion recognition unit 130 uses the image of the other party to

You can determine the type of emotion of the other person, and determine the intensity of the determined type of emotion by using the strength of your voice.

[61] On the other hand, by using the source obtained through other means such as the collection unit 120

There may be a conflict between the other's emotions and the external emotions. In this case, changing the type of external emotions according to the perceived emotions based on different sources may correspond to the correction of the external emotions.

[62] The emotion recognition unit 130 may correct the perceived external emotion through analysis of the image of the other party if the external emotion and the sound emotion are different from each other and the intensity of the sound emotion satisfies the set value. For example, the external appearance While emotions are perceived as'joy', sound emotions can be recognized as'sadness.' In this case, if the intensity of the sound emotion, for example, the strength of the voice, the decibels, or the volume is above the set value, the emotion recognition unit ( 130) can modify the external emotion of'joy' to the state of'sadness'. The display unit 150 is the appearance modified by the emotion recognition unit 130 2020/175969 1 ^» （：1^1{2020/002928 You can express your emotions.

[63] The emotion recognition unit 130 may provide the determined external emotion to the display unit 150 as it is, when the external emotion and the sound emotion are the same or the intensity of the sound emotion does not satisfy the set value. For example, , If the intensity of the sound emotion is less than the set value, the emotion recognition unit 130 may ignore the sound emotion and provide the already recognized external emotion'joy' to the display unit 150 as it is.

[64] An object recognition unit 170 can be provided to guide users with low vision.

[65] The photographing unit 110 can photograph surrounding objects. The object recognition unit 170 is

The surrounding object photographed by the photographing unit 0 can be recognized. The display unit 150 can display object information corresponding to the recognition result of the object by voice or vibration. Even if the user does not visually recognize the surrounding object, With the help of the recognition device of the present invention, surrounding objects can be recognized audibly or tactilely.

[66] On the other hand, in order to accurately grasp the emotion of the other party, the recognition device

You can use it.

[67] As an example, the emotion recognition unit 130 may identify person 1 excluding the other party among objects. The emotion recognition unit 130 analyzes the number of person 1s corresponding to the object, and determines the number of characters 1 If you are satisfied, you can highly correct the intensity of your external feelings.

[68] The other party can refrain from expressing external emotions when other people are around. For example, even in a very happy situation, the other party can express joy calmly in consideration of the people around him. To reflect such an environment, the emotion recognition unit 130 can determine the number of the surrounding person I excluding the other person. If the identified number of characters satisfies the set number, the intensity of the currently recognized appearance can be highly corrected. Through the analysis of the other party's facial expressions, the other party's emotions can be recognized as'normal joy.' Person 17}If there are 2 people corresponding to the set number, emotion

The recognition unit 130 may correct the intensity of the emotion as high as the setting step. By the correction, the external emotion of the other party is corrected to be “very joyful” and may be displayed through the display unit 150.

[69] On the other hand, even if the number of prosthesis of the neighboring person satisfies the set number, the other party's emotional expression may be different depending on whether the person moves or not.

If you are moving based on the person around you 17}, it can be determined that you are located on the street where the other bankers currently exist. In this case, it is highly likely that the pedestrian will not react to the other person's emotional expression. Recognized opponents can honestly express their feelings, similar to a normal situation.

1] Surrounding Person 171-If you are stopped in the vicinity of the other person, person I can react to the other person's emotional expression. For example, when several people are waiting on a crosswalk, the other person I is in the other person's intense expression of emotions. Alternatively, the other person 1 who stops around the other party may be an acquaintance of the other party. In this case, the acquaintance of the other party 2020/175969 1»（：1/10公020/002928 Can react to the other party's emotional expression. The other party who is aware of this fact can express his/her emotional expression calmly compared to the general situation. The other party's sentiment expressed as may be in a state where the intensity of the original sentiment information is weakened. Therefore, the other person's actual feelings can be stronger than those expressed externally.

Is

It is possible to analyze the movement of a person excluding the other party among surrounding objects photographed by the photographing department (:B0).

[73] The emotion recognition unit 130, when it is determined that one surrounding person is moving,

It can be provided to the display unit 150 as it is.

When it is determined that the surrounding person I is stationary, the emotion recognition unit 130 can correct the intensity of the external emotion high. The emotion recognition unit 130 displays an external emotion that is highly corrected in intensity instead of the original external emotion. Can be provided to

[75] In the recognition device of the present invention, a character that recognizes a character photographed in the photographing unit (0)

A recognition unit 180 may be provided. The emotion recognition unit 130 may analyze the contents of the character III included in the image photographed by the photographing unit 0. The emotion recognition unit 130 may be defined by the letter III. The appearance emotion can be corrected according to the analysis result of the content. The display unit 150 can display the appearance emotion corrected according to the analysis result of the content.

In this case, the other party

Regardless of whether or not there are other people around you, you can calmly express your current feelings. At this time, the actual feelings of the other party can have a higher intensity than the externally expressed feelings. The recognition device of the present invention analyzes characters. Through this, you can correct the external feeling with an intensity higher than the intensity of the emotion expressed by the other party.

[77] The emotion recognition unit 130 can analyze various characters included in the image and determine the swap environment in which the other party is facing. The environment in which the other party is facing corresponds to the current location of the other party.

21

880 ₇₈ 1 Content information such as facility information, text on books, letters, text messages, menu boards, product labels, screens, etc. that affect the emotions of the other party may be included.

As a result of analyzing the content of the text message of the smartphone that the other party is looking at, it can be determined that you have passed the XXX company. At this time, the emotion recognition unit 130 corrects the intensity high even if the external information of the other party is ``a little joy''. , It is possible to modify the external appearance information with'very joyful'. The display unit 150 can display the external feeling of the other party as'very joyful'. The user can accurately grasp the actual feeling of the other party and deal with the other party.

In the recognition apparatus, an object recognition unit 170, a character recognition unit 180, and a selection unit 190 may be provided.

As described above, the object recognition unit 170 may recognize surrounding objects photographed by the photographing unit 110.

[8 The character recognition unit 180 can recognize the character 111 photographed by the photographing unit (0).

The selection unit 190 is among the appearance information, recognition results of objects, and recognition results of characters. 2020/175969 1»（：1^1{2020/002928 Through the display unit 150, the object to be displayed can be selected according to the user's choice.

According to the selection unit 190, the user can use the object recognition unit 170 for the purpose of recognizing surrounding objects, or the user can use the character recognition unit 180 for the purpose of recognizing the surrounding characters. Display unit 150 ) May display the object selected by the selection unit 190.

When the appearance information is selected by the selection unit 190, the emotion recognition unit 130 may recognize the appearance emotion.

[84] When the appearance information is selected by the selection unit 190, the object recognition unit 170 and text

At least one of the recognition units 180 can recognize objects or characters while operating together with the emotion recognition unit 130.

[85] The emotion recognition unit 130 corresponding to the recognition result of the object from the object recognition unit 170

If you can subscribe to object information, you can use the object information to revise your appearance.

[86] The emotion recognition unit 130 corresponding to the recognition result of the character from the character recognition unit 180

When the content information of the text is subscribed, the appearance emotion can be modified using the content information.

[87] On the other hand, while dealing with the other party, it may be difficult for the user to shoot another location with the shooting unit (0) to correct the appearance of the person, because it may be mistaken as a disregard for the other party. .

[88] When confronting the other party, the storage unit 140 may be provided as a means for obtaining the external feeling of the other party in advance so that the user can show the attitude of focusing only on the other party.

In the storage unit 140, a photographed image captured by the photographing unit 0 may be stored for a set time.

[9 If the image of the other party is provided to the emotion recognition unit 130 at the present time,

The recognition unit 170 or the character recognition unit 180 may be provided with a photographed image stored for a set time before the current time point.

[91] The object recognition unit 170 may recognize surrounding objects included in the photographed image.

[92] The character recognition unit 180 may recognize characters included in the photographed image.

[93] The emotion recognition unit 130 may receive object information corresponding to the object recognition result from the object recognition unit 170, or may receive content information corresponding to the character recognition result from the character recognition unit 180.

[94] The emotion recognition unit 130 uses object information or content information to

The emotion may be corrected. The display unit 150 may display the appearance emotion corrected by the emotion recognition unit 130.

According to the present embodiment, when the user faces the other party, an analysis can be performed on the captured image photographed before the time of the other party's face-to-face and stored in the storage unit 140. The character is recognized, and the emotion recognition unit 130 may correct the external emotion of the other party facing by using the corresponding surrounding object or character. 2020/175969 1»（：1^1{2020/002928

[96] The recognition device of the present invention is manufactured in the form of glasses or a mounting device.

The recognition device of the present invention may be worn by the user. Alternatively, the recognition device of the present invention may be manufactured in the form of a mobile device to follow the user or be worn by the user. At this time, the photographing unit (0) equipped with glasses or the like may be formed to shoot the user's front side. According to the present embodiment, even when the user who starts to face the other party continuously looks at the other party, the external appearance of the other party can be accurately corrected by using the pre-stored peripheral environment. As a result, the actual feeling of the other party can be followed. A user who knows the other person's actual feelings accurately can respond correctly to the other's feelings.

3 is a flow chart showing the recognition method of the present invention.

The recognition method of FIG. 3 can be performed by the recognition device of FIG. 1.

[99] The photographing unit (0) can shoot the surrounding area desired by the user 510).

[10 This storage unit 140 may store the photographed image 520).

[101] The emotion recognition unit 130 may analyze the photographed image and extract the other party who communicates with the user 530).

When the other party is extracted from the photographed image, the emotion recognition unit 130 may recognize and correct the external emotion of the other party (540).

[103] The display unit 150 may display the appearance emotion corrected by the emotion recognition unit 130 (550).

[104] The step 540 of recognizing and correcting the external emotion can be subdivided as follows.

The selection unit 190 may select any one of emotion recognition, object recognition, and character recognition according to the user's selection (541).

[106] When object recognition is selected 541), the object recognition unit 170 can recognize surrounding objects photographed by the photographing unit 110 543). At this time, the recognition of the surrounding objects can be performed in real time. 150) may display the information of the surrounding object recognized by the object recognition unit 170 by voice or vibration.

[107] When character recognition is selected, 541), the character recognition unit 180 can recognize the characters photographed in the photographing unit 110 544). At this time, the character recognition can be performed in real time. The display unit 150 may display content information of a character recognized by the character recognition unit 180 by voice or vibration.

[108] When emotion recognition is selected 541), the emotion recognition unit 130 is projected on the shooting image

Using the other person's facial expressions or gestures, you can recognize your external emotions 542).

[109] The emotion recognition unit 130 may recognize a nearby object or recognize a text through the analysis of the photographed image previously stored in the storage unit 140. The recognition of an object or text performed during emotion recognition is performed in real time. In addition, the recognition of objects or characters performed during emotional recognition has a characteristic in that it targets the photographed image previously stored in the storage unit 140.

[11 This emotion recognition unit 130 recognizes object information or text corresponding to the object recognition result. 2020/175969 1»（：1^1{2020/002928 You can use the content information corresponding to the result to correct the appearance feeling.

[111] The emotion recognition unit 130 may determine the level of quietness that requires a quiet environment through the analysis of object information or content information, and correct the intensity of external emotion in proportion to the level of quietness.

[112] The quietness of the location can be determined through the text included in the photographed image.

[113] As an example, a quietness level per number of surrounding people included in a photographed image may be set. If the number of neighboring people is 1, it can be set as quietness level 1, if the number of neighbors is 2, quietness level 2, etc. In the case of the second level of emotional pleasure of the other party, if the level of quietness is 1, the emotion recognition unit (130 ) Can correct the external information in three stages of joy, which increases the intensity of the current emotion by one level. If the current level of emotion and joy of the other party is at the second level, and if the quietness is 2, the emotion recognition unit 130 adjusts the strength of the current emotion to the second level. The appearance information can be corrected in 4 steps of increased joy.

[114] If the current location determined through the text included in the photographed image is a library, a quietness level of 5 can be given. If the current location is a market, a quietness level of 1 can be given. If the quietness level is 5, the emotion recognition unit 130 can correct the appearance information to 7 levels of joy, which raises the current level of emotion by 5 levels. If the current level of emotion is 2 levels of the other party's emotions, the level of quietness is 1, the emotion recognition unit ( 130) can correct the appearance information in three stages of joy, which raises the intensity of the current emotion by one level.

[115] The present invention can perform recognition of five basic emotions (eg, joy, surprise, sadness, anger, fear). The present invention has 17 complex emotions (eg, positive and negative, arousal and taste awakening). According to the three-dimensional concept of domination and obedience, it is possible to perform recognition of complex emotions, including intensity (intensity) levels).

[116] The present invention has a function of presenting a communication language. Specifically, the present invention has a maximum of 75 emotions (eg, energetic, cool, disappointed, fearful, a lot of surprise, despair, embarrassment, anxiety, annoyance). They can recognize and express their feelings of anger, irritability, wanting to get out, energetic, fun, and shy.

[117] Regarding emotion recognition, the present invention uses deep learning artificial intelligence technique.

The present invention can recognize a variety of complex emotions that were previously difficult to recognize by using a high-dimensional combination method in complex expressions.

[118] Regarding object recognition or emotion recognition, the present invention first advances environmental awareness and reduces the number of objects to objects that can be found within the identified environment in order to compensate for the recognition rate that decreases significantly when the number of objects increases. Can be raised.

[119] In the expression recognition technology, the present invention can input the motion flow of the sequence image into the Dictionary learning method, and design a recognition technology through the pre-learning of latent motions for each expression. The present invention is a variety of methods for extracting past expression features. PCA (Principle Component Analysis), ICA (Independent Component Analysis), ASM (Active Shape 2020/175969 1»（：1^1{2020/002928

Model), A AM (Active Appearance Model), etc., it has strong characteristics against changes in the lighting environment and differences in face color.

[120] The present invention can use an object recognition algorithm. Specifically, about 100

Object recognition and voice guidance algorithms that can detect indoor and outdoor atmospheres can be used.

[121] The present invention can use a character recognition algorithm. Specifically, an algorithm capable of recognizing characters in 102 languages including Korean, Chinese, Japanese, and English can be used.

[122] The present invention can use a basic emotional real-time recognition algorithm. Specifically,

An algorithm capable of real-time recognition of five basic emotions (eg, joy, surprise, sadness, anger, and fear) can be used.

[123] The present invention can use 17 complex expression and emotion recognition algorithms.

Specifically, AU (action unit) detection and LSTM-based complex emotion recognition function can be used.

[124] Regarding object recognition, the present invention can perform data collection and database construction, data learning and model creation, and object recognition YOLO.

[125] Regarding character recognition, the present invention can perform data collection and database construction, data learning and model generation, and character recognition tesseract-ocr use.

[126] Regarding the real-time emotion analysis system supporting facial expression recognition, the present invention

It is possible to collect expression data suitable for emotion measurement, establish a database, a real-time emotion analysis system, and convert to a communication language.

[127] Regarding the implementation of the integrated use system, the present invention can implement the object recognition, text recognition, emotion recognition and communication language speech integrated system.

[128] Feature vector extraction and AU detection for facial expression changes according to 23 emotion changes

Regarding the complex expression recognition algorithm based on the basis, the present invention is based on the feature extraction and AU detection method for landmark point and LSTM-based expression recognition, 23 expression recognition methods based on AU detection and LSTM features, and expression DB-based expression recognition. A method of improving the recognition rate of complex expressions through optimization can be used.

[129] The present invention can use emotional characteristic modeling based on complex expression recognition.

[130] The present invention can provide numerical indicators for emotional evaluation based on 23 expression recognition.

[131] The emotion device or method of the present invention can measure emotion as follows.

[132] The emotion recognition unit 130 may judge, recognize, or classify the emotion of the other party by using the expression extracted from the image of the other party. The expression may be the primary basis for judgment of the emotion. However, the expression may be a few seconds after expression. It is common to disappear within. Also, in the case of Asians, it is difficult to apply to emotion judgment because they do not make big expressions.

Using facial expressions, 75 emotions can be measured or classified. In the present invention, emotions can be evaluated by taking into account the movement of the facial muscles. 2020/175969 1»（：1^1{2020/002928 Depending on muscle mass, habits, etc., it may vary from person to person.

Data needs to be collected and analyzed.

[133] The emotion recognition unit 130 may judge, recognize, or classify the emotion of the other party by using the eyes extracted from the image of the other party. The eyes are the size of the eyes, the movement of the eyes, the direction of the eyes, the movement speed of the eyes, and the muscles of the eyes. It can be defined as a combination of the movement of the eyelids, the degree of opening and closing of the eyelids, the gaze time, the gap between the brows and the wrinkles. Using the eyes of the eyes, it can be defined as feelings of emotion (mood, hobbies, emotional atmosphere, emotions, etc.)

Including high-order complex emotions that occur during development) can be grasped; therefore, eyes can be used to measure emotions such as emotional stability and instability, passion and indifference, shyness, guilt, disgust, curiosity, and empathy.

[134] The emotion recognition unit 130 uses the movement of the neck extracted from the image of the other party

Can judge, recognize or classify the emotions of the other person. Neck movement can include the direction of the head, the angle between the head and the neck, the angle between the head and the back, the speed of the movement of the neck, the movement of the shoulder, and the degree and angle of the bending of the shoulder. Using neck movement, feelings such as depression and lethargy can be accurately measured.

[135] The emotion recognition unit 130 includes posture, gesture, and behavior extracted from the image of the other party.

Gestures can be used to judge, recognize, or classify the emotions of the other party. The emotion recognition unit 130 comprehensively judges the movements of the neck, arms, back, and legs, and the physical openness, aggression, physical pain, and excitement (elevated) ), you can measure your favor.

[136] The emotion recognition unit 130 may judge, recognize, or classify the emotion of the other party by using the voice of the other party. The height of the voice, the length of the voice, the wave of the voice, the intonation of the voice, and the voice of the other party are large and small. The other person's emotions can be classified by measuring and classifying the length of the lost paragraph and the intensity of the voice.

If there is no data, it can be the main data of emotion recognition, and if it matches the expression, it can increase the accuracy of expression recognition.

[137] The emotion recognition unit 130 may judge the other person’s emotions by using the vocabulary analyzed through the voice of the other party. The emotion recognition unit 130 uses a specific vocabulary related to emotion among the vocabulary expressed by the voice of the other party. You can judge, recognize, or classify the other person's feelings.

[138] On the other hand, a plurality of elements capable of grasping the emotions of the other party may be input through various input means such as a photographing unit and a collection unit. The emotion recognition unit 130 receives elements (emotion analysis information) obtained through multiple channels. You can do it like this:

[139] The emotion recognition unit 130, through the image of the other party or the voice of the other party,

Multiple elements can be extracted from which emotion can be grasped. The factors can include expressions, eyes, movements of the neck, gestures, voices, vocabulary, etc. For example, the emotion recognition unit analyzes the other person's image to analyze the other person's gestures and expressions. The emotion recognition unit can grasp at least one of the voice of the other party and the vocabulary spoken by the other party through the collection unit. 2020/175969 1»（：1^1{2020/002928

[14 This emotion recognition unit 130 receives the data obtained first in time among a plurality of elements.

Alternatively, the emotion recognition unit 130 may recognize a specific element used to recognize the emotion of the other party according to whether each element satisfies a set intensity or whether each element lasts for a set time. Can be selected.

[141] For example, expressions, eyes, movements, gestures, voices, and vocabulary related to the other party

When obtained, the emotion recognition unit 130 can analyze the intensity of each element. For example, it is assumed that the expression level 1, the eyes 3 level, the movement level 2, the gesture level 5, the voice level 2, and the vocabulary level 0 When the set intensity is set to 3 levels, the emotion recognition unit 130 can recognize the emotion of the other party by using only the eyes and gestures with the intensity 3 or higher.

[142] For example, expressions, eye movements, movements, gestures, voices, and vocabulary related to the other party

Once acquired, the emotion recognition unit 130 can analyze the duration of each element. For example, it is assumed that the expression is 3 seconds, the eyes are 4 seconds, the neck movement is 1 second, the gesture is 2 seconds, the voice is 3 seconds, and the vocabulary is 2 seconds. When the set time is set to 3 seconds, the emotion recognition unit 130 can recognize/determine/classify the other person's emotions using only the expression, eyes, and voice whose duration is 3 seconds or more.

[143] On the other hand, multiple factors having the same intensity and duration at the same time

At this time, the emotion recognition unit 130 may give priority to classify emotions in the order of gesture, voice, expression, vocabulary, and neck movement. Or, the emotion recognition unit ( 130) can give priority to classify emotions in the order of gesture, voice, expression, vocabulary, and eyes. According to this embodiment, when various complex emotion elements are identified, the priority to accurately grasp the emotion of the other party is given. Can be assigned to each complex emotion element.

[144]

4 is a diagram showing a computing device according to an embodiment of the present invention. The computing device TNW0 of FIG. 4 may be a device (eg, a recognition device) described in this specification.

In the embodiment of FIG. 4, the computing device TNW0 may include at least one processor (TN110), a transmission/reception device (TN120), and a memory (TN130). In addition, the computing device (TNW0) is a storage device. (TN140), input interface device (TN150), output interface device (TN160), etc. may be further included. Components included in the computing device (TNW0) are connected by a bus (TN170) to communicate with each other. can do.

[147] The processor (TN110) can execute a program command stored in at least one of the memory (TN130) and the storage device (TN140). The processor (TN110) is a central processing unit (CPU), It may mean a graphics processing unit (GPU: graphics processing unit), or a dedicated processor in which the methods according to the embodiment of the present invention are performed. The processor (TN110) is a process, function, and It can be configured to implement methods, etc. The processor (TN110) can control each component of the computing device (TNW0). 2020/175969 1»（：1^1{2020/002928

[148] Each of the memory (TN130) and the storage device (TN140) can store various information related to the operation of the processor (TN110). The memory (TN130) and the storage device (TN140) are each a volatile storage medium and a nonvolatile storage medium. It can be composed of at least one of them. For example, the memory (TN130) may consist of at least one of read only memory (ROM) and random access memory (RAM).

[149] The transmitting and receiving device (TN120) can transmit or receive a wired signal or a wireless signal. The transmitting and receiving device (TN120) is connected to the network and can perform communication.

[15] On the other hand, embodiments of the present invention are only through the apparatus and/or method described so far.

It is not implemented, but may be implemented through a program that implements a function corresponding to the configuration of the embodiment of the present invention or a recording medium in which the program is recorded, and this implementation is usually carried out in the technical field to which the present invention belongs from the description of the above-described embodiment. This is something that can be easily implemented if you are a technician.

[151] Although the embodiments of the present invention have been described in detail above,

The scope of the rights is not limited thereto, and various modifications and improvements of ordinary technicians using the basic concept of the present invention defined in the following claims also belong to the scope of the present invention.

Claims

2020/175969 1»（：1/10公020/002928 Claims

[Claim 1] A photographing unit for photographing the other party facing the user;

An emotion recognition unit for recognizing an external emotion of the other party through analysis of the image of the other party photographed by the photographing unit;

A display unit for displaying the external feeling by voice or vibration; Recognition device comprising a.

[Claim 2] The method of claim 1,

A collection unit for collecting the voice of the other party is provided, and the emotion recognition unit recognizes the voice emotion of the other party through the analysis of the voice, and corrects the appearance emotion using the recognized sound emotion,

The display unit is a recognition device that displays the corrected appearance feeling.

[Claim 3] The method of claim 1,

A collection unit that collects the voices of the other party is provided, and a plurality of display information is provided on the display unit to indicate the strengths and weaknesses of the emotions for the same type of external emotion.

The emotion recognition unit recognizes the type of external emotion through the analysis of the image of the other party,

The emotion recognition unit determines the strength and weakness of the type of external emotion recognized according to the strength of the voice,

The display unit is a recognition device for displaying specific display information indicating the type of the appearance feeling and the determination result of the strength and weakness.

[Claim 4] The method of claim 1,

The emotion recognition unit recognizes at least one of the other party’s expression and gesture through analysis of the other’s image,

The emotion recognition unit A recognition device that recognizes the external emotion by using the expression or the gesture.

[Claim 5] The method of claim 1,

A collection unit for collecting the voice of the other party is provided, and the emotion recognition unit recognizes the voice emotion of the other party through analysis of the voices collected by the collection unit,

If the external emotion and the sound emotion are different from each other, and the intensity of the sound emotion satisfies the set value, the emotion recognition unit corrects the recognized external emotion through analysis of the partner image,

The display unit displays the appearance emotion modified by the emotion recognition unit, and the emotion recognition unit indicates that the external appearance is instrumented if the external emotion and the sound emotion are the same, or the intensity of the sound emotion does not satisfy the set value. 2020/175969 1»（：1^1{2020/002928 A recognition device that provides emotions to the above display as it is.

[Claim 6] In paragraph 1,

The photographing unit photographs surrounding objects,

An object recognition unit for recognizing the object photographed by the photographing unit is provided, and the display unit displays object information corresponding to the recognition result of the object by voice or vibration.

[Claim 7] In paragraph 1,

An object recognition unit for recognizing a surrounding object photographed by the photographing unit is provided, and the emotion recognition unit analyzes the object recognized by the object recognition unit, and corrects the appearance emotion according to the analysis result,

The display unit is a recognition device that displays the appearance emotion corrected according to the analysis result of the thing.

[Claim 8] In paragraph 1,

An object recognition unit for recognizing surrounding objects photographed by the photographing unit is provided, and the emotion recognition unit identifies a person excluding the other party among the objects,

The emotion recognition unit analyzes the number of the characters, and when the number of the characters satisfies the set number, a recognition device that enhances the intensity of the external emotion. [Claim 9] In paragraph 1,

The emotion recognition unit analyzes whether the person moves or not, and the emotion recognition unit provides the external emotion to the display unit as it is, when it is determined that the person is moving,

The emotion recognition unit, when it is determined that the person has stopped, a recognition device that highly corrects the intensity of the appearance emotion, and provides a highly corrected appearance emotion to the display unit instead of the original appearance emotion.

[In paragraph 1 of claim 1,

A character recognition unit that recognizes the character photographed by the photographing unit is provided, and the emotion recognition unit analyzes the content of the character, and corrects the appearance emotion according to the analysis result of the content,

The display unit is a recognition device that displays the appearance emotion corrected according to the analysis result of the above contents.

[Claim 11] In paragraph 1,

Object recognition department, character recognition department, and selection department are prepared,

The object recognition unit recognizes a surrounding object photographed by the photographing unit, and the character recognition unit recognizes a character photographed by the photographing unit, 2020/175969 1»（：1^1{2020/002928 The selection unit selects the object to be displayed through the display unit among the appearance information, the recognition result of the object, and the recognition result of the character according to the user's selection,

The display unit is a recognition device that displays the object selected by the selection unit.

[Claim 12] In paragraph 11,

When the appearance information is selected by the selection unit, the emotion recognition unit recognizes the appearance emotion,

When the appearance information is selected by the selection unit, at least one of the object recognition unit and the character recognition unit recognizes the object or the character while operating together with the emotion recognition unit,

When the emotion recognition unit receives object information corresponding to the recognition result of the object from the object recognition unit, it uses the object information to correct the appearance emotion,

When the emotion recognition unit can subscribe to the content information of the character corresponding to the recognition result of the character from the character recognition unit, a recognition device that uses the content information to correct the appearance emotion.

[Claim 13] In paragraph 1,

A storage unit is provided in which the photographed image captured by the photographing unit is stored for a set time,

When the other person image is provided to the emotion recognition unit at the current point in time, an object recognition unit or a character recognition unit is provided to provide the photographed image stored for the set time before the current point of time,

The object recognition unit recognizes a surrounding object included in the captured image, the character recognition unit recognizes a character included in the captured image, and the emotion recognition unit provides object information corresponding to the object recognition result from the object recognition unit. Or receive content information corresponding to the character recognition result from the character recognition unit,

The emotion recognition unit corrects the appearance emotion originated by using the object information or the content information,

The display unit is a recognition device for displaying an external emotion modified by the emotion recognition unit.

[Claim 14] In paragraph 1,

The emotion recognition unit extracts a plurality of elements capable of grasping the emotion of the other party through the image of the other party or the voice of the other party, and the emotion recognition unit uses the data obtained first in time among the plurality of factors to determine the emotion of the other party. Depending on whether or not each element satisfies a set intensity, or whether each element lasts for a set time, a specific factor used to recognize the emotions of the other party is identified. 2020/175969 1»（：1^1{2020/002928 Recognition device selected.

[Claim 15] The method of claim 1,

A collection unit that collects the voice of the other party is provided, and the emotion recognition unit recognizes at least one of the other’s gestures, expressions, movements, and eyes through the analysis of the other’s image, and the emotion recognition unit communicates through the collection unit. Grasp at least one of the other's voice and vocabulary,

The emotion recognition unit gives priority to classify emotions in the order of the gesture, the voice, the expression, the vocabulary, and the movement of the neck, or the emotion recognition unit gives the above gesture, the voice, the expression, the vocabulary, the eyes A recognition device that gives priority to classifying emotions in the order of.

[Claim 16] In the recognition method performed by the recognition device,

Photographing the surroundings;

Saving the photographed image;

Analyzing the captured image and giving a person to talk to the user; When the other party is extracted, recognizing and correcting the external feeling of the other party;

Including; displaying the corrected appearance feeling;

The step of recognizing and correcting the appearance emotion,

Recognizing the external emotion by using the expression or gesture of the other party projected on the photographed image,

Through the analysis of the previously stored photographed image, the object is recognized or text is recognized,

A recognition method for correcting the appearance emotion by using object information corresponding to the recognition result of the object or content information corresponding to the recognition result of the character.

[Claim 17] In paragraph 16,

Through the analysis of the object information or the content information to determine the level of quietness that requires a quiet environment,

Recognition method for correcting the intensity of external emotion in proportion to the quietness.