JP2024155774A - Electronics - Google Patents

Abstract

[Problem] To execute an appropriate action. [Solution] An electronic device according to an embodiment includes a control unit that recognizes a user's action, determines its own action using history data updated based on the recognized user's action and information about the user, and controls a control target based on the determined own action. [Selected Figure] Figure 2

Description

The disclosed embodiment relates to an electronic device.

Conventionally, technology has been disclosed for determining an appropriate robot behavior in response to a user's state (see, for example, Patent Document 1). Patent Document 1 discloses a method for recognizing a user's reaction when a robot performs a specific action, and if the robot is unable to determine an action to be taken in response to the recognized user's reaction, the robot's behavior is updated by receiving information about an action appropriate to the recognized user's state from a server.

Patent No. 6053847

However, conventional technology leaves room for improvement in terms of executing appropriate actions in response to user actions.

The present invention has been made in consideration of the above, and aims to provide an electronic device that can perform appropriate actions.

The electronic device according to one aspect of the embodiment includes a control unit that recognizes a user's behavior, determines its own behavior using history data updated based on the recognized user's behavior and information about the user, and controls a control target based on the determined own behavior.

According to one aspect of the embodiment, appropriate action can be taken.

FIG. 1 is a diagram illustrating an example of a control system according to the present embodiment. FIG. 2 is a diagram illustrating a schematic functional configuration of the robot. FIG. 3 is a diagram illustrating an example of an operational flow relating to an operation for determining an action in a robot. FIG. 4 is a diagram illustrating an example of a hardware configuration of a computer that functions as a robot and a server. FIG. 5 is a diagram showing an emotion map onto which a plurality of emotions are mapped. FIG. 6 is a diagram showing another example of an emotion map. FIG. 7 is a diagram showing an example of an emotion table. FIG. 8 is a diagram showing an example of an emotion table.

The present invention will be described below through the embodiments, but the following embodiments do not limit the invention according to the claims. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Furthermore, in this embodiment, a "robot" is used as an example of an electronic device. Note that the electronic device may be a stuffed toy, a portable terminal device such as a smartphone, an input device such as a smart speaker, etc., in addition to a robot.

FIG. 1 is a diagram illustrating an example of a control system 1 according to the present embodiment. As shown in FIG. 1, the control system 1 includes a plurality of robots 100, a linked device 400, and a server 300. Each of the plurality of robots 100 is managed by a user.

The robot 100 converses with the user and provides the user with video. At this time, the robot 100 cooperates with a server 300 or the like with which it can communicate via the communication network 20 to converse with the user and provide the user with video, etc. For example, the robot 100 not only learns appropriate conversation by itself, but also cooperates with the server 300 to learn how to have a more appropriate conversation with the user. The robot 100 also records captured video data of the user, etc. in the server 300, and requests the video data, etc. from the server 300 as necessary to provide it to the user.

The robot 100 also has an emotion value that represents the type of emotion it feels. For example, the robot 100 has emotion values that represent the strength of each of the emotions: "happiness," "anger," "sorrow," "pleasure," "discomfort," "relief," "anxiety," "sorrow," "excitement," "worry," "relief," "fulfillment," "emptiness," and "neutral." When the robot 100 converses with the user in a state where the emotion value of excitement is high, for example, it speaks at a fast speed. In this way, the robot 100 can express its own emotions through its actions.

The robot 100 may be configured to determine the behavior of the robot 100 corresponding to the emotion of the user 10 by matching a sentence generation model, a so-called AI (Artificial Intelligence) chat engine, with an emotion engine. Specifically, the robot 100 may be configured to recognize the behavior of the user 10, determine the emotion of the user 10 regarding the user's behavior, and determine the behavior of the robot 100 corresponding to the determined emotion.

More specifically, when the robot 100 recognizes the behavior of the user 10, the robot 100 automatically generates behavioral content that the robot 100 should take in response to the behavior of the user 10 using a preset sentence generation model. The sentence generation model may be interpreted as an algorithm and calculation for automatic dialogue processing by text. The sentence generation model is publicly known as disclosed in, for example, JP 2018-081444 A and chatGPT (Internet search <URL: https://openai.com/blog/chatgpt>), so a detailed description thereof will be omitted. Such a sentence generation model is configured by a large language model (LLM: Large Language Model). As described above, in this embodiment, by combining a large language model and an emotion engine, it is possible to reflect the emotions of the user 10 and the robot 100 and various linguistic information in the behavior of the robot 100. In other words, according to this embodiment, a synergistic effect can be obtained by combining a sentence generation model and an emotion engine.

The robot 100 also has a function of recognizing the user's actions. The robot 100 recognizes the user's actions by analyzing the user's facial image acquired by the camera function and the user's voice acquired by the microphone function. The robot 100 determines the action to be performed by the robot 100 based on the recognized user's actions, etc.

The robot 100 stores rules that define the actions that the robot 100 will take based on the user's emotions, the robot's 100 emotions, and the user's actions, and performs various actions according to the rules.

Specifically, the robot 100 has reaction rules for determining the behavior of the robot 100 based on the user's emotions, the robot 100's emotions, and the user's actions. For example, the reaction rules define the behavior of the robot 100 as "laughing" when the user's action is "laughing". The reaction rules also define the behavior of the robot 100 as "apologizing" when the user's action is "anger". The reaction rules also define the behavior of the robot 100 as "answering" when the user's action is "asking a question". The reaction rules also define the behavior of the robot 100 as "calling out" when the user's action is "sadness".

When the robot 100 recognizes the user's behavior as "anger" based on the reaction rules, it selects the behavior of "apologizing" defined in the reaction rules as the behavior to be executed by the robot 100. For example, when the robot 100 selects the behavior of "apologizing", it performs the motion of "apologizing" and outputs a voice expressing the words "apologize".

In addition, when the robot's 100 emotions are "normal" (i.e., "happy" = 0, "anger" = 0, "sad" = 0, "happy" = 0) and the condition that the user's state is "alone and looks lonely" is met, it is defined that the robot's 100 emotions will change to "worried" and that it will be able to execute the action of "calling out."

When the robot 100 recognizes based on the reaction rules that the current emotion of the robot 100 is "normal" and that the user is alone and seems lonely, the robot 100 increases the emotion value of "sadness" of the robot 100. In addition, the robot 100 selects the action of "calling out" defined in the reaction rules as the action to be performed toward the user. For example, when the robot 100 selects the action of "calling out", it converts the words "What's wrong?", which express concern, into a worried voice and outputs it.

The robot 100 also transmits to the server 300 user reaction information indicating that this action has elicited a positive reaction from the user. The user reaction information includes, for example, the user action of "getting angry," the robot 100 action of "apologizing," the fact that the user reaction was positive, and the user's attributes.

The server 300 stores the user reaction information received from each robot 100. The server 300 then analyzes the user reaction information from each robot 100 and updates the reaction rules.

The robot 100 receives the updated reaction rules from the server 300 by inquiring about the updated reaction rules from the server 300. The robot 100 incorporates the updated reaction rules into the reaction rules stored in the robot 100. This allows the robot 100 to incorporate reaction rules acquired by other robots 100 into its own reaction rules. When the reaction rules are updated, they may be automatically transmitted from the server 300 to the robot 100.

The robot 100 can also perform actions in cooperation with the linked devices 400. The linked devices 400 are, for example, karaoke machines, wine cellars, refrigerators, terminal devices (such as PCs (Personal Computers), smartphones, and tablets), washing machines, automobiles, cameras, toilet equipment, electric toothbrushes, televisions, displays, furniture (such as closets), medicine boxes, musical instruments, lighting equipment, and exercise toys (such as unicycles). These linked devices 400 are communicatively connected to the robot 100 via the communication network 20, and transmit and receive information to and from the robot 100. With this configuration, the linked devices 400 follow instructions from the robot 100 to control themselves and converse with the user.

In this disclosure, we will explain examples of performing various actions for a user through collaboration between a linked device 400, such as a terminal device (PC (Personal Computer) 400a, smartphone 400b, tablet 400c, etc.) and a robot 100.

Specifically, the robot 100 recognizes the user's behavior, determines its own behavior using the history data 222 updated based on the recognized user's behavior and information about the user, and controls the control target based on its own determined behavior. Specifically, the robot 100 updates the stored history data 222 when at least one of the information about the user, information about the user's experience or information about the user's attributes, changes.

As described above, when a user has a new experience, the robot 100 updates the user's history data 222 based on the experience. In addition, when the user's attributes change, the robot 100 adapts the user's history data 222.

Specifically, the robot 100 updates the history data 222 by associating the user's experience sensed through the five senses with the user's emotion as information about the user's experience. For example, when the user performs a new experience of "cooking," the robot 100 updates the history data 222 by associating the user's experience sensed through the five senses with the user's emotion value of "joy," such as "(cooking) is fun!".

The robot 100 also updates the history data 222 based on the user's attribute information, including at least one of the user's name, age, sex, interests, hobbies, inclinations, lifestyle, personality, educational background, work history, place of residence, income, and family structure. For example, the robot 100 updates the history data 222 when "hobbies" as the user's attribute information changes. As a specific example, when "cooking" is added to the user's "hobbies," the robot 100 updates the history data 222 so that it is associated with "(cooking as a hobby) is fun!" and the "emotion value of joy," as described above.

Furthermore, when the robot 100 recognizes a change in information about the user, it makes an utterance to confirm with the user whether or not there has been a change. Specifically, when the robot 100 recognizes the action of the user cooking, it confirms with the user whether or not there has been a change in the hobby, for example, by asking, "Is cooking your hobby?". Then, the robot 100 updates the history data 222 when at least one of the confirmation of the change in information about the user and the user makes an utterance informing the robot that there has been a change in information about the user. For example, the robot 100 can recognize that the user attribute information has changed due to the user inputting information such as "Hobby: Cooking" and update the history data 222. Also, for example, when the robot 100 is informed that the user attribute information has changed by the user saying, "My hobby is cooking," or the like, it can update the history data 222.

Furthermore, the robot 100 recognizes changes in the user's experience or emotions through conversation with the user, and updates the history data 222 based on the changes in the user's experience or emotions. For example, the robot 100 can ask the user "Is cooking your hobby?" and update the history data 222 when it is confirmed that the hobby has changed to "cooking" as a result of the user's response such as "Yes, cooking has become my hobby," etc. The robot 100 can also ask the user a question such as "Do you enjoy cooking?" and update the history data 222 based on the user's response such as "Yes, cooking is fun," "I've been into cooking lately," etc.

Furthermore, the robot 100 recognizes changes in information about the user based on the user's state and behavior, and updates the history data 222. For example, when the user is cooking, the robot 100 can recognize the user's state, such as "The user appears to be enjoying cooking," and update the user's attribute information to "Hobby: Cooking," and update the information about the user's experience to "Cooking: Increase in joy emotion value."

In this manner, in the present disclosure, the robot 100 performs actions in cooperation with a terminal device (such as a PC, smartphone, or tablet) and, when a user has a new experience, updates the history data 222 of the user based on the experience. Furthermore, when the user attributes change, the robot 100 can adapt the history data 222 of the user. In other words, the robot 100 according to the present disclosure can execute appropriate actions for the user.

Figure 2 is a diagram showing a schematic functional configuration of the robot 100. The robot 100 is composed of a control unit having a sensor unit 200, a sensor module unit 210, a storage unit 220, a user state recognition unit 230, an emotion determination unit 232, a behavior recognition unit 234, a behavior determination unit 236, a memory control unit 238, a behavior control unit 250, a control target 252, and a communication processing unit 280.

The control object 252 includes a display device, a speaker, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and gestures of the robot 100 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 100 can be expressed by controlling these motors. In addition, the facial expressions of the robot 100 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 100. For example, the display device is provided on the chest of the robot 100. In addition, the facial expressions of the robot 100 can also be expressed by controlling the display of the display device. The display device may display the contents of a conversation with a user as text. The posture, gestures, and facial expressions of the robot 100 are examples of the attitude of the robot 100.

The sensor unit 200 includes a microphone 201, a 3D depth sensor 202, a 2D camera 203, a distance sensor 204, an acceleration sensor 205, a thermosensor 206, and a touch sensor 207. The microphone 201 continuously detects sound and outputs sound data. The microphone 201 may be provided on the head of the robot 100 and may have a function of performing binaural recording. The 3D depth sensor 202 detects the contour of an object by continuously irradiating an infrared pattern and analyzing the infrared pattern from infrared images continuously captured by the infrared camera. The 2D camera 203 is an example of an image sensor. The 2D camera 203 captures an image using visible light and generates visible light video information. The contour of the object may be detected from the video information generated by the 2D camera 203. The distance sensor 204 detects the distance to the object by irradiating, for example, a laser or ultrasonic wave. The acceleration sensor 205 is, for example, a gyro sensor, and detects the acceleration of the robot 100. The thermosensor 206 detects the temperature around the robot 100. The touch sensor 207 is a sensor that detects a touch operation by the user, and is disposed, for example, on the head and hands of the robot 100. The sensor unit 200 may also include a clock, a sensor for motor feedback, etc.

Note that, among the components of the robot 100 shown in FIG. 2, the components other than the control target 252 and the sensor unit 200 are examples of components of the behavior control system of the robot 100. The behavior control system of the robot 100 controls the control target 252.

The storage unit 220 includes reaction rules 221 and history data 222. The history data 222 includes the user's past emotion values and behavioral history. The emotion values and behavioral history are recorded for each user, for example, by being associated with the user's identification information. At least a part of the storage unit 220 is implemented by a storage medium such as a memory. It may include a person DB that stores the user's face image, user attribute information, etc. Note that the user attribute information here may include the user's name, age, sex, interests, concerns, hobbies, inclinations, lifestyle, personality, educational background, work history, place of residence, income, family structure, etc. Note that the functions of the components of the robot 100 shown in FIG. 2, except for the control target 252, the sensor unit 200, and the storage unit 220, can be realized by the CPU operating based on a program. For example, the functions of these components can be implemented as the operation of the CPU by the operating system (OS) and a program operating on the OS.

The sensor module unit 210 includes a voice emotion recognition unit 211, a speech understanding unit 212, a facial expression recognition unit 213, and a face recognition unit 214. Information detected by the sensor unit 200 is input to the sensor module unit 210. The sensor module unit 210 analyzes the information detected by the sensor unit 200 and outputs the analysis result to the user state recognition unit 230.

The voice emotion recognition unit 211 of the sensor module unit 210 analyzes the user's voice detected by the microphone 201 to recognize the user's emotions. For example, the voice emotion recognition unit 211 extracts features such as frequency components of the voice, and recognizes the user's emotions based on the extracted features. The speech understanding unit 212 analyzes the user's voice detected by the microphone 201, and outputs text information representing the content of the user's utterance. For example, the speech understanding unit 212 can analyze the content of a question asked by the user to the robot 100, such as "Yes, cooking is fun," or "I've been into cooking lately," and output text information representing the content of the user's utterance.

The facial expression recognition unit 213 recognizes the user's facial expression and emotions from the image of the user captured by the 2D camera 203. For example, the facial expression recognition unit 213 recognizes the user's facial expression and emotions based on the shape and positional relationship of the eyes and mouth. For example, the facial expression recognition unit 213 can recognize the facial expression and emotions when the user is performing a predetermined action or when the user is asking the robot 100 a question. Note that the above-mentioned "predetermined action" is, for example, an action such as the user engaging in a hobby, and the type and content of the action are not particularly limited.

The face recognition unit 214 recognizes the user's face. The face recognition unit 214 recognizes the user by matching a face image stored in a person DB (not shown) with the face image of the user captured by the 2D camera 203.

The user state recognition unit 230 recognizes the user's state based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 mainly performs processing related to perception using the analysis results of the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as "The user is cooking" or "The user has a happy expression on his/her face" and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as "The user is cooking happily."

The emotion determination unit 232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user's state are input to a pre-trained neural network to obtain an emotion value indicating the user's emotion.

Here, the emotion value indicating the user's emotion is a value indicating the positive or negative state of the user's emotion. For example, if the user's emotion is a cheerful emotion accompanied by a sense of pleasure or comfort, such as "joy," "pleasure," "comfort," "relief," "excitement," "relief," and "fulfillment," it will show a positive value, and the more cheerful the emotion, the larger the value. If the user's emotion is an unpleasant emotion, such as "anger," "sorrow," "discomfort," "anxiety," "sorrow," "worry," and "emptiness," it will show a negative value, and the more unpleasant the emotion, the larger the absolute value of the negative value will be. If the user's emotion is none of the above ("normal"), it will show a value of 0.

In addition, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230.

The emotion value of the robot 100 includes emotion values for each of a number of emotion categories, and is, for example, a value (0 to 5) indicating the strength of each of the emotions "joy," "anger," "sorrow," and "happiness."

Specifically, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 according to rules for updating the emotion value of the robot 100 that are determined in association with the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230.

For example, if the user state recognition unit 230 recognizes that the user looks lonely, the emotion determination unit 232 increases the "sad" emotion value of the robot 100. Also, if the user state recognition unit 230 recognizes that the user is smiling, the emotion determination unit 232 increases the "happy" emotion value of the robot 100.

The emotion determination unit 232 may determine the emotion value indicating the emotion of the robot 100 by further considering the state of the robot 100. For example, when the battery level of the robot 100 is low or when the surrounding environment of the robot 100 is completely dark, the emotion value of "sadness" of the robot 100 may be increased. Furthermore, when a user continues to talk to the robot 100 despite the battery level being low, the emotion value of "anger" may be increased.

The behavior recognition unit 234 recognizes the user's behavior based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the information analyzed by the sensor module unit 210 and the recognized user's state are input to a pre-trained neural network, the probability of each of a number of predetermined behavior categories (e.g., "laughing," "angry," "asking a question," "sad") is obtained, and the behavior category with the highest probability is recognized as the user's behavior. For example, the behavior recognition unit 234 recognizes the user's behavior such as "holding a terminal device in hand," "operating a terminal device," and "performing a specified action."

As described above, in this embodiment, the robot 100 acquires the user's speech content after identifying the user, but in order to acquire and use the speech content, the robot 100 obtains the necessary consent in accordance with laws and regulations from the user, and the behavior control system of the robot 100 according to this embodiment takes into consideration the protection of the user's personal information and privacy.

The behavior determination unit 236 determines a behavior corresponding to the user's behavior recognized by the behavior recognition unit 234 based on the user's current emotion value determined by the emotion determination unit 232, the history data 222 of past emotion values determined by the emotion determination unit 232 before the user's current emotion value was determined, and the emotion value of the robot 100. In this embodiment, the behavior determination unit 236 uses one most recent emotion value included in the history data 222 as the user's past emotion value, but the disclosed technology is not limited to this aspect. For example, the behavior determination unit 236 may use a plurality of most recent emotion values as the user's past emotion value, or may use an emotion value from a unit period ago, such as one day ago. In addition, the behavior determination unit 236 may determine a behavior corresponding to the user's behavior by further considering not only the current emotion value of the robot 100 but also the history of the past emotion values of the robot 100. The behavior determined by the behavior determination unit 236 includes gestures performed by the robot 100 or the contents of the speech of the robot 100.

The behavior determination unit 236 may determine a behavior corresponding to the user's behavior based on the emotion of the robot 100. For example, when the robot 100 is verbally abused by the user or is arrogant toward the user (i.e., when the user's reaction is poor), when the surrounding noise is loud and the user's voice cannot be detected, when the battery level of the robot 100 is low, etc., if the emotion value of "anger" or "sorrow" of the robot 100 increases, the behavior determination unit 236 may determine a behavior corresponding to the user's behavior according to the increase in the emotion value of "anger" or "sorrow". Also, when the user's reaction is good or the battery level of the robot 100 is high, etc., if the emotion value of "joy" or "pleasure" of the robot 100 increases, the behavior determination unit 236 may determine a behavior corresponding to the user's behavior according to the increase in the emotion value of "joy" or "pleasure". Furthermore, the behavior decision unit 236 may decide to take a different behavior toward a user who has increased the robot 100's emotion values of "anger" or "sadness" than the behavior toward a user who has increased the robot 100's emotion values of "joy" or "pleasure." In this way, the behavior decision unit 236 may decide to take a different behavior depending on the robot's own emotion itself and how the user has changed the robot 100's emotion through the user's action.

The behavior decision unit 236 according to this embodiment decides the behavior of the robot 100 as the behavior corresponding to the user's behavior, based on a combination of the user's past emotion value and current emotion value, the emotion value of the robot 100, the user's behavior, and the reaction rules 221. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, the behavior decision unit 236 decides the behavior corresponding to the user's behavior, which is to change the user's emotion value to a positive value.

The reaction rules 221 define the behavior of the robot 100 according to a combination of the user's past and current emotion values, the emotion value of the robot 100, and the user's behavior. For example, when the user's past emotion value is a positive value and the current emotion value is a negative value, and the user's behavior is sadness, a combination of gestures and speech content when asking a question to encourage the user with gestures is defined as the behavior of the robot 100.

For example, the reaction rules 221 define the behavior of the robot 100 for all combinations of patterns of the robot 100's emotion values (1296 patterns, which are the fourth power of six values of "joy", "anger", "sorrow", and "pleasure", from "0" to "5"); combination patterns of the user's past emotion values and current emotion values; and the user's behavior patterns. That is, for each pattern of the robot 100's emotion values, the behavior of the robot 100 is defined according to the user's behavior pattern for each of a plurality of combinations of the user's past emotion values and current emotion values, such as negative and negative values, negative and positive values, positive and negative values, positive and positive values, negative and normal, and normal and normal. Note that the behavior determination unit 236 may transition to an operation mode that determines the behavior of the robot 100 using the history data 222, for example, when the user makes an utterance intending to continue a conversation from a past topic, such as "I want to talk about that topic we talked about last time."

The reaction rules 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each of the patterns (1296 patterns) of the emotion value of the robot 100. Alternatively, the reaction rules 221 may define at least one of a gesture and a statement as the behavior of the robot 100 for each group of patterns of the emotion value of the robot 100.

For each gesture included in the behavior of the robot 100 defined in the reaction rules 221, the strength of the gesture is predefined. For each utterance content included in the behavior of the robot 100 defined in the reaction rules 221, the strength of the utterance content is predefined.

For example, the reaction rules 221 define the behavior of the robot 100 corresponding to a behavioral pattern such as an utterance related to a user's request when holding a terminal device in the hand, operating the terminal device, or performing a specific action. An example of an utterance related to a user's request may be a question to the robot 100 such as "My hobby is cooking."

The memory control unit 238 determines whether or not to store data including the user's behavior in the history data 222 based on the predetermined behavior strength for the behavior determined by the behavior determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232.

Specifically, if the total intensity value, which is the sum of the emotion values for each of the multiple emotion classifications of the robot 100, the predetermined intensity for the gesture included in the behavior determined by the behavior determination unit 236, and the predetermined intensity for the speech content included in the behavior determined by the behavior determination unit 236, is equal to or greater than a threshold value, it is determined that data including the user's behavior is to be stored in the history data 222.

When the memory control unit 238 decides to store data including the user's behavior in the history data 222, it stores in the history data 222 the behavior determined by the behavior determination unit 236, information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago (e.g., all peripheral information such as data on the sound, images, and smells of the scene), and the user's state recognized by the user state recognition unit 230 (e.g., the user's facial expression, emotions, etc.).

The storage control unit 238 also updates the stored history data 222 when at least one of the information about the user, information about the user's experience or the user's attribute information, changes.

As described above, when a user has a new experience, the memory control unit 238 updates the user's history data 222 based on that experience. Specifically, the memory control unit 238 updates the history data 222 by associating the user's experience sensed through the five senses with the user's emotions as information about the user's experience. For example, when a user has a new experience of "cooking," the memory control unit 238 updates the history data 222 by associating the user's experience sensed through the five senses with the user's emotion value of "joy" that "(cooking) is fun!".

As described above, the memory control unit 238 also adapts the history data 222 of a user when the user attributes change. Specifically, the memory control unit 238 updates the history data 222 based on the user's attribute information including at least one of the user's name, age, sex, interests, concerns, hobbies, inclinations, lifestyle, personality, educational background, work history, place of residence, income, and family structure. For example, when "hobbies" as the user's attribute information changes, the memory control unit 238 updates the history data 222 based on the updated user's attribute information. As a specific example, when "cooking" is added to the user's "hobbies," the memory control unit 238 updates the history data 222 so that it is associated with "(cooking as a hobby) is fun!" and the "emotion value of joy" as described above.

Furthermore, when a change in information about the user is recognized by the behavior control unit 250 described below, the memory control unit 238 updates the history data 222 based on the changed information about the user. For example, the memory control unit 238 can recognize that information about the user, including user attribute information, has changed when the user inputs information such as "Hobby: Cooking", and can update the history data 222. Furthermore, for example, the memory control unit 238 can update the history data 222 when the user notifies the memory control unit 238 that information about the user, including user attribute information, has changed by uttering an utterance such as "My hobby is cooking."

Furthermore, the memory control unit 238 recognizes changes in the user's experience or emotions through conversation with the user, and updates the history data 222 based on the changes in the user's experience or emotions. For example, the memory control unit 238 can ask the user "Is cooking your hobby?" and update the history data 222 when it is confirmed that the hobby has changed to "cooking" based on the user's response by speech such as "Yes, cooking has become my hobby." The memory control unit 238 can also ask the user a question such as "Do you enjoy cooking?" and update the history data 222 based on the user's response to the question such as "Yes, cooking is fun," or "I've been into cooking lately."

The memory control unit 238 also recognizes changes in information about the user based on the user's state and behavior, and updates the history data 222. For example, when the user is cooking, the memory control unit 238 can recognize the user's state by saying, "The user appears to be enjoying cooking," and update the user's attribute information to "Hobby: Cooking," and update the information about the user's experience to "Cooking: Increase in joy emotion value."

The behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior determination unit 236 determines an behavior including speaking, the behavior control unit 250 outputs a sound from a speaker included in the control target 252. At this time, the behavior control unit 250 may determine the speaking speed of the sound based on the emotion value of the robot 100. For example, the behavior control unit 250 determines a faster speaking speed as the emotion value of the robot 100 increases. In this way, the behavior control unit 250 determines the execution form of the behavior determined by the behavior determination unit 236 based on the emotion value determined by the emotion determination unit 232.

For example, when the behavior control unit 250 recognizes a change in information about the user, it makes an utterance to the user to confirm whether or not there has been a change. Specifically, when the behavior control unit 250 recognizes the user's action of cooking, it checks with the user whether or not their hobby has changed, such as by asking, "Is cooking your hobby?" Then, the aforementioned memory control unit 238 updates the history data 222 when at least one of the following is confirmed: a change in information about the user; or, the user makes an utterance to inform the robot 100 that there has been a change in information about the user.

The behavior control unit 250 may recognize a change in the user's emotions in response to the execution of the behavior determined by the behavior determination unit 236. For example, the change in emotions may be recognized based on the user's voice or facial expression. Alternatively, the change in the user's emotions may be recognized based on the detection of an impact by a touch sensor included in the sensor unit 200. If an impact is detected by the touch sensor included in the sensor unit 200, the user's emotions may be recognized as having worsened, and if the detection result of the touch sensor included in the sensor unit 200 indicates that the user's reaction is laughing or happiness, the user's emotions may be recognized as having improved. Information indicating the user's reaction is output to the communication processing unit 280.

In addition, after the behavior control unit 250 executes the behavior determined by the behavior determination unit 236 in the execution form determined according to the emotion of the robot 100, the emotion determination unit 232 further changes the emotion value of the robot 100 based on the user's reaction to the execution of the behavior. Specifically, the emotion determination unit 232 increases the emotion value of "happiness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is not bad. In addition, the emotion determination unit 232 increases the emotion value of "sadness" of the robot 100 when the user's reaction to the behavior determined by the behavior determination unit 236 being performed on the user in the execution form determined by the behavior control unit 250 is bad.

Furthermore, the behavior control unit 250 expresses the emotion of the robot 100 based on the determined emotion value of the robot 100. For example, when the behavior control unit 250 increases the emotion value of "happiness" of the robot 100, it controls the control object 252 to make the robot 100 perform a happy gesture. When the behavior control unit 250 increases the emotion value of "sadness" of the robot 100, it controls the control object 252 to make the robot 100 assume a droopy posture.

The communication processing unit 280 is responsible for communication with the server 300. As described above, the communication processing unit 280 transmits user reaction information to the server 300. The communication processing unit 280 also receives updated reaction rules from the server 300. When the communication processing unit 280 receives updated reaction rules from the server 300, it updates the reaction rules 221. The communication processing unit 280 can transmit and receive information to and from the linked device 400.

The server 300 communicates between each robot 100 and the server 300, receives user reaction information sent from the robot 100, and updates the reaction rules based on reaction rules that include actions that have received positive reactions.

Figure 3 is a diagram that shows an example of an operation flow related to the operation of determining the behavior of the robot 100. The operation flow shown in Figure 3 is executed repeatedly. At this time, it is assumed that information analyzed by the sensor module unit 210 is input. Note that "S" in the operation flow indicates the step that is executed.

First, in step S101, the user state recognition unit 230 recognizes the user's state based on the information analyzed by the sensor module unit 210. For example, the user state recognition unit 230 generates perceptual information such as "The user is cooking" or "The user has a happy expression on his/her face" and performs processing to understand the meaning of the generated perceptual information. For example, the user state recognition unit 230 generates semantic information such as "The user is cooking happily."

In step S102, the emotion determination unit 232 determines an emotion value indicating the user's emotion based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230.

In step S103, the emotion determination unit 232 determines an emotion value indicating the emotion of the robot 100 based on the information analyzed by the sensor module unit 210 and the state of the user recognized by the user state recognition unit 230. The emotion determination unit 232 adds the determined emotion value of the user to the history data 222.

In step S104, the behavior recognition unit 234 recognizes the user's behavior classification based on the information analyzed by the sensor module unit 210 and the user's state recognized by the user state recognition unit 230. For example, the behavior recognition unit 234 recognizes the user's behavior such as "holding a terminal device in hand," "operating a terminal device," and "performing a specified action."

In step S105, the behavior decision unit 236 decides the behavior of the robot 100 based on a combination of the user's current emotion value determined in step S102 and the past emotion value included in the history data 222, the emotion value of the robot 100, the user's behavior recognized by the behavior recognition unit 234, and the reaction rules 221.

In step S106, the behavior control unit 250 controls the control target 252 based on the behavior determined by the behavior determination unit 236. For example, when the behavior control unit 250 recognizes that the user is cooking, it checks with the user whether their hobby has changed, for example by asking, "Is cooking your hobby?"

In step S107, the memory control unit 238 calculates a total intensity value based on the intensity of the action determined by the action determination unit 236 and the emotion value of the robot 100 determined by the emotion determination unit 232. On the other hand, the memory control unit 238 updates the stored history data 222 when at least one of the information on the user's experience or the user's attribute information changes as information on the user. For example, when the user performs a new experience of "cooking," the memory control unit 238 updates the history data 222 by associating the user's experience sensed through the five senses with the user's emotion value of "joy," such as "(cooking) is fun!". Also, for example, when "cooking" is added to the user's "hobbies," the memory control unit 238 updates the history data 222 so that "(cooking as a hobby) is fun!" is associated with the "emotion value of joy" as described above.

In step S108, the memory control unit 238 determines whether the total intensity value is equal to or greater than the threshold value. If the total intensity value is less than the threshold value, the process ends without storing data including the user's behavior in the history data 222. On the other hand, if the total intensity value is equal to or greater than the threshold value, the process proceeds to step S109.

In step S109, the behavior determined by the behavior determination unit 236, the information analyzed by the sensor module unit 210 from the present time up to a certain period of time ago, and the user's state recognized by the user state recognition unit 230 are stored in the history data 222.

As described above, the robot 100 is equipped with a control unit that recognizes the user's behavior, determines its own behavior using history data updated based on the recognized user's behavior and information about the user, and controls a control target based on its own determined behavior. Specifically, the robot 100 updates the stored history data when at least one of the information about the user, information about the user's experience or information about the user's attributes, changes. This allows the robot 100 to keep the history data (emotion database) up to date, enabling more accurate emotion prediction.

Specifically, the control unit of the robot 100 updates the history data by associating the user's experience sensed through the five senses with the user's emotions as information about the user's experience. In this way, when a user has a new experience, the control unit of the robot 100 updates the user's history data based on that experience, thereby keeping the history data (emotion database) up to date and enabling more accurate emotion prediction.

The control unit of the robot 100 also updates the history data based on the user's attribute information, which includes at least one of the user's name, age, sex, interests, hobbies, inclinations, lifestyle, personality, educational background, work history, place of residence, income, and family structure. In this way, when the user's attribute information changes, the control unit of the robot 100 updates the user's history data based on the user's attribute information, thereby keeping the history data (emotion database) up to date and enabling more accurate emotion prediction.

For example, when the control unit of the robot 100 recognizes a change in information about the user, it speaks to the user to confirm whether there has been a change. Then, the control unit of the robot 100 updates the history data when at least one of the following is confirmed: a change in information about the user is confirmed; or the user speaks to the robot to inform the robot that there has been a change in information about the user. In this way, the control unit of the robot 100 can recognize changes in information about the user, including information about the user's experience and user attribute information, through conversation with the user, and update the history data (emotion database). This allows the robot 100 to keep the history data (emotion database) up to date through conversation and dialogue with the user without having to acquire the latest data each time, enabling more accurate emotion prediction.

Furthermore, the control unit of the robot 100 recognizes changes in the user's experience or emotions through conversation with the user, and updates the history data based on the changes in the user's experience or emotions. In this way, the control unit of the robot 100 can recognize changes in information about the user, including information about the user's experience and user attribute information, through conversation with the user, and update the history data (emotion database). This allows the robot 100 to keep the history data (emotion database) up to date through daily conversations and dialogue with the user, enabling more accurate emotion prediction.

In the above embodiment, the robot 100 recognizes the user using a facial image of the user, but the disclosed technology is not limited to this aspect. For example, the robot 100 may recognize the user using a voice emitted by the user, the user's email address, the user's SNS ID, or an ID card with a built-in wireless IC tag that the user possesses.

The robot 100 is an example of an electronic device equipped with a behavior control system. The application of the behavior control system is not limited to the robot 100, and the behavior control system can be applied to various electronic devices. Furthermore, the functions of the server 300 may be implemented by one or more computers. At least some of the functions of the server 300 may be implemented by a virtual machine. Furthermore, at least some of the functions of the server 300 may be implemented in the cloud.

Figure 4 is a diagram showing an example of a hardware configuration of a computer 1200 functioning as the robot 100 and the server 300. A program installed on the computer 1200 can cause the computer 1200 to function as one or more "parts" of an apparatus according to the present embodiment, or to execute operations or one or more "parts" associated with an apparatus according to the present embodiment, and/or to execute a process or steps of the process according to the present embodiment. Such a program can be executed by the CPU 1212 to cause the computer 1200 to execute specific operations associated with some or all of the blocks of the flowcharts and block diagrams described herein.

The computer 1200 according to this embodiment includes a CPU 1212, a RAM 1214, and a graphics controller 1216, which are connected to each other by a host controller 1210. The computer 1200 also includes input/output units such as a communication interface 1222, a storage device 1224, a DVD drive, and an IC card drive, which are connected to the host controller 1210 via an input/output controller 1220. The DVD drive may be a DVD-ROM drive, a DVD-RAM drive, etc. The storage device 1224 may be a hard disk drive, a solid state drive, etc. The computer 1200 also includes input/output units such as a ROM 1230 and a keyboard, which are connected to the input/output controller 1220 via an input/output chip 1240.

The CPU 1212 operates according to the programs stored in the ROM 1230 and the RAM 1214, thereby controlling each unit. The graphics controller 1216 acquires image data generated by the CPU 1212 into a frame buffer or the like provided in the RAM 1214 or into itself, and causes the image data to be displayed on the display device 1218.

The communication interface 1222 communicates with other electronic devices via a network. The storage device 1224 stores programs and data used by the CPU 1212 in the computer 1200. The DVD drive reads programs or data from a DVD-ROM or the like and provides them to the storage device 1224. The IC card drive reads programs and data from an IC card and/or writes programs and data to an IC card.

ROM 1230 stores therein a boot program, etc., executed by computer 1200 upon activation, and/or a program that depends on the hardware of computer 1200. I/O chip 1240 may also connect various I/O units to I/O controller 1220 via USB ports, parallel ports, serial ports, keyboard ports, mouse ports, etc.

The programs are provided by a computer-readable storage medium such as a DVD-ROM or an IC card. The programs are read from the computer-readable storage medium, installed in storage device 1224, RAM 1214, or ROM 1230, which are also examples of computer-readable storage media, and executed by CPU 1212. The information processing described in these programs is read by computer 1200, and brings about cooperation between the programs and the various types of hardware resources described above. An apparatus or method may be constructed by realizing the operation or processing of information according to the use of computer 1200.

For example, when communication is performed between computer 1200 and an external device, CPU 1212 may execute a communication program loaded into RAM 1214 and instruct communication interface 1222 to perform communication processing based on the processing described in the communication program. Under the control of CPU 1212, communication interface 1222 reads transmission data stored in a transmission buffer area provided in RAM 1214, storage device 1224, a DVD-ROM, or a recording medium such as an IC card, and transmits the read transmission data to the network, or writes received data received from the network to a reception buffer area or the like provided on the recording medium.

The CPU 1212 may also cause all or a necessary portion of a file or database stored in an external recording medium such as the storage device 1224, a DVD drive (DVD-ROM), an IC card, etc. to be read into the RAM 1214, and perform various types of processing on the data on the RAM 1214. The CPU 1212 may then write back the processed data to the external recording medium.

Various types of information, such as various types of programs, data, tables, and databases, may be stored in the recording medium and undergo information processing. The CPU 1212 may perform various types of processing on the data read from the RAM 1214, including various types of operations, information processing, conditional judgment, conditional branching, unconditional branching, information search/replacement, etc., as described throughout this disclosure and specified by the instruction sequence of the program, and write back the results to the RAM 1214. The CPU 1212 may also search for information in a file, database, etc. in the recording medium. For example, when multiple entries each having an attribute value of a first attribute associated with an attribute value of a second attribute are stored in the recording medium, the CPU 1212 may search for an entry whose attribute value of the first attribute matches a specified condition from among the multiple entries, read the attribute value of the second attribute stored in the entry, and thereby obtain the attribute value of the second attribute associated with the first attribute that satisfies a predetermined condition.

The above-mentioned programs or software modules may be stored in a computer-readable storage medium on the computer 1200 or in the vicinity of the computer 1200. In addition, a recording medium such as a hard disk or RAM provided in a server system connected to a dedicated communication network or the Internet can be used as a computer-readable storage medium, thereby providing the programs to the computer 1200 via the network.

The blocks in the flowcharts and block diagrams in this embodiment may represent stages of a process in which an operation is performed or "parts" of a device responsible for performing the operation. Particular stages and "parts" may be implemented by dedicated circuitry, programmable circuitry provided with computer-readable instructions stored on a computer-readable storage medium, and/or a processor provided with computer-readable instructions stored on a computer-readable storage medium. Dedicated circuitry may include digital and/or analog hardware circuitry, and may include integrated circuits (ICs) and/or discrete circuits. Programmable circuitry may include reconfigurable hardware circuitry including AND, OR, XOR, NAND, NOR, and other logical operations, flip-flops, registers, and memory elements, such as, for example, field programmable gate arrays (FPGAs), programmable logic arrays (PLAs), and the like.

A computer-readable storage medium may include any tangible device capable of storing instructions that are executed by a suitable device, such that a computer-readable storage medium having instructions stored thereon comprises an article of manufacture that includes instructions that can be executed to create means for performing the operations specified in the flowchart or block diagram. Examples of computer-readable storage media may include electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, and the like. More specific examples of computer-readable storage media may include floppy disks, diskettes, hard disks, random access memories (RAMs), read-only memories (ROMs), erasable programmable read-only memories (EPROMs or flash memories), electrically erasable programmable read-only memories (EEPROMs), static random access memories (SRAMs), compact disk read-only memories (CD-ROMs), digital versatile disks (DVDs), Blu-ray disks, memory sticks, integrated circuit cards, and the like.

The computer readable instructions may include either assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source or object code written in any combination of one or more programming languages, including object-oriented programming languages such as Smalltalk (registered trademark), JAVA (registered trademark), C++, etc., and conventional procedural programming languages such as the "C" programming language or similar programming languages.

The computer-readable instructions may be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or a programmable circuit, either locally or over a local area network (LAN), a wide area network (WAN), such as the Internet, so that the processor of the general-purpose computer, special-purpose computer, or other programmable data processing apparatus, or the programmable circuit, executes the computer-readable instructions to generate means for performing the operations specified in the flowcharts or block diagrams. Examples of processors include computer processors, processing units, microprocessors, digital signal processors, controllers, microcontrollers, etc.

(Other embodiments)
The robot 100 described above may be mounted on a stuffed toy, or may be applied to a control device connected wirelessly or by wire to a control target device (speaker or camera) mounted on the stuffed toy.

The emotion determination unit 232 may determine the user's emotion according to a specific mapping. Specifically, the emotion determination unit 232 may determine the user's emotion according to an emotion map (see FIG. 5), which is a specific mapping.

5 is a diagram showing an emotion map 700 on which multiple emotions are mapped. In the emotion map 700, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. Emotions that represent states and actions arising from a state of mind are arranged on the outer sides of the concentric circles. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 700, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

(1) For example, if the emotion engine, which is the emotion determination unit 232 of the robot 100, detects emotions at approximately 100 msec, the frequency of the determination of the reaction action of the robot 100 (e.g., a backchannel) may be set to at least the same timing as the detection frequency of the emotion engine (100 msec), or may be set to an earlier timing. The detection frequency of the emotion engine may be interpreted as the sampling rate.

By detecting emotions in about 100 msec and immediately performing a corresponding reaction (e.g., a backchannel), unnatural backchannels can be avoided, resulting in a natural dialogue that reads the atmosphere. The reaction (backchannel, etc.) is performed according to the directionality and degree (strength) of the mandala in the robot's 100 emotion map 700. Note that the detection frequency (sampling rate) of the emotion engine is not limited to 100 ms, and may be changed according to the situation (e.g., when playing sports), the age of the user, etc.

(2) The directionality of emotions and the strength of their intensity may be set in advance in reference to the emotion map 700, and the movement of the interjections and the strength of the interjections may be set. For example, if the robot 100 feels a sense of stability, security, etc., the robot 100 may nod and continue listening. If the robot 100 feels anxious, confused, or suspicious, the robot 100 may tilt its head or stop shaking its head.

These emotions are distributed in the three o'clock direction of emotion map 700, and usually fluctuate between relief and anxiety. In the right half of emotion map 700, situational awareness takes precedence over internal sensations, resulting in a sense of calm.

(3) If the robot 100 feels good after being praised, the filler "ah" may be inserted before the line, and if the robot 100 feels hurt after receiving harsh words, the filler "ugh!" may be inserted before the line. Also, a physical reaction such as the robot 100 crouching down while saying "ugh!" may be included. These emotions are distributed around 9 o'clock on the emotion map 700.

(4) In the left half of the emotion map 700, internal sensations (reactions) are more important than situational awareness. This can give the impression that the person is reacting unconsciously.

When the robot 100 feels a favorable feeling in its situational awareness while also feeling an internal sensation (reaction) of satisfaction, the robot 100 may nod deeply while looking at the other person, or may say "uh-huh." In this way, the robot 100 may generate a behavior that shows a balanced favorable feeling toward the other person, that is, acceptance and tolerance toward the other person. Such emotions are distributed around 12 o'clock on the emotion map 700.

Conversely, even when the robot 100 is aware of a situation while experiencing an internal sensation (reaction) of discomfort, the robot 100 may shake its head when it feels disgust, or turn the eye LEDs red and glare at the other person when it feels hatred. These types of emotions are distributed around the 6 o'clock position on the emotion map 700.

(5) The inside of emotion map 700 represents what is going on inside one's mind, while the outside of emotion map 700 represents behavior, so the further out one goes on emotion map 700, the more visible (more visible in behavior).

(6) When listening to someone with a sense of relief, which is distributed around the 3 o'clock area of the emotion map 700, the robot 100 may lightly nod its head and say "hmm," but when it comes to love, which is distributed around 12 o'clock, it may nod vigorously, nodding its head deeply.

The emotion determination unit 232 inputs the information analyzed by the sensor module unit 210 and the recognized state of the user 10 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 700, and determines the emotion of the user 10. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210 and the recognized state of the user 10, and emotion values indicating each emotion shown in the emotion map 700. In addition, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Figure 6. Figure 6 is a diagram showing another example of an emotion map. Figure 6 shows an example in which multiple emotions, such as "peace of mind," "calm," and "reassuring," have similar emotion values.

Furthermore, the emotion determination unit 232 may determine the emotion of the robot 100 according to a specific mapping. Specifically, the emotion determination unit 232 inputs the information analyzed by the sensor module unit 210, the state of the user 10 recognized by the user state recognition unit 230, and the state of the robot 100 into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 700, and determines the emotion of the robot 100. This neural network is pre-trained based on multiple learning data that are combinations of the information analyzed by the sensor module unit 210, the recognized state of the user 10, and the state of the robot 100, and emotion values indicating each emotion shown in the emotion map 700. For example, the neural network is trained based on learning data indicating that when the robot 100 is recognized as being stroked by the user 10 from the output of the touch sensor 207, the emotional value becomes "happy" at "3," and when the robot 100 is recognized as being hit by the user 10 from the output of the acceleration sensor 205, the emotional value becomes "anger" at "3." Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 6.

The emotion determination unit 232 may also determine the emotion of the robot 100 based on the behavioral content of the robot 100 generated by the sentence generation model. Specifically, the emotion determination unit 232 inputs the behavioral content of the robot 100 generated by the sentence generation model into a pre-trained neural network, obtains emotion values indicating each emotion shown in the emotion map 700, and integrates the obtained emotion values indicating each emotion with the emotion values indicating each emotion of the current robot 100 to update the emotion of the robot 100. For example, the emotion values indicating each emotion obtained and the emotion values indicating each emotion of the current robot 100 are averaged and integrated. This neural network is pre-trained based on multiple learning data that are combinations of texts indicating the behavioral content of the robot 100 generated by the sentence generation model and emotion values indicating each emotion shown in the emotion map 700.

For example, if the speech content of the robot 100, "That's great. You're lucky," is obtained as the behavioral content of the robot 100 generated by the sentence generation model, then when the text representing this speech content is input to the neural network, a high emotion value is obtained for the emotion "happy," and the emotion of the robot 100 is updated so that the emotion value of the emotion "happy" becomes higher.

The behavior decision unit 236 generates the robot's behavior by adding fixed sentences to the text representing the user's behavior, the user's emotions, and the robot's emotions, and inputting the results into a sentence generation model with a dialogue function.

For example, the behavior determination unit 236 obtains text representing the state of the robot 100 from the emotion of the robot 100 determined by the emotion determination unit 232, using an emotion table such as that shown in FIG. 7. FIG. 7 is a diagram showing an example of an emotion table. Here, in the emotion table, an index number is assigned to each emotion value for each type of emotion, and text representing the state of the robot 100 is stored for each index number.

When the emotion of the robot 100 determined by the emotion determination unit 232 corresponds to index number "2", the text "very happy state" is obtained. Note that when the emotion of the robot 100 corresponds to multiple index numbers, multiple pieces of text representing the state of the robot 100 are obtained.

Also, for the emotions of the user 10, an emotion table as shown in FIG. 8 is prepared. FIG. 8 is a diagram showing an example of an emotion table. Here, if the user's behavior is "talk to AAA", the emotion of the robot 100 is index number "2", and the emotion of the user 10 is index number "3", then "The robot is in a very happy state. The user is in a normal happy state. The user spoke to me with 'AAA'. How would you respond as the robot?" is input into the sentence generation model to obtain the robot's behavior content. The behavior decision unit 236 decides on the robot's behavior from this behavior content. Note that "AAA" is the name (nickname) given to the robot 100 by the user.

In this way, the robot 100 can change its behavior according to the index number that corresponds to the robot's emotions, so the user gets the impression that the robot 100 has a heart, and is encouraged to take actions such as talking to the robot.

The behavior decision unit 236 may generate the robot's behavior content by adding not only text representing the user's behavior, the user's emotions, and the robot's emotions, but also text representing the contents of the history data 222, adding a fixed sentence for asking about the robot's behavior content corresponding to the user's behavior, and inputting the result into a sentence generation model with a dialogue function. This allows the robot 100 to change its behavior according to the history data representing the user's emotions and behavior, so that the user has the impression that the robot has a personality, and is encouraged to take actions such as talking to the robot. The history data may also further include the robot's emotions and behavior.

The present invention has been described above using an embodiment, but the technical scope of the present invention is not limited to the scope described in the above embodiment. It is clear to those skilled in the art that various modifications and improvements can be made to the above embodiment. It is clear from the claims that forms with such modifications or improvements can also be included in the technical scope of the present invention.

The order of execution of each process, such as operations, procedures, steps, and stages, in the devices, systems, programs, and methods shown in the claims, specifications, and drawings is not specifically stated as "before" or "prior to," and it should be noted that the processes may be performed in any order, unless the output of a previous process is used in a later process. Even if the operational flow in the claims, specifications, and drawings is explained using "first," "next," etc. for convenience, it does not mean that it is necessary to perform the processes in this order.

1 Control system 20 Communication network 100 Robot 200 Sensor unit 201 Microphone 202 3D depth sensor 203 2D camera 204 Distance sensor 205 Acceleration sensor 206 Thermosensor 207 Touch sensor 210 Sensor module unit 211 Voice emotion recognition unit 212 Speech understanding unit 213 Facial expression recognition unit 214 Face recognition unit 220 Storage unit 221 Response rule 222 History data 230 User state recognition unit 232 Emotion determination unit 234 Behavior recognition unit 236 Behavior determination unit 238 Memory control unit 250 Behavior control unit 252 Control target 280 Communication processing unit 300 Server 400 Linked device

Claims (8)

a control unit that recognizes a user's behavior, determines its own behavior using history data updated based on the recognized user's behavior and information about the user, and controls a control target based on the determined own behavior;
An electronic device comprising: The control unit is
updating the stored history data when at least one of information regarding the user's experience and attribute information regarding the user changes as the information regarding the user;
2. The electronic device according to claim 1. The control unit is
updating the history data by associating the user's experience sensed through the five senses with the user's emotions as information about the user's experience;
3. The electronic device according to claim 2. The control unit is
updating the history data based on attribute information of the user including at least one of the user's name, age, sex, interests, concerns, hobbies, inclinations, lifestyle, personality, educational background, work history, place of residence, income, and family structure;
3. The electronic device according to claim 2. The control unit is
When a change in information about the user is recognized, an utterance is made to the user to confirm whether or not there is a change.
4. The electronic device according to claim 2 or 3. The control unit is
updating the history data when at least one of a change in information about the user is confirmed or an utterance is made from the user to the electronic device to inform the electronic device that there has been a change in information about the user;
3. The electronic device according to claim 2. The control unit is
recognizing changes in the user's experience or emotions through conversation with the user, and updating the historical data based on the changes in the user's experience or emotions;
4. The electronic device according to claim 2 or 3. The electronic device includes:
It is installed in the stuffed toy or connected wirelessly or by wire to a controlled device installed in the stuffed toy,
2. The electronic device according to claim 1.

Images