JP7392377B2 - Equipment, information processing methods, programs, information processing systems, and information processing system methods - Google Patents

Description

The present invention relates to a device, an information processing method, a program, an information processing system, and a method of the information processing system.

Various techniques have become known in recent years. For example, a video conferencing system that makes it possible to easily recognize who is speaking while displaying other meeting participants, and allows for natural and detailed recognition of video information about meeting participants. has been disclosed (for example, see Patent Document 1).

Further, in a system including a surveillance camera that images the entire surveillance area, a close-up camera that images a person's face, and a surveillance device, the surveillance device may 2. Description of the Related Art A technique has been disclosed in which a motion is detected and an image captured by a close-up camera is recorded according to the detected motion of a person (see, for example, Patent Document 2).

Furthermore, a robot device that can be moved freely by remote control has been disclosed (see, for example, Patent Document 3). Operators can view and hear the status of the meeting, show facial expressions, speak, present materials, and shake hands through a robot device placed in a remote conference room. can.

Japanese Patent Application Publication No. 7-336660 Japanese Patent Application Publication No. 10-336632 Japanese Patent Application Publication No. 2002-46088

Patent Document 1 merely discloses a video conference system, Patent Document 2 merely discloses a monitoring device that uses images, and Patent Document 3 discloses a video conferencing system that uses images while freely moving and transmitting information by remote control. It merely discloses a robotic device capable of interacting. However, when a device is used by a target (for example, a person), it is considered more convenient for the target if the processing by the device is executed after taking the target's emotions into consideration.

Therefore, it is desired to provide a technology that allows processing to be performed more conveniently for the target.

In order to solve the above problem, according to one aspect of the present invention, an emotion identification section that identifies the emotion of a target in the video based on images captured by a plurality of imaging sections; and a processing unit that performs processing corresponding to the identified emotion , the processing unit configured to perform processing corresponding to the identified emotion when there is a target whose emotion value of hostility is equal to or higher than a first threshold value. generates a free-viewpoint video in which a graphic in the shape of the device is rendered into a bird's-eye view of the device from diagonally above based on the images captured by the plurality of imaging units, and the free-viewpoint video is used by the operator. A device is provided that controls a communication unit to transmit information to an operating terminal of the device.

The processing unit may cause the device to perform a predetermined operation based on the position of the target when there is a target whose gaze emotion value among the emotions is equal to or higher than a second threshold. May be controlled.

The emotional value of the line of sight may be identified based on a positional relationship between the line of sight and the device.

The processing unit may control the moving unit so that the device approaches the target when an emotion value of a first emotion among the emotions satisfies a first condition.

The processing unit may control the communication unit so that the device and the operation terminal are connected when the emotion value of a second emotion among the emotions satisfies a second condition. .

The operation terminal connected to the device may be selected based on an emotion value of a third emotion among the emotions.

The processing unit may control a speaker so that the sound emitted by the operator received from the operation terminal is output after the device and the operator's operation terminal are connected.

The processing unit controls a display device included in the device so that content corresponding to the second condition is displayed when the emotional value of the second emotion satisfies the second condition. Good too.

The processing unit may control the communication unit so that the free viewpoint video is transmitted to the operation terminal when the emotional value of the second emotion satisfies the second condition.

According to another aspect of the present invention, the steps include: identifying an emotion of a target appearing in the video based on the video captured by a plurality of imaging units; and performing processing corresponding to the emotion ; When there is a target whose emotional value of hostility among the emotions is equal to or higher than the first threshold value, the above-mentioned object is added to an overhead image obtained by looking down on the device from diagonally above, based on the images captured by the plurality of imaging units. An information processing method is provided, which includes: generating a free-viewpoint video by rendering a graphic shaped like a device; and controlling a communication unit so that the free-viewpoint video is transmitted to an operator's operation terminal .

According to another aspect of the present invention, the computer includes an emotion identification unit that identifies the emotion of a target in the video based on images captured by a plurality of imaging units; and a processing unit that performs processing corresponding to the emotion , wherein the processing unit performs processing corresponding to the emotion when there is a target whose emotion value of hostility among the emotions is equal to or higher than a first threshold value. A free-viewpoint video is generated by rendering a graphic in the shape of the device onto a bird's-eye view of the device from diagonally above, based on images captured by a plurality of imaging units, and the free-viewpoint video is displayed on an operator's operating terminal. A program is provided for controlling the communication unit so that the communication unit functions as a device .

According to another aspect of the present invention, there is provided an information processing system including a device and a management center, wherein the device is configured to determine the emotion of a target depicted in the video based on the video captured by a plurality of imaging units. an emotion identification unit that identifies the emotion, and a processing unit that controls a communication unit so that the device and an operator's operation terminal are connected when a predetermined emotional value of the emotions satisfies a predetermined condition; If there is a target whose emotional value of hostility among the emotions is equal to or higher than a first threshold, the processing unit controls the device based on the images captured by the plurality of imaging units. Generates a free-viewpoint video by rendering a graphic in the shape of the device on an overhead video viewed diagonally from above, controls a communication unit so that the free-viewpoint video is transmitted to an operating terminal of an operator, and the management center An information processing system is provided, including a communication unit that selects the operator based on the emotion.

According to another aspect of the present invention, it is possible to identify the emotion of a target shown in the video based on the video captured by a plurality of imaging units, and to identify the emotion of a target shown in the video based on the video captured by a plurality of imaging units; If the conditions are met, the communication unit is controlled so that the device and the operation terminal of the operator selected based on the emotion are connected, and the emotion value of hostility among the emotions is a first threshold value. If there is a target as described above, a free-viewpoint video is created by rendering a graphic in the shape of the device into an overhead video that looks down on the device from diagonally above, based on the images captured by the plurality of imaging units. and controlling a communication unit so that the free-viewpoint video is generated and transmitted to an operating terminal of an operator .

As described above, according to the present invention, a technique is provided that allows processing to be performed more conveniently for the target.

1 is a diagram illustrating a configuration example of an information processing system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating an example of a configuration of a data table linking objects, positions, and emotions; FIG. 3 is a diagram illustrating an example of the configuration of a data table linking emotions and content. It is a figure which shows the example of a structure of the linking data table of an operator and an emotion. FIG. 3 is a diagram showing an example of an area monitoring screen displayed by a display unit. 2 is a flowchart illustrating an example of the operation of the information processing system according to the embodiment. 2 is a flowchart illustrating an example of the operation of the information processing system according to the embodiment. FIG. 2 is a diagram showing a hardware configuration of an information processing device as an example of a robot according to the embodiment. It is a figure showing an example of driving route information. FIG. 3 is a diagram showing an example of a current route and an alternative route expressed on a map. FIG. 3 is a diagram illustrating an example of obstacle discovery points expressed on a map. FIG. 3 is a diagram illustrating an example of a current route expressed in an image of a robot's line of sight. FIG. 6 is a diagram illustrating an example of route correction candidates expressed in a robot's line of sight image. FIG. 6 is a diagram illustrating an example of a current route after route correction expressed in an image of the robot's line of sight. 7 is a table showing an example of the correspondence between the identification result of whether a detour is possible and the action of the robot. FIG. 3 is a diagram illustrating an example of a current route expressed in an image of a robot's line of sight. FIG. 3 is a diagram illustrating an example of a current route expressed in a free viewpoint video. FIG. 3 is a diagram showing an example of a manual route expressed on a map. FIG. 3 is a diagram illustrating an example of an image of the line of sight of a robot traveling toward a target position along a manual route.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted.

Further, in this specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numbers after the same reference numeral. However, if there is no particular need to distinguish between a plurality of components having substantially the same functional configuration, only the same reference numerals are given. Further, similar components in different embodiments may be distinguished by attaching different alphabets after the same reference numerals. However, when it is not necessary to particularly distinguish between similar components of different embodiments, only the same reference numerals are given.

(1. Details of embodiment)
Next, details of embodiments of the present invention will be described.

(1-1. Explanation of system configuration)
FIG. 1 is a diagram showing a configuration example of an information processing system according to an embodiment of the present invention. As shown in FIG. 1, the information processing system 1 includes a robot 100 and a management center 160. The robot 100 and the management center 160 are connected to an IP (Internet Protocol) network 150 as an example of a network, and are configured to be able to communicate with each other via the IP network 150.

Here, the term robot generally refers to an "artificial human", "a device that performs actions and tasks similar to those of humans", "a device that performs actions and tasks based on instructions by computer operation", and "a device that performs autonomous actions and tasks". "A device operated by a computer", "A machine that has a manipulation function or a movement function under automatic control, can perform various tasks according to a program, and is used in industry" (JIS B 0134-1998), "A robot that provides services to humans" (JIS B 0187:2005) "An intelligent mechanical system that has the three elemental technologies of sensors, intelligence/control systems, and drive systems" (Robot Policy Study Group of the Ministry of Economy, Trade and Industry).

The robot 100 functions as an example of a device that executes processing according to the emotion of a target. In the embodiment of the present invention, it is mainly assumed that the object whose emotions are considered by the robot 100 is a person. However, the object for which emotions are considered may be an animal other than a human (for example, a dog or a cat), a machine (for example, a robot, etc.), or any other object whose emotion can be estimated. It may be. Furthermore, in the embodiment of the present invention, it is mainly assumed that there is one robot 100. However, a plurality of robots 100 may exist.

As shown in FIG. 1, the robot 100 includes a network 101, a monitoring section 102, a communication section 103, a communication section 104, an artificial intelligence section 105, a movement section 106, and a control database 141.

The network 101 relays communication between functional blocks inside the robot 100. The form of network 101 is not limited. For example, the network 101 may include a LAN (Local Area Network), a USB (Universal Serial Bus), and the like. The functional blocks inside the robot 100 may form a P2P (Peer to Peer) network on the network 101, and perform P2P communication between the functional blocks. In this case, the robot 100 has a node information management unit (not shown) that manages node information of each functional block (node), and the node information management unit receives connection destination node information when there is a request from the functional block. is given, and the functional block performs P2P connection and P2P communication based on the node information given from the node information management section. The node information management unit receives the registration of the node name, URI (Uniform Resource Identifier), communication port number, IP address, etc. for each functional block when each functional block is activated, and manages it as node information, for example, when connecting from the functional block. When a request is received using the destination node name, the node information is referred to using the node name, and the URI, communication port number, IP address, etc. corresponding to the node name are provided as node information to the request source functional block.

The monitoring unit 102 has a function of monitoring the surroundings. In the embodiment of the present invention, it is mainly assumed that the monitoring unit 102 includes an area monitoring camera 111, and the area monitoring camera 111 captures an image of the surrounding area. However, instead of or in addition to the area monitoring camera 111, the monitoring unit 102 may include a microphone that detects surrounding sounds, or may include an odor sensor that detects surrounding odors. or may include other sensors.

The communication unit 103 is configured with a communication interface and is connected to the IP network 150. For example, the communication unit 103 can communicate with the management center 160 via the IP network 150.

The communication unit 104 has a function of communicating with a target. More specifically, the communication unit 104 includes an input device that receives input from a target, and an output device that outputs output to the target. In the embodiment of the present invention, the communication unit 104 includes a touch panel 121 stacked with a detection device that detects an operation from a target and a display that presents visual information to the target, and a touch panel 121 that includes a detection device that detects an operation from a target and a display that presents visual information to the target. It is mainly assumed that the device includes a speaker 122 that presents a sound (sound), and a microphone 123 that detects a sound (such as a voice) emitted from a target. However, there is no limitation as to what kind of devices the communication unit 104 specifically includes.

The moving unit 106 includes wheels and a motor, and moves on the floor by rotating the wheels by the motor under the control of the artificial intelligence unit 105. As described above, although the embodiment of the present invention mainly assumes a case where the robot 100 can move on the floor, the robot 100 does not need to be able to move on the floor. For example, under the control of the artificial intelligence unit 105, the robot 100 may be able to move only a part of its body (for example, the head) while remaining stationary on the floor. Further, the robot 100 may be a flying object (drone) that moves in the air. In that case, the moving unit 106 is equipped with a propeller and a motor, and the motor rotates the propeller under control by the artificial intelligence unit 105. Move through the air by

The artificial intelligence unit 105 includes an arithmetic unit such as a CPU (Central Processing Unit), and its functions are performed by loading a program stored in a memory (not shown) into a RAM (Random Access Memory) and executing it by the CPU. It can be realized. At this time, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, the artificial intelligence unit 105 may be configured with dedicated hardware, or may be configured with a combination of multiple pieces of hardware.

More specifically, the artificial intelligence unit 105 determines the operation of the robot 100 based on the results of monitoring by the area monitoring camera 111. Then, the artificial intelligence unit 105 controls the robot 100 so that the robot 100 operates based on the determination result. As shown in FIG. 1, the artificial intelligence section 105 includes an emotion judgment section 131 and a behavior processing section 132.

The control database 141 has a data table 142 (FIG. 2) linking objects, positions, and emotions, and a data table 143 (FIG. 3) linking emotions and content. Note that the control database 141 is stored in a storage device. Here, the storage device stores a program for controlling the operation of the robot 100. The storage device also stores various data (including the control database 141 and node information managed by the node information management unit). The program includes an OS (Operating System), an application, and the like.

Based on the video captured by the area monitoring camera 111, the emotion determining unit 131 detects the position of the object shown in the video. In the embodiment of the present invention, a case is mainly assumed in which the number of objects detected by the emotion determining unit 131 is one. However, the emotion determining unit 131 may detect a plurality of objects. Then, the emotion determining unit 131 identifies the detected emotion of the target.

In the embodiment of the present invention, it is mainly assumed that the emotion determining unit 131 identifies various types of emotions of the target (emotions of interest, trouble, impatience, crying, hostility, and gaze) as examples of the target's emotions. do. In particular, feelings of hostility may include feelings of anger. Moreover, the emotion of crying may include the emotion of sadness. Furthermore, the emotion of gaze can correspond to the emotion of wanting to see something. However, the types of emotions identified by the emotion determining unit 131 are not limited to these. More specifically, the emotion determining unit 131 calculates the value of each type of emotion (emotion value).

The emotion determining unit 131 assigns a target ID to each detected target, calculates the processing order for each detected target (derives the processing order), and assigns the assigned target ID and the position of the target to each identified target. The emotion value of the type and the processing order are registered in the object-position-emotion association data table 142 (FIG. 2) of the control database 141. The processing order will be explained in detail later.

FIG. 2 is a diagram showing a configuration example of a data table 142 linking objects, positions, and emotions. As shown in FIG. 2, the data table 142 linking objects, positions, and emotions includes "object ID," "position," "interest," "trouble," "impatient," and "crying." , "hostility", and "gaze" are associated with their emotional values and "order."

“Target ID” is information for uniquely identifying a target. The "position" of an object is the three-dimensional position of the object. In the example shown in FIG. 2, the three-dimensional position of the object is expressed by a combination of X, Y, and Z coordinates. However, the three-dimensional position of the object is not limited to this format. Furthermore, the larger each emotion value is, the stronger the emotion of the type is. In the example shown in FIG. 2, it is assumed that the minimum emotional value is "0" and the maximum emotional value is "100". However, the minimum and maximum emotional values are not limited to these values.

In particular, the emotional value of "gaze" indicates the strength of the target's gaze toward the robot 100. Here, the strength of the gaze may be identified by the emotion determining unit 131 based on the positional relationship between the gaze and the robot 100. More specifically, in the embodiments of the present invention, it is mainly assumed that the strength of the line of sight is the amount of time that the target continuously looks at the robot 100 . However, the strength of the gaze is not limited to this example. For example, the strength of the line of sight may be a value related to the angle between the line of sight direction of the target and the direction from the target to the robot 100, or a value related to the distance between the position of the target and the position of the robot 100. good.

For example, it can be said that the smaller the angle between the direction of the target's line of sight and the direction from the target to the robot 100, the more strongly the target is directing its line of sight toward the robot 100. Therefore, the emotion determining unit 131 may calculate the emotion value of the "gaze" to be larger as the angle between the direction of the target's line of sight and the direction from the target toward the robot 100 is smaller. Alternatively, it can be said that the smaller the distance between the position of the target and the position of the robot 100, the stronger the target is directing its line of sight toward the robot 100. Therefore, the emotion determining unit 131 may calculate the emotion value of the "gaze" to be larger as the distance from the object to the robot 100 is smaller.

“Order” indicates the processing order by the robot 100. In the embodiment of the present invention, it is mainly assumed that the processing order is in descending order of the total value of all or part of the emotional values. In Figure 2, the object with the highest total emotional value (object ID = 30003) has the fastest processing order (“order” = 1), and the object with the next highest total value (object ID = 20002) has the highest processing order (“order” = 1). An example is shown in which the processing order is the next earliest ("order" = 2) and the processing order of the object with the smallest total value (object ID = 10001) is the latest ("order" = 3). However, the processing order is not limited to this example. For example, the processing order may be in descending order of the emotional value of any one.

The behavior processing unit 132 performs processing corresponding to the emotion identified by the emotion determination unit 131. According to this configuration, the robot 100 can perform processing that is more convenient for the target. Here, it is assumed that the behavior processing unit 132 executes only the process with the earliest processing order (highest value). However, the behavior processing unit 132 may execute only a predetermined number of processes in order of earliest processing order, or may execute all processes in order of earliest processing order.

For example, the behavior processing unit 132 may notify the operator's operation terminal 164 if there is a target whose emotion value of hostility among the emotions identified by the emotion determination unit 131 is equal to or higher than a first threshold. The communication unit 103 is controlled accordingly. The communication unit 103 reports to the management center 160 under the control of the behavior processing unit 132. According to such a configuration, an advantageous effect is obtained in that the operator can quickly monitor a target with which he feels hostile.

Here, the operating terminal 164 that receives the notification may be determined in advance or may be determined depending on the situation at that time (for example, the operating terminal 164 of an available operator may receive the notification. ). At this time, the behavior processing unit 132 controls the communication unit 103 so that the video captured by the area monitoring camera 111 and the position of the object with hostile feelings are transmitted to the operating terminal 164 along with the report.

In addition, if there is a target whose gaze emotion value among the emotions identified by the emotion judgment unit 131 is equal to or higher than the second threshold (for example, if the emotion value of hostility is the first If the emotional value of the line of sight is less than the threshold and the emotional value of the line of sight is greater than or equal to the second threshold, the robot 100 is controlled so that the robot 100 performs a predetermined action based on the position of the target. According to this configuration, it is possible to quickly notify the target that the robot 100 has recognized the existence of the target.

The predetermined motion based on the position of the target is not limited to a specific motion. For example, the predetermined action based on the position of the target may be an action in which the robot 100 moves its head so that the front side of the head of the robot 100 (for example, the face) is shown to the target, or an action in which the robot 100 It may be an action of moving the body of the robot 100 so that the front side (for example, the belly) of the body of the robot 100 is shown to the subject, an action of bringing the touch panel 121 closer to the subject, or any other motion. It's okay.

Furthermore, the behavior processing unit 132 is identified by the emotion determination unit 131 (for example, when the emotion value of hostility is less than a first threshold and the emotion value of gaze is less than a second threshold). The process can be determined based on the emotion and the emotion-content association data table 143 prepared in advance.

FIG. 3 is a diagram showing a configuration example of the emotion-content association data table 143. As shown in FIG. 3, the emotion-content association data table 143 includes information such as "threshold of interest", "threshold of trouble", "threshold of impatience", "threshold of crying", and content. are associated with each other. The behavior processing unit 132 controls so that the corresponding content is output when the value of each emotion (interest, trouble, impatience, crying) identified by the emotion determination unit 131 is equal to or greater than the corresponding threshold value.

The content may be HTML (HyperText Markup Language), XML (Extensible Markup Language), CSS (Cascading Style Sheets), or JSON (JavaScript (registered trademark) Ob It may be formed by combining descriptions in a format such as eject Notation). Furthermore, the content may be formed to include characters, moving images, sounds, and the like. Alternatively, a URL (Uniform Resource Locator) indicating the location of a resource related to the content may be set as the content, and the content may be controlled to be acquired and output based on the URL. Further, since the example shown in FIG. 3 is only an example of the association between emotions and content, the association between emotions and content is not limited to the example shown in FIG. 3.

In the example shown in FIG. 3, messages, audio, and center connection are listed as examples of content. However, the example shown in FIG. 3 is only an example of linking emotions and content. If the emotion value of the interest identified by the emotion determination unit 131 is equal to or higher than the “interest threshold,” the behavior processing unit 132 displays the message “Are you interested in 〇〇” as an example of the content on the touch panel? The touch panel 121 is controlled so as to be displayed by the touch panel 121. According to such a configuration, the effect that the operator can easily communicate with the target can be enjoyed. The specific content of the message is not limited.

Furthermore, the behavior processing unit 132 determines a condition (first condition) under which the emotion value of a certain emotion (first emotion) among the emotions identified by the emotion determination unit 131 requires the robot 100 to approach the target. If the conditions are satisfied, the moving unit 106 may be controlled so that the robot 100 approaches the target. According to this configuration, it is possible to enjoy the effect that it becomes possible to shorten the time until the robot 100 starts communicating with the target.

The condition that requires the robot 100 to approach the target may be that the emotion value of interest is equal to or higher than an "interest threshold." Alternatively, the condition that requires the robot 100 to approach the target may be a condition that another emotional value (for example, an emotional value of trouble, impatience, or crying) is equal to or higher than a corresponding threshold value.

The behavior processing unit 132 may cause the robot 100 to approach the target at a position where the target's line of sight falls (or so that the robot 100 enters the target's field of view). Furthermore, the robot 100 may approach the target in the same way each time, or may approach the target differently depending on the situation. For example, the speed at which the robot 100 approaches the target (the moving speed of the robot 100) may be the same each time, or may vary based on a certain emotional value of the target's emotions or the gender of the target. good. Alternatively, the route by which the robot 100 approaches the target may be the same each time, or may be changed based on a certain emotional value of the target's emotions, the gender of the target, or the like.

Further, if the emotional value of trouble identified by the emotion determining unit 131 is equal to or higher than the "troubled threshold", the behavior processing unit 132 displays a message "What did you do" as an example of the content on the touch panel 121. The communication unit 103 is controlled so that the communication unit 103 performs the operation “connection to the operation terminal of the operator of the management center” as an example of the content. The communication unit 103 transmits a connection request to the management center 160 under the control of the behavior processing unit 132. The connection request may include each emotion value. However, the conditions for connection to the operator's operation terminal 164 are not limited to this example.

That is, the behavior processing unit 132 satisfies the condition (second condition) that the emotional value of a certain emotion (second emotion) among the emotions identified by the emotion determining unit 131 needs to be handled by the management center 160. In this case, the touch panel 121 may be controlled so that the content corresponding to the condition is displayed, and the communication unit 103 may be controlled so that the robot 100 and the operator's operation terminal 164 are connected. According to such a configuration, the operator can enjoy the effect of being able to check the state of the target at an early stage.

At this time, the condition that needs to be handled by the management center 160 may be a condition that another emotional value (for example, an emotional value of interest, impatience, or crying) is equal to or higher than a threshold value. As will be explained later, the operator may be selected at the management center 160. After the robot 100 and the operator's operation terminal 164 are connected, the behavior processing unit 132 controls the speaker 122 so that the voice emitted by the operator received from the operation terminal 164 is output, and also controls the area monitoring. The communication unit 103 is controlled so that the video imaged by the camera 111 is transmitted to the operation terminal 164.

In addition, when the emotion value of impatience identified by the emotion determination unit 131 is equal to or higher than the “impatient threshold,” the behavior processing unit 132 displays the message “Are you looking for something” as an example of the content on the touch panel 121? The touch panel 121 is controlled so as to be displayed. According to such a configuration, the effect that the operator can easily communicate with the target can be enjoyed. The specific content of the message is not limited.

Further, the behavior processing unit 132 causes the speaker 122 to output audio (music) as an example of content when the emotional value of crying identified by the emotion determining unit 131 is equal to or higher than the “crying threshold”. The speaker 122 is controlled to. According to such a configuration, the effect that the operator can easily communicate with the target can be enjoyed. The specific content of the audio is not limited.

The management center 160 remotely manages the robot 100. In the embodiment of the present invention, it is mainly assumed that one robot 100 is managed by the management center 160. However, multiple robots 100 may be managed by the management center 160. As shown in FIG. 1, the management center 160 includes a gateway device 161, a network 162, a communication section 163, a management database 165, and a plurality of operation terminals 164.

Gateway device 161 is connected to IP network 150. For example, the gateway device 161 can communicate with the robot 100 via the IP network 150. More specifically, the gateway device 161 receives an operation detected by the touch panel 121, a sound detected by the microphone 123, a connection request, an image captured by the area monitoring camera 111, a target position, a target emotion, etc. , transmits the sound (eg, voice, etc.) emitted by the operator to the robot 100.

The network 162 relays communication between functional blocks within the management center 160. The form of network 162 is not limited. For example, the network 162 may include a LAN (Local Area Network).

The management database 165 has a data table 166 (FIG. 4) that associates operators with emotions. Note that the management database 165 is stored in a storage device. Here, the storage device stores a program for controlling the operation of the management center 160. The storage device also stores various data (including the management database 165). The program includes an OS (Operating System), an application, and the like.

The communication section 163 includes a connection control section 171. When the gateway device 161 receives the notification sent from the robot 100, the communication unit 163 sends the notification to the operation terminal 164. As described above, the operating terminal 164 that receives the report may be determined in advance, or may be determined depending on the situation at that time.

On the other hand, when the gateway device 161 receives a connection request transmitted from the robot 100, the communication unit 163 selects an operator based on the connection request. FIG. 4 is a diagram showing an example of the configuration of the data table 166 linking operators and emotions. As shown in FIG. 4, the operator-emotion association data table 166 includes "operator ID", "interest threshold", "trouble threshold", "impatient threshold", and "crying threshold". It becomes associated.

"Operator ID" is information for uniquely identifying an operator.

The “threshold of interest” is the threshold of the emotional value of interest. Similarly, the "threshold of trouble" is the threshold of the emotional value of trouble, the "threshold of impatience" is the threshold of the emotional value of impatience, and the "threshold of crying" is the threshold of the emotional value of crying. . Note that the types of emotions with which operators are associated with thresholds are not limited to these. Here, it is assumed that the minimum emotional value is "0", so in the example shown in FIG. 4, the operator ID "aaa" has only the interest threshold "50". The operator ID "bbb" is substantially associated with only the distress threshold "50", and the operator ID "ccc" is substantially associated with only the impatience threshold "50" and the crying threshold. are associated.

The communication unit 163 selects an operator based on the emotional value of a certain emotion (third emotion) among the emotional values included in the connection request. According to this configuration, an operator suitable for the target emotion can be selected. Here, it is assumed that the connection request includes emotional values of interest, trouble, impatience, and crying. If there is an operator ID whose emotional values for interest, trouble, impatience, and crying are equal to or higher than the corresponding threshold values in the operator-emotion association data table 166, the communication unit 163 stores the operator ID (operator ).

When the communication unit 163 selects an operator, the communication unit 163 controls the connection between the robot 100 and the operator's operating terminal 164 using the connection control unit 171. For example, the connection control by the connection control unit 171 may be realized by WebRTC (Web Real-Time Communication). When the robot 100 and the operating terminal 164 are connected, communication between the robot 100 and the operating terminal 164 is started (communication between the target and the operator is started).

The operation terminal 164 includes a control section (not shown). The control unit includes an arithmetic unit such as a CPU (Central Processing Unit), and its functions are realized by loading a program stored in a memory (not shown) into a RAM (Random Access Memory) and executing it by the CPU. obtain. At this time, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, the control unit may be configured by dedicated hardware or a combination of multiple pieces of hardware.

The operation terminal 164 includes a communication section 181, an operation section 182, and a display section 183. The communication unit 181 is configured with a communication interface and is connected to the IP network 150. For example, the communication unit 181 can communicate with the robot 100 via the IP network 150. The operation unit 182 detects an operation from an operator (for example, an operation to start communication, etc.).

When the communication unit 181 receives the report, the display unit 183 displays the image captured by the area monitoring camera 111. This provides the operator with the advantage of being able to quickly monitor targets with hostile feelings.

On the other hand, when the operation terminal 164 and the robot 100 are connected, the display unit 183 displays an enlarged image of the object (enlarged image) based on the image captured by the area monitoring camera 111 and the position of the object. indicate. Furthermore, the display unit 183 displays each emotion value of the target.

FIG. 5 is a diagram showing an example of the area monitoring screen 185 displayed by the display unit 183. As shown in FIG. 5, as an example, the display unit 183 displays an area monitoring screen 185. The area monitoring screen 185 includes an enlarged image of the target 301, and the robot 100 (corresponding to the robot 100) is also included in the enlarged image. The robot 100 may be located at the center of the enlarged image. Note that the robot 100 includes the moving unit 106 and the touch panel 121 as described above, and also includes cameras 111a to 111d as examples of the area monitoring camera 111. In the embodiment of the present invention, a free viewpoint video (an overhead video of the robot 100 (a free viewpoint around the robot 100) as displayed on the area monitoring screen 185 is based on the video captured by each of the cameras 111a to 111d. A so-called Flying View (registered trademark) image) is formed (rendered) in which a graphic in the shape of the robot 100 (a so-called avatar image) is rendered on the image (an image in which the robot 100 is viewed diagonally from above). Good too. Regarding the process of forming a free viewpoint image, for example, see Reference 1 (Oki Electric Industry Co., Ltd., Hidetaka Komai, "Flying View", [Online], INTERNET, [Retrieved October 8, 2019], <URL: You may apply the technology described in https://www.oki.com/jp/iot/doc/2018/img/ceatec/ceatec_flyingview.pdf>.The number of cameras and mounting positions are not limited.Target The ID of the target 301 and each emotional value are displayed near the area 301. The operator can view the area monitoring screen 185 displayed on the display unit 183 in this manner.

Note that when the operation unit 182 detects a communication start operation from the operator and the operation terminal 164 detects a sound (for example, a voice) emitted by the operator, the operation terminal 164 detects the sound emitted by the operator. The gateway device 161 is controlled so that the information is transmitted to the robot 100 via the gateway device 161. This allows the subject to hear the sounds emitted by the operator. On the other hand, when the sound emitted from the object is detected by the microphone 123 of the robot 100, the operation terminal 164 outputs the sound emitted from the object. This allows the operator to hear the sound emitted by the object.

The configuration of the information processing system 1 according to the embodiment of the present invention has been described above.

(1-2. System operation)
Next, the operation of the information processing system 1 according to the embodiment of the present invention will be described. 6 and 7 are flowcharts illustrating an example of the operation of the information processing system 1 according to the embodiment of the present invention. Note that, since the configuration of the information processing system 1 has been described in detail as above, the operation of the information processing system 1 will be explained here in a simpler manner than the explanation of the configuration of the information processing system 1.

First, as shown in FIG. 6, the area monitoring camera 111 of the robot 100 obtains an image by capturing an image of the surrounding area (S201). Next, the emotion determination unit 131 of the robot 100 creates data linking the object, the position (of the object), and the emotion (of the object) based on the image obtained by the area monitoring camera 111 (S202). The emotion determining unit 131 registers the data linking the object, the position (of the object), and the emotion (of the object) in the linking data table 142 (FIG. 2) of the object, position, and emotion.

The behavior processing unit 132 determines whether or not there is a target whose hostile emotion value is equal to or higher than the threshold value, based on data linking the target, the position (of the target), and the emotion (of the target) (S203).

If there is a target whose hostile emotion value is equal to or higher than the threshold (“YES” in S203), the communication unit 103 of the robot 100 communicates with the management center 160 via the IP network 150, and controls the operation of the management center 160. A notification is sent to the terminal 164 (S211). When the display unit 183 displays the image captured by the area monitoring camera 111, the operator of the operating terminal 164 can monitor the target with hostile feelings by viewing the image (S212).

If there is no target with a hostile emotion value equal to or higher than the threshold value (“NO” in S203), the behavior processing unit 132 performs processing based on the data linking the target, the position (of the target), and the emotion (of the target). , calculates the processing order of the target (derives the processing order) (S204). Then, the behavior processing unit 132 registers the processing order of the objects in the object-position-emotion association data table 142 (FIG. 2).

Next, the behavior processing unit 132 selects data with the highest processing order (data with the earliest processing order) from the data linking the object, the position (of the object), and the emotion (of the object) (S205). . Note that in the following operation, a case will be described in which only processing is performed for data with the highest processing order, but as described above, a predetermined number of processes may be performed in ascending order of processing order, All processes may be executed in ascending order of processing order.

Subsequently, as shown in FIG. 7, the behavior processing unit 132 determines whether the selected target is directing its line of sight toward the robot 100 (S206). Whether or not the target is directing its line of sight toward the robot 100 can be determined based on whether the emotional value of the target's line of sight is equal to or greater than a threshold value. If the behavior processing unit 132 determines that the target is directing its line of sight toward the robot 100 (“YES” in S206), the behavior processing unit 132 controls the robot 100 so that the robot 100 shows the front of the robot 100 to the target (S221). Instead, as described above, the behavior processing unit 132 may control the robot 100 to bring the touch panel 121 closer to the target. The behavior processing unit 132 proceeds to S207.

On the other hand, if the behavior processing unit 132 determines that the target is not directing its line of sight toward the robot 100 ("NO" in S206), the process proceeds to S207. When proceeding to S207, the behavior processing unit 132 determines whether the robot 100 needs to approach the target (S207). If the behavior processing unit 132 determines that the robot 100 needs to approach the target (“YES” in S207), it controls the moving unit 106 so that the robot 100 approaches the target (S222). The behavior processing unit 132 proceeds to S208.

On the other hand, if the behavior processing unit 132 determines that the robot 100 does not need to approach the target ("NO" in S207), the process proceeds to S208. When proceeding to S208, the behavior processing unit 132 determines whether the management center 160 needs to take action (S208). If the behavior processing unit 132 determines that the management center 160 needs to take action (“YES” in S208), the communication unit 103 communicates with the management center 160 via the IP network 150, and the communication unit 163 A connection is made to the operation terminal 164 of the operator selected based on the target emotion and the operator-emotion association data table 166 (FIG. 4) (S223).

Subsequently, the operating terminal 164 displays an enlarged image of the target on the area monitoring screen 185 (FIG. 5) based on the image captured by the area monitoring camera 111 and the position of the target (S224). This allows the operator to emit voice (for example, advice, etc.) to the target while monitoring the target. The behavior processing unit 132 of the robot 100 acquires data (processing judgment data) that matches each emotion value of the target from the emotion-content association data table 143 (FIG. 3). Then, the behavior processing unit 132 causes the content to be displayed on the touch panel 121 and outputs the operator's voice from the speaker 122 according to the acquired processing judgment data (S225).

On the other hand, if the behavior processing unit 132 determines that there is no need for the management center 160 to take any action (“NO” in S208), the behavior processing unit 132 determines each emotion value of the target from the emotion-content association data table 143 (FIG. 3). Obtain matching data (processing judgment data). Then, the behavior processing unit 132 displays the content on the touch panel 121 according to the acquired processing judgment data, and causes the speaker 122 to output a pre-registered voice (automatic voice) (S209).

The operation of the information processing system 1 according to the embodiment of the present invention has been described above.

(2. Hardware configuration example)
Next, an example of the hardware configuration of the robot 100 according to the embodiment of the present invention will be described. However, the hardware configuration example of the management center 160 according to the embodiment of the present invention can be similarly implemented.

An example of the hardware configuration of the information processing device 900 will be described below as an example of the hardware configuration of the robot 100 according to the embodiment of the present invention. Note that the example hardware configuration of the information processing device 900 described below is only an example of the hardware configuration of the robot 100. Therefore, in the hardware configuration of the robot 100, unnecessary configurations may be deleted from the hardware configuration of the information processing device 900 described below, or new configurations may be added.

FIG. 8 is a diagram showing a hardware configuration of an information processing device 900 as an example of the robot 100 according to the embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface. With 907 , an input device 908, an output device 909, a storage device 910, and a communication device 911.

The CPU 901 functions as an arithmetic processing device and a control device, and controls overall operations within the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus and the like.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. It is composed of etc. By operating the input device 908, a user operating the information processing device 900 can input various data to the information processing device 900 and instruct processing operations.

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 is configured with, for example, an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network. Further, the communication device 911 may support either wireless communication or wired communication.

The example hardware configuration of the robot 100 according to the embodiment of the present invention has been described above.

(3. Summary)
As described above, according to the embodiment of the present invention, the emotion determining unit 131 (emotion identification unit) identifies the emotion of the target in the video based on the video captured by the area monitoring camera 111 (imaging unit). and a behavior processing unit 132 (processing unit) that performs processing corresponding to the emotion identified by the emotion determination unit 131. According to this configuration, it becomes possible to perform processing that is more convenient for the target.

The behavior processing unit 132 may control the communication unit 103 to notify the operator's operation terminal 164 if there is a target whose emotional value of hostility among the emotions is equal to or higher than a first threshold value. . According to such a configuration, an advantageous effect is obtained in that the operator can quickly monitor a target with which he feels hostile. Furthermore, the behavior processing unit 132 may control the communication unit 103 so that the video is transmitted to the operation terminal 164 when there is a target whose emotional value of hostility is equal to or higher than the first threshold value.

The behavior processing unit 132 controls the robot 100 so that the robot 100 performs a predetermined action based on the position of the target when there is a target whose gaze emotion value is equal to or higher than a second threshold. You may. According to this configuration, it is possible to quickly notify the target that the robot 100 has recognized the existence of the target. Here, the emotional value of the line of sight may be identified based on the positional relationship between the line of sight and the robot 100.

The behavior processing unit 132 causes the robot 100 to approach the target when the emotional value of the first emotion among the emotions satisfies a first condition (a condition that requires the robot 100 to approach the target). The moving unit 106 may also be controlled. According to this configuration, it is possible to enjoy the effect that it becomes possible to shorten the time until the robot 100 starts communicating with the target.

The behavior processing unit 132 connects the robot 100 and the operating terminal 164 when the emotional value of the second emotion among the emotions satisfies a second condition (a condition that the management center 160 needs to respond to). The communication unit 103 may be controlled so as to Here, the operating terminal 164 connected to the robot 100 may be selected based on the emotional value of the third emotion among the emotions. According to this configuration, an operator suitable for the target emotion can be selected.

Here, after the robot 100 and the operator's operating terminal 164 are connected, the behavior processing unit 132 controls the speaker 122 so that the voice uttered by the operator received from the operating terminal 164 is output. good. The subject will be able to hear the sounds made by the operator. According to such a configuration, the effect that the operator can easily communicate with the target can be enjoyed.

When the emotional value of the second emotion satisfies the second condition, the behavior processing unit 132 controls the touch panel 121 (display device) included in the robot 100 so that content corresponding to the second condition is displayed. You may. According to such a configuration, the subject can view the displayed content. Alternatively, the behavior processing unit 132 may control the communication unit 103 so that the video is transmitted to the operation terminal 164 when the emotional value of the second emotion satisfies the second condition. According to this configuration, the operator can monitor the target video.

Although preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea stated in the claims. It is understood that these also naturally fall within the technical scope of the present invention.

For example, in the above case, the behavior processing unit 132 controls the operator's operation terminal 164 to be notified when there is a target with hostile feelings, and the image captured by the area monitoring camera 111 is displayed on the operation terminal 164. We have explained an example of controlling the data so that it is sent to At this time, the behavior processing unit 132 may control so that sensing data obtained by other sensors is also transmitted to the operating terminal 164. More specifically, the behavior processing unit 132 may activate other sensors around the object with hostile feelings to monitor the object with hostile feelings, or may already monitor the object using sensors. If there is another robot that has feelings of hostility, the sensor of the other robot may be used to monitor the object that has hostile feelings.

The robot 100 according to the embodiment of the present invention can perform tasks in various fields. For example, the robot 100 can perform tasks that were previously performed manually by security guards in the field of security services. For example, by having the robot 100 perform processing corresponding to the emotion of the target, the target can be easily monitored (for example, a lost target can be easily identified).

(robot autonomous running)
The robot 100 according to the embodiment of the present invention runs independently. For example, the robot 100 runs independently based on the travel route information shown in FIG. 9 . Here, the traveling route information is information formed based on map information, route information (route 1), initial position information (A), target position information (B), and the like. The map information may include the locations of floors and walls, and the like. The robot 100 may hold travel route information set in advance in a storage device, a microSD card (not shown), or the like, or may download travel route information via a network. Here, as shown in FIG. 9, the driving route information is two-dimensional information (information formed based on map information expressed vertically and horizontally, route information, initial position information, target position information, etc.) We mainly assume that However, the driving route information may be three-dimensional information (information formed based on map information expressed in length, width, height, route information, initial position information, target position information, etc.).

(Obstacle detection in autonomous robot running)
The robot 100 according to the embodiment of the present invention is capable of detecting obstacles on a running path. For example, the robot 100 may detect obstacles based on image information obtained by the area monitoring camera 111, or detect obstacles based on sensor information obtained by a sensor unit (ultrasonic sensor, millimeter wave radar, etc.). Detection may be performed, or obstacle detection may be performed based on imaging information and sensor information.

(Obstacle avoidance in robot autonomous running)
When the robot 100 according to the embodiment of the present invention detects an obstacle on the travel path, it attempts to avoid the obstacle. For example, when the robot 100 detects an obstacle on the current route (route 1 in FIG. 10), the robot 100 identifies route correction candidates that can avoid the obstacle on the current route (route 1 in FIG. 10) based on the traveling route information. Route correction that avoids obstacles within the current route (Route 1 in Figure 10), Static alternative route correction that identifies an alternative route set apart from the current route and avoids obstacles on the alternative route (Route 1 in Figure 10) 2) Check whether obstacles can be avoided by dynamic alternative route correction (Route 3 in Figure 10), which identifies an alternative route discovered by AI (Artificial Intelligent) in addition to the current route, and avoids obstacles using the alternative route. to decide.

Here, a description will be given of route correction that identifies route correction candidates that can avoid obstacles on the current route and avoids the obstacles on the current route. For example, assume that the robot 100 discovers an obstacle at a certain point (C in FIG. 11) while autonomously traveling along the current route (route 1 in FIG. 11) (FIG. 12). Then, based on the imaging information and sensor information, the robot 100 identifies the positional relationship between the current route it is traveling on and the obstacles, as well as the width and height of the passable space. It identifies passability and identifies route correction candidates that can avoid obstacles on the current route (FIG. 13). Then, the robot 100 performs route correction to avoid obstacles within the current route (route after route correction 1 in FIG. 14) according to route correction candidates that can avoid obstacles on the current route.

Here, a description will be given of dynamic alternative route correction in which an alternative route discovered by AI is identified separately from the current route and obstacles are avoided on the alternative route. For example, suppose that the robot 100 discovers an obstacle at a certain point (C in FIG. 11) while autonomously traveling along the current route (route 1 in FIG. 11) (FIG. 12). Then, based on the traveling route information, the robot 100 derives the shortest route leading from the point where the obstacle is discovered (C in FIG. 11) to the target position (B in FIG. 11). For example, the robot 100 uses AI to derive the shortest route from the point where the obstacle is discovered (C in FIG. 11) to the target position (B in FIG. 11) based on an RRT (Rapidly Exploring Random Tree) algorithm or the like. At this time, the robot 100 may derive the pure shortest route leading from the point where the obstacle is discovered (C in FIG. 11) to the target position (B in FIG. 11C), the shortest route that returns to the current route and leads to the target position (B in FIG. 11) while avoiding obstacles may be derived.

The robot 100 then performs dynamic alternative route correction (route 3 in FIG. 10) that identifies an alternative route discovered by AI in addition to the current route and avoids obstacles on the alternative route. Note that route 3 in Figure 10 is the pure shortest route leading from the point where the obstacle was discovered (C in Figure 11) to the target position (B in Figure 11). ) is the shortest route that returns to the current route while avoiding obstacles and leads to the target position (B in FIG. 11).

(Operator notification when an obstacle is detected during robot autonomous travel)
The robot 100 according to the embodiment of the present invention notifies the operator when an obstacle is detected. At this time, for example, the robot 100 may notify the operator when an obstacle is detected, regardless of whether the obstacle can be avoided. Furthermore, when notifying the operator, the robot 100 may notify the operator of the identified obstacle information (information representing obstacles such as cardboard, fallen trees, dangerous objects, etc.).

Here, the correspondence relationship between the identification result of whether a detour is possible and the action of the robot 100 is summarized as shown in FIG. 15.

(operator remote control)
The robot 100 according to the embodiment of the present invention detects obstacles, and when it becomes unable to move independently, it notifies the operator to that effect (about the inability to move independently and the detected obstacle), and remotely It may be possible to shift to an operation mode and allow the operator to set a detour route. The transition to the remote operation mode can be performed by the robot 100 notifying the operator's operation terminal 164 of transition to the remote operation mode, and the operator performing a transition operation on the operation terminal 164.

When shifting to the remote operation mode, the operator can monitor the surroundings of the robot 100 on the area monitoring screen displayed on the operation terminal 164, and can observe the state in which the robot 100 is placed. Here, in the remote control mode, the operator sets a detour route. For example, if the robot 100 discovers an obstacle at position C in FIG. and set a new route (manual route in Figure 18). In this case, the robot 100 moves from the position where the obstacle was discovered (C in Figure 11) to the initial position (A in Figure 11) based on the settings, and follows the new route (manual route in Figure 18) to the target position ( The vehicle travels toward B) in FIG.

Note that, after setting a new route in the remote control mode, the operator may view an image of the line of sight of the robot 100 traveling along the new route toward the target position. Figure 19 shows the movement from the position where the obstacle was found (C in Figure 11) to the initial position (A in Figure 11) and towards the target position (B in Figure 18) using a new route (manual route in Figure 18). 18. This is an image of the line of sight of the robot 100 as it runs, and is an image at the position A in FIG.

(Area monitoring screen)
The image displayed on the area monitoring screen may be an image of the line of sight of the robot 1000, or may be an image of a free viewpoint formed as described above. Alternatively, the operator may be able to switch on the area monitoring screen whether the image displayed on the area monitoring screen is an image of the line of sight of the robot 100 or an image of a free viewpoint.

1 Information processing system 100 Robot (equipment)
101 Network 102 Monitoring Department 103 Communication Department 104 Communication Department 105 Artificial Intelligence Department 106 Movement Department 111 Area Monitoring Camera 121 Touch Panel 122 Speaker 123 Microphone 131 Emotion Judgment Department 132 Behavior Processing Department 141 Control Database 142 Data linking objects, positions, and emotions Table 143 Data table linking emotions and content 150 IP network 160 Management center 161 Gateway device 162 Network 163 Communication unit 164 Operation terminal 165 Management database 166 Data table linking operators and emotions 171 Connection control unit 181 Communication unit 182 Operation Section 183 Display section 185 Area monitoring screen 301 Target

Claims (13)

an emotion identification unit that identifies the emotion of a target depicted in the video based on the video captured by the plurality of imaging units;
a processing unit that performs processing corresponding to the emotion identified by the emotion identification unit;
A device comprising:
When there is a target whose emotional value of hostility among the emotions is equal to or higher than a first threshold, the processing unit moves the device diagonally upward based on the images captured by the plurality of imaging units. generating a free-viewpoint video by rendering a graphic in the shape of the equipment on an overhead video viewed from above, and controlling a communication unit so that the free-viewpoint video is transmitted to an operating terminal of an operator;
device. The processing unit may cause the device to perform a predetermined operation based on the position of the target when there is a target whose gaze emotion value among the emotions is equal to or higher than a second threshold. Control,
The device according to claim 1. The emotional value of the line of sight is identified based on the positional relationship between the line of sight and the device,
The device according to claim 2. The processing unit controls the moving unit so that the device approaches the target when the emotional value of the first emotion among the emotions satisfies a first condition.
The device according to claim 1. The processing unit controls the communication unit so that the device and the operation terminal are connected when the emotion value of a second emotion among the emotions satisfies a second condition.
The device according to claim 1. The operating terminal connected to the device is selected based on the emotional value of a third emotion among the emotions.
The device according to claim 5 . The processing unit controls a speaker so that the sound emitted by the operator received from the operation terminal is output after the device and the operation terminal are connected.
The device according to claim 5 . The processing unit controls a display device included in the device so that content corresponding to the second condition is displayed when the emotional value of the second emotion satisfies the second condition.
The device according to claim 5 . The processing unit controls the communication unit so that the free viewpoint video is transmitted to the operating terminal when the emotional value of the second emotion satisfies the second condition.
The device according to claim 5 . identifying the emotion of a target depicted in the video based on the video captured by a plurality of imaging units;
performing processing corresponding to the emotion;
If there is a target whose emotional value of hostility among the emotions is equal to or higher than the first threshold, an overhead image of the device viewed from diagonally above is created based on the images captured by the plurality of imaging units. generating a free-viewpoint video by rendering a graphic shaped like the device, and controlling a communication unit so that the free-viewpoint video is transmitted to an operator's operation terminal;
information processing methods, including computer,
an emotion identification unit that identifies the emotion of a target depicted in the video based on the video captured by the plurality of imaging units;
a processing unit that performs processing corresponding to the emotion identified by the emotion identification unit;
A device comprising:
When there is a target whose emotional value of hostility among the emotions is equal to or higher than a first threshold, the processing unit moves the device diagonally upward based on the images captured by the plurality of imaging units. generating a free-viewpoint video by rendering a graphic in the shape of the equipment on an overhead video viewed from above, and controlling a communication unit so that the free-viewpoint video is transmitted to an operating terminal of an operator;
A program to function as a device . An information processing system having equipment and a management center,
The equipment includes:
an emotion identification unit that identifies the emotion of a target depicted in the video based on the video captured by the plurality of imaging units;
a processing unit that controls a communication unit so that when a predetermined emotional value of the emotions satisfies a predetermined condition, the device and an operator's operation terminal are connected;
When there is a target whose emotional value of hostility among the emotions is equal to or higher than a first threshold, the processing unit moves the device diagonally upward based on the images captured by the plurality of imaging units. generating a free-viewpoint video by rendering a graphic in the shape of the equipment on an overhead video viewed from above, and controlling a communication unit so that the free-viewpoint video is transmitted to an operator's operating terminal;
The management center is
comprising a communication unit that selects the operator based on the emotion;
Information processing system. identifying the emotion of a target depicted in the video based on the video captured by a plurality of imaging units;
If a predetermined emotional value among the emotions satisfies a predetermined condition, controlling a communication unit so that the device and an operating terminal of an operator selected based on the emotion are connected;
If there is a target whose emotional value of hostility among the emotions is equal to or higher than the first threshold, an overhead image of the device viewed diagonally from above is generated based on the images captured by the plurality of imaging units. generating a free-viewpoint video by rendering a graphic shaped like the device, and controlling a communication unit so that the free-viewpoint video is transmitted to an operating terminal of an operator;
A method for an information processing system, comprising:

Images