KR102519599B1 - Multimodal based interaction robot, and control method for the same - Google Patents

Abstract

Disclosed are a multimodal-based interaction robot and a control method thereof, capable of improving the satisfaction of a user about an interaction service. The interaction robot includes: an information collecting unit including multiple sensing units related to multiple different modalities one to one; an information output unit outputting interaction feedback related to a motion executed by the interaction robot; a driving unit controlling a movement of the interaction robot; and an information processing unit generating first situation recognition information expressing a recognized situation about at least one of the interaction robot and the user based on sensing data from at least one of the multiple sensing units. The information processing unit selects at least one interaction motion based on the first situation recognition information and controls at least one among the information output unit and the driving unit. The information processing unit also executes each interaction motion selected.

Description

Multimodal based interaction robot and its control method

The present invention relates to an interaction robot, and more particularly, to a technology for providing an interaction service to a user using a multimodal-based interaction robot having a multi-sensing structure corresponding to the multi-sensory functions of the human body.

This application is an application claiming priority to Korean Patent Application No. 10-2021-0147193 filed on October 29, 2021, and all contents disclosed in the specification and drawings of the application are incorporated into this application by reference.

Recently, the demand for and application of robot-based services, such as guidance robots, continues to be common, and in particular, as the number of households with one child and the elderly population increases rapidly, advanced interaction from the user's point of view Attempts to satisfy service needs through robots are also increasing.

Most of the existing robot-based services stay at the level of providing fragmentary responses or information related to the user's command or action (eg, question), which is the target of service, so that a higher level of interaction between the robot and the user is possible. The function to do so is lacking.

An interaction service is a service that can be provided through robot control based on a two-way interaction between a robot and a user, beyond a one-way interaction from a robot to a user or from a user to a robot.

However, the conventional method in which a robot interprets input information (eg, a user's voice) of a single modality in a single dimension, such as voice-to-text conversion, and performs related operations has limitations in satisfying the user's desire for natural conversation. distinct

The present invention has been made to solve the above problems, and an object of the present invention is to provide a multimodal-based interaction robot having a multi-sensing structure corresponding to the multi-sensory function of the human body, and a control method thereof.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the examples of the present invention. Furthermore, it will be readily apparent that the objects and advantages of the present invention may be realized by means of the instrumentalities and combinations thereof set forth in the claims.

A multimodal-based interaction robot according to an aspect of the present invention includes: an information collection unit including a plurality of sensing units that are one-to-one associated with a plurality of different modalities; an information output unit outputting interaction feedback related to the motion executed by the interaction robot; a driving unit controlling movement of the interaction robot; and an information processing unit configured to generate first context-aware information representing a recognized situation with respect to at least one of the user and the interaction robot, based on sensing data from at least one of the plurality of sensing units. do. The information processing unit is configured to select at least one interaction operation based on the first situation awareness information, and execute each selected interaction operation by controlling at least one of the information output unit and the driving unit.

The plurality of sensing units may include a first sensing unit that generates first sensing data including at least one of a 2D image, a 3D image, and a brightness value around the interaction robot. The information processing unit generates, based on the first sensing data, time-based-situation data representing object characteristics for each object located around the interaction robot, and based on the time-based-situation data, the first situation Cognitive information can be created.

The information processing unit may generate position data of at least one object around the interaction robot by comparing images from two cameras of the first sensing unit. The time-based-context data may include location data of the at least one object.

The plurality of sensing units may include a second sensing unit configured to sense sounds around the interaction robot and generate second sensing data. The information processing unit may generate hearing-based context data around the interaction robot based on the second sensing data, and generate the first contextual information based on the hearing-based context data.

The information processing unit may generate position data of at least one sound source around the interaction robot by comparing audio signals from at least two of the plurality of microphones included in the second sensing unit. The auditory-based context data may include location data of the at least one sound source.

The plurality of sensing units may include a third sensing unit generating third sensing data by detecting a contact event with respect to a predetermined region of a body of the interaction robot. The information processing unit may generate tactile-based context data of the interaction robot based on the third sensing data. Based on the tactile-based contextual data, the first contextual information may be generated.

The plurality of sensing units may include a fourth sensing unit configured to sense a standby state around the interaction robot and generate fourth sensing data. The information processing unit may generate olfactory-based context data of the interaction robot based on the fourth sensing data, and may generate the first context-aware information based on the olfactory-based context data.

The interaction robot may further include a power storage unit supplying power to the information collection unit, the information output unit, the driving unit, and the information processing unit. The plurality of sensing units may include a sixth sensing unit configured to sense a state of the power storage unit and generate sixth sensing data. The information processing unit may generate the first contextual information based on the sixth sensing data.

The interaction robot may further include a wireless communication unit that accesses at least one external information medium through a wireless communication network and collects second context awareness information expressing an undetectable external situation through the information collection unit. The information processing unit may select the at least one interaction operation further based on the second context awareness information.

The interaction robot may further include a heat dissipation member mechanically coupled between an outer surface of the interaction robot and the information processing unit so that thermal energy generated by the information processing unit is transferred to the body.

The control method of the interaction robot according to another aspect of the present invention includes collecting sensing data from at least one sensing unit among the plurality of sensing units; generating first context-aware information representing a recognized situation with respect to at least one of the user and the interaction robot, based on sensing data from the at least one sensing unit; selecting at least one interaction action based on the first contextual information; and executing each of the selected interaction operations by controlling at least one of the information output unit and the driving unit.

A computer program according to another aspect of the present invention executes the control method of the interaction robot.

According to at least one of the embodiments of the present invention, the multimodal-based interaction robot has a multi-sensing structure corresponding to the multi-sensory function of the human body, and the sensing data related to two or more modalities sensed through the multi-sensing structure are mutually supplemented. Overall, by recognizing the situation related to at least one of the user and the interaction robot at a level corresponding to the five human senses, the advanced interaction service (e.g., natural two-way conversation between the user and the robot) corresponding to the perceived situation is provided to the user. can be provided to

In addition, according to at least one of the embodiments of the present invention, external situation information that cannot be directly sensed through the multi-sensing structure provided in the interaction robot is collected through a wireless communication network, and the situation recognized using the multi-sensing structure. The user's satisfaction with the interaction service may be improved by combining information with external context information and selecting at least one interaction action provided to the user.

In addition, according to at least one of the embodiments of the present invention, by executing at least one interaction operation through collaboration between an IoT electronic device accessible through a wireless communication network and an interaction robot, interaction services that can be provided for each situation case can be diversified. there is.

When each interaction operation is executed, by additionally controlling the operation of an IoT electronic device accessible through a wireless communication network, it is possible to diversify the interaction services that can be provided for each situation case.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the claims.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and together with the detailed description of the present invention serve to further understand the technical idea of the present invention, the present invention is the details described in such drawings should not be construed as limited to
1 is a diagram showing the configuration of an interaction robot according to an embodiment of the present invention by way of example.
FIG. 2 is a view showing a front view of the interaction robot of FIG. 1 implemented in a humanoid form by way of example.
FIG. 3 is a view showing a rear side of the interaction robot of FIG. 1 implemented in a humanoid form by way of example.
4 is a flowchart exemplarily showing a method of controlling an interaction robot according to an embodiment of the present invention.
5 to 8 are diagrams referenced to describe various interaction operations executable through the control method according to the embodiment of FIG. 4 .
9 is a flowchart exemplarily showing a method of controlling an interaction robot according to another embodiment of the present invention.
10 and 11 are diagrams referenced to describe various interaction operations executable through the control method according to another embodiment of FIG. 9 .

Objects and technical configurations of the present invention and details of the operational effects thereof will be more clearly understood by the following detailed description based on the accompanying drawings in the specification of the present invention. An embodiment according to the present invention will be described in detail with reference to the accompanying drawings.

The embodiments disclosed herein should not be construed or used as limiting the scope of the present invention. It goes without saying that the description, including the embodiments herein, has a variety of applications for those skilled in the art. Therefore, any embodiments described in the detailed description of the present invention are illustrative for better explaining the present invention, and the scope of the present invention is not intended to be limited to the examples.

The functional blocks shown in the drawings and described below are only examples of possible implementations. Other functional blocks may be used in other implementations without departing from the spirit and scope of the detailed description. Also, while one or more functional blocks of the present invention are represented as separate blocks, one or more of the functional blocks of the present invention may be a combination of various hardware and software configurations that perform the same function.

Terms including ordinal numbers, such as first and second, are used for the purpose of distinguishing one of the various elements from the rest, and are not used to limit the elements by such terms.

In addition, the expression of including certain components simply indicates that the corresponding components exist as an expression of “open type”, and should not be understood as excluding additional components.

Furthermore, it should be understood that when a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but other components may exist in the middle. do.

1 is a view showing the configuration of an interaction robot according to an embodiment of the present invention by way of example, and FIG. 2 is a view showing the front side of the interaction robot of FIG. 1 exemplarily when implemented in a humanoid form, FIG. 3 is a view showing a rear side of the interaction robot of FIG. 1 implemented in a humanoid form by way of example.

Referring to FIGS. 1 to 3 , the interaction robot 100 includes an information collection unit 110 , an information output unit 120 , a driving unit 130 and an information processing unit 140 .

The information collection unit 110 includes a plurality of sensing units associated with a plurality of different modalities on a one-to-one basis, and is related to the internal and external environment of the interaction robot 100, that is, the situation of the space where the interaction robot 100 and the user as an interaction target are located. Provided to collect data. Hereinafter, it is assumed that the information collection unit 110 includes at least two of the first to sixth sensing units 111 to 116 described below.

The first sensing unit 111 generates first sensing data representing at least one of a 2D image, a 3D image, and a brightness value around the interaction robot 100 .

The first sensing unit 111 includes a wide-angle camera 211, a telephoto camera 212, an infrared sensor 213, an illuminance sensor 214, an ultrasonic sensor, a laser sensor, a radar sensor, a lidar sensor, and the like, ) may include at least one type of sensor that generates vision-based data that visually expresses a target area to be sensed, which is at least a part of a physical space where the object is located. A collection of time-based data generated for each sensor of the first sensing unit 111 may be referred to as first sensing data. The first sensing data may be used to determine whether or not at least one object located around the interaction robot 100 exists, extract object characteristics for each object, and the like.

Referring to FIG. 2 , at least one sensor of the first sensing unit 111 may be disposed on the head of the interaction robot. For example, the wide-angle camera 211 and the telephoto camera 212 may be disposed in the left eye and the right eye of the interaction robot 100, respectively. As another example, stereo cameras (a telephoto camera 212 and a wide-angle camera 211) are disposed in each of the left and right eyes of the interaction robot 100, and an infrared sensor 213 is disposed in the body of the interaction robot 100. It can be.

Based on the first sensing data, the information processing unit 140 may generate visual-based-situation data indicating object characteristics for each object located around the interaction robot 100 . In detail, the information processing unit 140 may process the first sensing data to recognize at least one object located in at least one sensing target area of the first sensing unit 111 . The information processing unit 140 may specify (identify) a user from among at least one recognized object. For example, the information processing unit 140 converts an object having characteristics matching pre-stored character characteristic information (eg, age, height, weight, gender, skin color, clothes, facial features, etc.) into an interaction service of the interaction robot 100. It can be identified as the user to whom it is provided.

The information processing unit 140 detects the user's specific body part (eg, the entire face, eyes, and torso) while detecting the user's position (eg, the direction relative to the direction of the interaction robot 100, the interaction robot 100) distance) can be calculated.

The information processing unit 140 may classify each object into a short distance object within a first reference distance or a long distance object outside the first reference distance from the interaction robot 100 . The information processing unit 140 may estimate a local situation based on object characteristics of each local object, and may estimate a far situation based on object characteristics of each far object. The information processing unit 140 may assign a higher priority to at least one interaction operation related to a local situation than at least one interaction operation related to a remote situation. The information processing unit 140 may sequentially execute a plurality of interaction operations according to the priority of each interaction operation. The information processing unit 140 compares an image from one of the two cameras (eg, the wide-angle camera 211) with an image from the other one (eg, the telephoto camera 212), and overlaps the two images from the two images. By extracting two partial images of the physically captured physical area and adjusting the scaling so that the two partial images have the same size and ratio using a distance estimation model learned through deep learning, Location data representing the location of at least one object may be generated. The location data of the object represents the distance between the corresponding object and the interaction robot 100 and the direction of the corresponding object with respect to the interaction robot 100, and may be included in the visual-based situation data.

The information processing unit 140 processes the first sensing data to obtain information expressed in a user's face image (eg, eyes, eyebrows, nose, mouth, complexion, etc.) or body motion (eg, jumping, hurray, squatting). The user's emotional state can be multi-layered and analyzed. For example, the user's emotional state is first classified as one of a positive state, a negative state, and a stable state, and then the positive state is optionally secondarily classified into joy, pleasure, comfort, hope, etc., and the negative state is anger and sadness. , pain, excitement, etc. are classified as secondary, and the stabilization state can be classified as concentration, sleep, daily life, etc. as secondary.

The second sensing unit 112 detects sounds around the interaction robot 100 and generates second sensing data. The sound detected by the second sensing unit 112 may be generated by a single sound source inside or outside a specific space where the interaction robot 100 and the user are located, or may be simultaneously generated by two or more sound sources.

The second sensing unit 112 may include a plurality of microphones 221 (hereinafter referred to as 'microphones 221'). At least two of the plurality of microphones 221 may be arranged to be spaced apart from each other by a predetermined distance or more. A collection of audio signals generated for each microphone of the second sensing unit 112 may be referred to as second sensing data. The second sensing data may indicate sound pressure, intensity, amplitude, frequency, pitch, wavelength, period, and the like of sound generated for each of at least one sound source. The second sensing data is identification of the location (eg, direction, distance) of each sound source, type of each sound source (eg, person (eg, male, female, adult, child), animal (eg, dog, cat), object (eg, , vacuum cleaner, TV), identification of external environment (eg, thunder, rain, hail)), sound content (eg, conversation, song, arguing, snoring), and recognition of voice commands from users. . Referring to FIG. 2 , like the first sensing unit 111 , at least one microphone 221 of the second sensing unit 112 may be disposed on the head of the interaction robot 100 . For example, one microphone 221 of the second sensing unit 112 may be disposed in the left ear of the interaction robot 100 and the other microphone 221 may be disposed in the right ear of the interaction robot 100 .

The information processing unit 140 may generate auditory-based context data around the interaction robot 100 based on the second sensing data. The information processing unit 140 processes the second sensing data through voice-to-text conversion, voice-to-emotion conversion, text-to-emotion conversion, etc., similarly to the above-described first sensing data, so that the user's voice (eg, laughter) , sigh, admiration) can be multi-layered and analyzed. For example, when the user utters "You're handsome", the information processing unit 140 obtains the text "You're handsome" from the spoken voice, and the characteristic items of the spoken voice (e.g., tone, tone, tone, nuance) , frequency, etc.) and/or the user's facial expression, it is possible to estimate whether the content of the spoken voice is an expression of praise or an expression of irony.

The information processing unit 140 compares audio signals from at least two microphones 221 among the plurality of microphones 221 included in the second sensing unit 112, and the location of at least one sound source around the interaction robot 100. data can be generated. The location data of the sound source may be included in the auditory-based context data. For example, the information processing unit 140 calculates a numerical value (eg, sound pressure level) of a characteristic item representing the sound detected by each of the plurality of microphones 221, and calculates the numerical value calculated for the same characteristic item. Compare, calculate the position of the sound source based on the difference between them, and direct the face of the head of the interaction robot 100 to the direction in which the sound associated with the relatively high numerical value is detected (ie, the position of the sound source), The driving unit 130 may be controlled.

The third sensing unit 113 detects a contact event with respect to a predetermined area of the body of the interaction robot 100 and generates third sensing data. The third sensing unit 113, such as the touch sensor 231 and the strain gauge 232, is the area, size (strength), direction, etc. of a contact event in which a human body or an object contacts a predetermined area of the interaction robot 100. It may include at least one sensor that generates tactile-based data representing The third sensing data may be used to recognize a contact-type command from a user, determine whether or not a collision occurs between the interaction robot 100 and an obstacle, and the like. Referring to FIG. 2 , at least one touch sensor 231 is disposed on each of the left arm, left hand, right arm, right hand, and head of the interaction robot 100, and a strain gauge is placed on the back of the body of the interaction robot 100. 232 may be placed.

The information processing unit 140 may generate tactile based-situation data of the interaction robot 100 based on the third sensing data. In detail, the information processing unit 140 may process the third sensing data to detect a contact-type command or action from the user or a collision event with a nearby object. For example, when a user applies a touch action to the upper part of the head of the interaction robot 100 where the touch sensor 231 is disposed, the information processing unit 140 generates a pattern of the corresponding touch action (e.g., tapping twice in a row, back and forth) stroking), and according to the identified pattern, the meaning of the user's contact command or action (eg, praise, power on, power off) can be estimated. The information processing unit 140 may execute at least an interaction operation related to the estimated meaning.

The fourth sensing unit 114 detects a standby state around the interaction robot 100 and generates fourth sensing data. The fourth sensing unit 114, such as the air quality sensor 241, the temperature sensor 242, the humidity sensor 243, etc., the atmospheric state (e.g., the concentration of carbon monoxide, carbon dioxide, yellow dust, fine dust, harmful gases, smoke, At least one type of sensor that detects air temperature, air humidity, etc.) and generates olfactory-based data may be included. A collection of olfactory sense-based data generated for each at least one sensor of the fourth sensing unit 114 may be referred to as fourth sensing data. Referring to FIGS. 2 and 3 , at least one sensor (eg, 242 and 243) of the fourth sensing unit 114 may be disposed on the front/rear surface of the head or body of the interaction robot 100.

The information processing unit 140 may generate olfactory based-situation data of the interaction robot 100 based on the fourth sensing data. The olfactory-based context data can be comprehensively analyzed with context data having other modalities and used to improve estimation accuracy for real situations.

The fifth sensing unit 115 detects motion of the interaction robot 100 and generates fifth sensing data. The fifth sensing unit 115 corresponds to bodily functions (eg, exercise, life, balance of posture) in charge of the cerebellum of the human body, such as a GPS receiver, a speed sensor, a gyro sensor, an acceleration sensor, and the like, the interaction robot 100 ) may include at least one type of sensor that generates motion-based data representing movement (eg, position, speed, direction, body tilt). At least one sensor of the fifth sensing unit 115 may be accommodated inside the body of the interaction robot 100 .

The information processing unit 140 processes the fifth sensed data and moves the interaction robot 100 (e.g., change in position and direction according to its own movement, change in position and direction due to other moving objects, stop, fall). ), and can selectively control the driving unit 130 accordingly.

The information output unit 120 outputs interaction feedback related to an operation executed by the interaction robot 100 . The interaction feedback may be a kind of signal fed back to the user in a visual, auditory and/or tactile form. The information output unit 120 may include at least one type of output means such as a speaker 121, a display 122, an optical output device 123, and a vibration motor. For example, referring to FIG. 2, a speaker 121 is disposed within a predetermined distance from a microphone 221 disposed on both sides of the head of the interaction robot 100, and a display 122 is provided on the body of the interaction robot 100. is disposed, and the light output device 123 may be disposed in at least one of the left eye and the right eye of the interaction robot 100.

The information processing unit 140 compares the ambient brightness value of the interaction robot 100 detected by the illuminance sensor 214 with the reference brightness value, and the light output device 123 operates according to the difference between the ambient brightness value and the reference brightness value. At least one of the light amount of the output optical signal and the screen brightness of the display 122 may be controlled. For example, at least one of the amount of light of the light signal and the screen brightness of the display 122 may be reduced in a dark environment, and vice versa in a bright environment. Accordingly, the visibility of the interaction robot 100 for the user is increased, the image quality of the wide-angle camera 211 and the telephoto camera 212 is improved, and electrical energy stored in the power storage unit 160 is not consumed unnecessarily. can do.

The driving unit 130 may include a plurality of power joints 131 to 136 for controlling the movement of the interaction robot 100 . Each electric joint mechanically connects the two physical components of the interaction robot 100 and controls the relative posture of one of the two physical components to the other.

The first power joint 131 connects the head and body of the interaction robot 100 . The second power joint 132 connects the left arm and the left hand. The third power joint 133 connects the right arm and the right hand. The fourth power joint 134 connects the left arm and the body. The fifth power joint 135 connects the right arm and the body. The sixth power joint 136 and the seventh power joint 137 correspond to the left leg and the right leg coupled to the body of the interaction robot 100 .

Each power joint may have a power rotation joint structure having one or more degrees of freedom in which movement occurs by a driving force of a motor. For example, various head actions (eg, nodding, tilting, doridori, bowing) are performed by the first power joint 131, and various hand actions are performed by the second power joint 132 and the third power joint 133 ( For example, shaking hands, grasping an object, greeting a hand, etc.) can be performed, and various arm actions (eg, lifting an object gripped by the hand, extending an arm) can be performed by the fourth power joint 134 and the fifth power joint 135 etc.) can be performed.

In each of the sixth power joint 136 and the seventh power joint 137, the driving force of the motor provided thereto is transmitted to the ground through a friction means (eg, a support plate) provided on the leg, and the interaction robot 100 with respect to the ground It can be in charge of functions such as moving back and forth, turning in place, or jumping.

The interaction robot 100 may further include a power storage unit 160 . The power storage unit 160 supplies power to at least one of the information collection unit 110, the information output unit 120, the driving unit 130, and the information processing unit 140 of the interaction robot 100. The power storage unit 160 includes at least one rechargeable battery, and electrical energy stored in each battery is transmitted to at least one of the information collection unit 110, the information output unit 120, the driving unit 130, and the information processing unit 140. Used for one drive. The information processing unit 140 may monitor the power storage unit 160 and generate data representing the state of the power storage unit 160 (eg, battery charge level, remaining battery life, battery temperature). According to the state of the power storage unit 160, the information processing unit 140 generates user guide information for long-term use of the power storage unit 160 (eg, an appropriate amount of charge, an appropriate number of times of charging), and outputs the information to the information output unit 120. can be output to the user.

The information collection unit 110 may further include a sixth sensing unit 116 for detecting a state of the power storage unit 160 . For example, the sixth sensing unit 116 detects at least one of items indicating the state of the power storage unit 160, such as battery temperature, battery voltage, battery current, battery charge amount, and remaining battery life, to obtain sixth sensing data. can create The information processing unit 140 evaluates the quality of the power storage unit 160 (eg, in the range of 0 to 100) based on the sixth sensing data, and sets the power storage unit 160 to normal mode, power saving mode, and sleep mode. can be controlled by either one. The normal mode may be a mode in which all functions (operations) of the interaction robot 100 are performed without restriction when the goodness level is the first level (eg, 80 to 100), and the power saving mode is when the goodness level is the second level (eg, 80 to 100). For example, 60 to 80), it may be a mode that limits (disables) some non-essential functions of the interaction robot 100, and the sleep mode is an interaction robot when the goodness level is the third level (eg, ~60). It may be a mode for disabling all non-essential functions except for the essential functions of (100). Data describing essential and non-essential functions may be pre-recorded in the information storage unit 141 . Essential functions (eg, battery charging function) and non-essential functions (eg, conversation function) may be registered/changed/deleted by the user. Alternatively, the information processing unit 140 may autonomously register/change/delete at least one of essential functions and non-essential functions as a management operation associated with at least one of the first contextual information and the second contextual information. .

The interaction robot 100 may further include a wireless communication unit 150 . The wireless communication unit 150 accesses at least one external information medium through a wireless communication network to provide second context-aware information (e.g., external weather, Collect the latest news, trends, etc.) by field.

The interaction robot 100 may further include a heat dissipation member 180 . The heat dissipation member 180 is for suppressing an excessive rise in temperature of the information processing unit 140, so that the thermal energy generated in the information processing unit 140 is discharged to the outside of the interaction robot 100, of the interaction robot 100. It is mechanically coupled between the outer surface and the information processing unit 140 . For example, the heat dissipation member 180 includes a first heat pipe 181 having one end coupled to the information processing unit 140 and the other end coupled to the outer surface of the body of the interaction robot 100, and the information processing unit 140 at one end. and at least one of a second heat pipe 182 having the other end coupled to the outer surface of the head of the interaction robot 100 and a heat dissipating hole 183 having at least one hole for heat dissipation. In this case, the first metal material (eg, aluminum alloy) constituting the first heat pipe and the second metal material (eg, copper alloy) constituting the second heat pipe are based on the position of the information processing unit 140. It may have a difference in thermal conductivity corresponding to a difference between the distance to the outer surface of the body and the distance to the outer surface of the head. For example, the first metal material may be an aluminum alloy, and the second metal material may be a copper alloy having higher thermal conductivity than the first metal material. In this case, the temperature felt when the user touches the skin on the outer surface of the head of the interaction robot 100 and the temperature felt when the user touches the skin on the outer surface of the body of the interaction robot 100 are approximately the same, so that the heat generation of the interaction robot 100 Users' confusion about levels can be reduced. The user determines whether the interaction robot 100 is in a normal state or a fatigue state (eg , overload of the information processing unit 140) or the like.

The interaction robot 100 may further include at least one function port 180 (eg, a USB port) in addition to the charging port for the power storage unit 160 . The size, manufacturing cost, sales/rental price, etc. of the interaction robot 100 may be limited. In addition, the types of sensors required to execute functions necessary for the user are also different according to the user's personality, life pattern, gender, life cycle, and the like. Therefore, instead of all of the above-described sensors, some sensors are implemented to be included as components of the interaction robot 100, but at least one of the other sensors (eg, human body temperature sensor, heart rate sensor, harmful substance sensor, etc.) is necessary. It is provided in a modular form so as to be connectable to the function port 170 of the interaction robot 100 by selectively expanding/reducing the detectable items and situational awareness capabilities of the interaction robot 100 to increase software and hardware resources of the interaction robot 100. can be operated efficiently. The information processing unit 140 may recognize each sensor connected to at least one function port 170 and activate at least one function (eg, data storage, related operation) related to each recognized sensor. For example, the information processing unit 140 may record the user's body temperature value sensed by the human body temperature sensor connected to the function port 170 in the information storage unit 141 by mapping the sensing time. As another example, the information processor 140 may execute a heartbeat-based health management application (app service) in response to the heartbeat sensor being connected to the function port 170 .

The information processing unit 140 is operably coupled to each component included in the interaction robot 100 and controls each component. The information processing unit 140, in terms of hardware, includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and microphones. As 221, it may be implemented to include at least one of microprocessors and electrical units for performing other functions.

The information storage unit 141 stores a computer program including instructions for causing the interaction robot 100 to perform a control method related to an interaction service described below. The information storage unit 141 includes a flash memory type, a hard disk type, a solid state disk type (SSD type), a silicon disk drive type (SDD type), and a multimedia card microphone 221. Low type (multimedia card micro type), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read-only memory), PROM (programmable read-only memory) may include at least one type of storage medium. The information storage unit 141 may be provided in the information processing unit 140 .

The information processing unit 140 performs situational awareness by combining sensing data from at least two of the plurality of sensing units 111 to 116, thereby performing situational awareness with only the sensing data from a single sensing unit. In comparison, the quality of the interaction service provided to the user can be improved. For example, when a first situation and a second situation are recognized based on the sensing data of one sensing unit, and a second situation and a third situation are recognized based on the sensing data of another sensing unit, the information processing unit 140 An interaction operation related to the commonly recognized second situation may be executed prior to an interaction operation related to the first situation or the third situation.

4 is a flowchart exemplarily showing a method of controlling an interaction robot according to an embodiment of the present invention.

1 to 4 , in step S410, the information processing unit 140 collects sensing data from at least one sensing unit among the plurality of sensing units 111 to 116.

In step S420, the information processing unit 140 generates first context-aware information representing a recognized situation with respect to at least one of the user and the interaction robot 100, based on the sensing data collected from at least one sensing unit. . The first context awareness information may be further based on state data of the power storage unit 160 . Preferably, the information processing unit 140 applies the artificial intelligence-based contextual awareness model previously stored in the information storage unit 141 to the sensing data collected from at least two sensing units, and obtains data from the at least two sensing units from the contextual awareness model. At least one of a user-only involvement situation, a robot-only involvement situation, and a user-robot complex involvement situation expressed by the collected sensing data may be estimated. A user-only involvement situation is a situation that concerns only the user and is not related to the robot. Conversely, the robot-only involvement situation is a situation that is related only to the interaction robot 100 and is not related to the user. Next, the user-robot complex involvement situation may be a situation in which both the user and the interaction robot 100 are involved.

In step S430, the information processing unit 140 selects at least one interaction action based on the first contextual information. That is, there are a plurality of interaction actions that can be provided by the interaction robot 100, and one or two or more actions related to the first contextual information are selected from among the plurality of interaction actions.

In step S440, the information processing unit 140 controls at least one of the information output unit 120 and the driving unit 130 to execute each interaction operation selected in step S430. That is, each interaction operation can be performed by the information output unit 120 alone, the drive unit 130 alone, or a cooperative action between the information output unit 120 and the drive unit 130. can

5 to 8 are diagrams referenced to describe various interaction operations executable through the control method according to the embodiment of FIG. 4 .

5 is a diagram referenced to describe a user following operation, which is one of various interaction operations. Referring to FIG. 5 , the information processing unit 140 performs a user following operation for a specific object based on the location of the specific object (eg, user) obtained by processing at least one of the first sensing data and the second sensing data. In order to execute, the driving unit 130 may be controlled. For example, the information processing unit 140 compares the location of the user estimated based on the first sensing data with the location of the sound source estimated based on the second sensing data, and determines that the user and the sound source match each other. , users can be set as followers. The information processing unit 140 monitors the spoken voice of the surrounding space with the second sensing unit 112 activated in the sleep mode, and then the voice call word 510 directed to the interaction robot 100 (sometimes referred to as 'hotword') ), it is possible to provide an actual interaction service such as following the user by switching from the sleep mode to the wake mode.

The information processing unit 140 controls the wide-angle camera 211 of the first sensing unit 111 on the condition that a person close to the interaction robot 100 is detected by the infrared sensor 213 of the first sensing unit 111. and the telephoto camera 212 by switching from an inactive state to an active state, so that the wide-angle camera 211 and the telephoto camera 212, which consume more power than the infrared sensor 213, are always activated. ) of electrical energy can be saved. In addition, when the interaction robot 100 is in the sleep mode, the information processing unit 140 recognizes a behavioral situation in which the user utters while looking at the interaction robot 100 even if the voice call word 510 is not detected. In this case, the same operation as when the voice paging word 510 is detected, that is, an operation of switching from the sleep mode to the wake-up mode may be performed. Accordingly, it is possible to reduce unnaturalness and discomfort caused by having to repeatedly utter the voice paging word 510 whenever the user wants to activate the interaction robot 100 from the sleep mode. For reference, the sleep mode refers to a state in which only some pre-determined functions necessary for detecting a user's call command are activated and the remaining functions are deactivated. The activation mode refers to a state in which all functions executable by the interaction robot 100 are activated without limitation. Each mode may be selected according to a user command or automatically selected when a pre-given condition related to the corresponding mode is satisfied.

The user following operation includes at least one of user gazing and following movement.

The gaze of the user may be performed by the information processing unit 140 controlling the first electric joint 131 so that the facial part of the head of the interaction robot 100 gazes at the user. When the rotation angle of the head adjustable by the first power joint 131 (eg, the range available for panning) is limited, the information processing unit 140 additionally provides the sixth power joint 136 and the seventh power joint 137. It can be controlled so that the facial part of the head of the interaction robot 100 is directed toward the user.

The following movement is performed by the information processing unit 140 controlling the sixth power joint 136 and the seventh power joint 137 so that the interaction robot 100 moves to a proximate position within a second reference distance from the user. can

The information processing unit 140 may generate a conversation message 520 related to the user following operation (for example, "Do you have something to say to me?") and output the conversation message 520 to the user through the information output unit 120. there is.

FIG. 6 is a diagram referenced to describe an operation of creating conversation content, which is one of various interaction operations. The information processing unit 140 may generate a dialog message that is fed back to the user in response to the current situation by applying the artificial intelligence-based dialog generation model to context data associated with at least two of the plurality of modalities described above.

For example, when the visual-based context data represents a situation in which the user is in the kitchen, the auditory-based context data represents a situation during eating, and the olfactory-based context data represents a situation in which a harmful gas is detected at a certain concentration or higher, the user Even if a conversation request from does not occur, preemptively, "It looks like you are eating. Did you turn off the gas stove by any chance?" The information output unit 120 may be controlled to output, through at least one of the speaker 121 and the display 122, a conversation message 610 having content corresponding to the recognized current state, such as the like. As described above, since the current situation is estimated by synthesizing situation data having two or more different modalities, situation estimation accuracy can be increased. In addition, since the content of the conversation message is generated according to the estimated situation, it is possible to provide a natural conversation service with the user compared to a method using only situation data having a single modality, and as a result, satisfy the satisfaction expected by the user. can make it

FIG. 7 is a diagram referenced to describe a dialog sound control operation, which is one of various interaction operations. The information processing unit 140 applies the artificial intelligence-based voice control model to situation data associated with at least one of the plurality of modalities described above, and provides the user with the speaker 121 of the information output unit 120 in response to the current situation. You can adjust the sound effect that is fed back to the user through this.

The information processing unit 140 may adjust the volume of the conversation message output to the user through the speaker 121 according to the distance between the interaction robot and the user. Here, relational data between the interaction robot-user distance and the volume may be previously stored in the information storage unit 141 . For example, when the robot-user distance is equal to or greater than the first distance (eg, 5 m), a dialogue message 710 having a volume corresponding to 1.2 times the reference value is output, and the robot-user distance is the second distance (eg, 1 m). If it is less than the reference value, a conversation message 720 with a volume corresponding to 0.8 times the reference value is output, and if the robot-user distance is between the first distance and the second distance, the conversation message may be output with a volume corresponding to the reference value. . As another example, when the auditory-based situation data indicates a situation in which noise is generated at a predetermined level or higher (eg, a vehicle in which the interaction robot 100 is riding is driving at high speed and a loud wind noise is generated), a conversation message equal to the size of the noise volume can be increased. Accordingly, it is possible to reduce discomfort felt by the user when the volume of the voice message is too low or too high, and unnecessary consumption of electrical energy stored in the power storage unit 160 can be reduced.

8 is a reference diagram for explaining an adaptive power saving operation, which is one of various interaction operations.

When a plurality of interaction actions are selected based on the first situation awareness information, the information processing unit 140 executes all of the plurality of interaction actions or only some of the plurality of interaction actions according to the quality of the power storage unit 160 . It is possible to execute and not execute the rest, or lower the execution level of at least one of the plurality of interaction operations.

The information processing unit 140 designates priorities among a plurality of interaction operations based on the first situation awareness information, and has a certain level of priority corresponding to the quality level of the power storage unit 160 among the plurality of interaction operations. Only interaction actions can be executed.

8 illustrates a case in which three interaction actions (user gaze, following movement, and outputting a conversation message) are selected based on the first context awareness information. First, if the quality level of the power storage unit 160 is the first level, the information processing unit 140 can execute all three interaction operations without reducing the execution level. At this time, the conversation message may be output through both the speaker 121 and the display 122 . Next, if the quality level of the power storage unit 160 is the second level, the information processing unit 140 disables or lowers the execution level of any one of the three interaction operations (eg, follow-up movement), and the other two interaction operations ( eg, user gaze and dialog message output). At this time, the information processing unit 140 determines a third reference distance greater than the second reference distance based on the difference between the first level and the current goodness level of the power storage unit 160 instead of disabling the following movement operation, thereby obtaining the second reference distance. An execution level of the following movement operation may be lowered to follow the user to a nearby location within the third reference distance instead of the distance. Also, when the execution level of the operation of outputting a dialog message is lowered, the display 122 is deactivated, and the dialog message may be output only through the speaker 121 . Subsequently, when the quality level of the power storage unit 160 is the third level, the information processing unit 140 may execute only one interaction operation, that is, outputting a conversation message through the speaker 121 . At this time, the volume of the dialog message output through the speaker 121 is equal to the difference between the current quality level of the power storage unit 160 and the second level compared to the case where the quality level of the power storage unit 160 is the third level. can be small

As described above, by executing the adaptive power saving operation according to the quality of the power storage unit 160, the power consumption of the power storage unit 160 is reduced to prevent the interaction robot 100 from being suddenly shut down, and the user Interaction service can be continued. Rule data defining the correlation between (i) situation cases, (ii) interaction actions selected for each situation case, and (iii) priority (and/or execution level) of interaction actions are stored in advance in the information storage unit 141 may have been

9 is a flowchart exemplarily showing a method of controlling an interaction robot according to another embodiment of the present invention. In describing the method of FIG. 9 , it is noted in advance that repeated descriptions of at least some of the contents common to the method described above with reference to FIG. 4 may be omitted.

1 to 3 and 9 , in step S910, the information processing unit 140 collects sensing data from at least one sensing unit among the plurality of sensing units 111 to 116.

In step S920, the information processing unit 140 generates first context-aware information representing a recognized situation with respect to at least one of the user and the interaction robot 100, based on the sensing data collected from at least one sensing unit. do.

In step S925, the information processing unit 140 collects second context-aware information representing an external situation through a wireless communication network.

In step S930, the information processing unit 140 selects at least one interaction action based on the first contextual information and the second contextually aware information. That is, the method of FIG. 9 is different from the method of FIG. 4 in that the second contextual information is additionally utilized in selecting an interaction operation. For example, in the situation illustrated in FIG. 6 , when external temperature is collected as the second contextual information, part of the content of the conversation message 610 based on the first contextual information is modified based on the second contextual information. Or new conversational messages that have been augmented (e.g. "Looks like you're eating. There seems to be a lot of smoke while cooking. The temperature in the kitchen is around 12 degrees, but it's -5 degrees outside. Turn on the air purifier instead of ventilating!" etc.) It is possible to control the information output unit 120 to output.

In step S940, the information processing unit 140 controls at least one of the information output unit 120 and the driving unit 130 and at least one of the wireless communication unit 150 to execute each interaction operation selected in step S930. That is, the method of FIG. 9 is different from the method of FIG. 4 in that the wireless communication unit 150 is additionally utilized in executing the interaction operation.

10 and 11 are diagrams referenced to describe various interaction operations executable through the control method according to another embodiment of FIG. 9 .

10 is a reference diagram for explaining an emotion support operation, which is one of various interaction operations. The user's emotional state changes from time to time. The information processing unit 140 executes an interaction operation, such as adaptively changing the content of a conversation message according to a change in the user's emotion, while new information collected through the wireless communication unit 150 and/or the information storage unit 141 ), an emotion supporting operation as an additional interaction operation for reinforcing a user's positive emotion or mitigating negative emotion may be executed using past information recorded in ).

Referring to FIG. 10 , the information processing unit 140, when the first contextual information indicates a user's positive emotional state, transmits contextual data related to the positive emotion (eg, an image including a user's face with a smiling expression) to a wireless communication network. It can be transmitted to the user's mobile terminal through , and this operation may also correspond to an interaction operation. The mobile terminal may time-synchronize and record the corresponding situation data wirelessly transmitted from the interaction robot 100, and the user may check the situation data representing his/her positive emotion history (timeline) through the mobile terminal.

Meanwhile, when the first situational awareness information indicates a user's negative emotional state, the information processing unit 140 transmits content data (eg, sayings, religious doctrines, meditation music, daily phrases, etc.) related to negative emotions to at least one external device. It may be collected from user history information pre-stored in the information medium or the information storage unit 141 . For example, the information processing unit 140 transmits emotion description data (representative tag) expressing the user's current negative emotion and situation description data expressing the surrounding situation to external information media or content recorded in the information storage unit 141. It is possible to compare each data with emotion description data and situation description data, and obtain content data having the highest similarity. The information processing unit 140 may generate a conversation message for alleviating the user's negative emotion through communication with the user from content data related to the negative emotion.

11 is a reference diagram for explaining a peripheral device control operation, which is one of various interaction operations. Referring to FIG. 11 , the information processing unit 140 detects at least one noise while detecting the dialog voice 1110 being uttered by the user, the sound source generating each noise based on the sound characteristics of each noise. type can be identified.

The information processing unit 140 accesses each of at least one electronic device (eg, TV, wireless robot vacuum cleaner) pre-registered as being controllable over the Internet of Things (IoT) through a wireless communication network, and determines the operating state of each electronic device. The indicated information may be collected as second situational awareness information. The information processing unit 140 may determine whether each electronic device is currently operating based on the operating state information of at least one electronic device included in the second situational awareness information, and additionally, each electronic device currently operating. It is possible to estimate the sound pressure level of the noise generated in and detected by the microphone 221 . In addition, the information processing unit 140 may set the electronic device as a noise suppression target when the electronic device determined to be currently operating matches the previously identified type of sound source.

Meanwhile, the information processing unit 140 transmits an audio signal having a predetermined sound pattern to each electronic device in order to accurately determine whether the sound source of a specific noise detected by the microphone 221 matches each electronic device currently operating. You can order playback. Next, the information processing unit 140 determines that the source of the specific noise is the corresponding electronic device when the sound pattern detected by the microphone 221 matches the sound pattern of the audio signal that has been commanded to be reproduced by the electronic device. can

The information processing unit 140 may transmit a noise reduction request to each electronic device set as a noise suppression target through a wireless communication network. In this case, the information processing unit 140 may set a sound pressure level for each electronic device to be noise suppressed based on the sound pressure level of the conversational voice 1110 . For example, as the sound pressure level of the dialog voice 1110 decreases, the sound pressure level for each electronic device is reduced, thereby ensuring accuracy of sensing the user's voice through the microphone 221 . Each electronic device receiving the noise reduction request may lower the operation level or stop the operation. For example, a wireless robot cleaner, which is a mobile electronic device, in response to a request for noise reduction, lowers the level of dust suction power from the current value, moves to an area farthest from the user among the remaining areas to be cleaned, and cleans the area. , the cleaning operation can be temporarily stopped. As another example, a TV, which is a non-portable electronic device, may turn off power, mute the volume of a current channel, or switch to another channel with a low noise level in response to a request for noise reduction.

Meanwhile, the information processor 140 may adjust the volume of the conversation message 1120 for the user output through the speaker 121 based on the sound pressure level for each noise detected together with the conversation voice 1110 . For example, as the sound pressure level of noise generated by a specific electronic device increases, the volume of the chat message 1120 fed back to the user may be increased so that the user can hear the chat message 1120 with a sufficiently high sensitivity. .

In addition to this, the information processing unit 140 is configured for a mobile electronic device (e.g., a wireless mobile vacuum cleaner) so that a path from the interaction robot 100 to the user can be sufficiently secured during the following movement operation (see FIG. 5). You can also set a movement restriction area. For example, the information processing unit 140 is a mobile electronic device within a predetermined safe distance from the user or the interaction robot 100 based on the visual-based context data, the auditory-based context data, and/or the second context-aware information. It can be determined whether or not is located. Next, the information processing unit 140 generates a virtual path from the interaction robot 100 to the user based on the positions of the interaction robot 100 and each user, and then causes the interaction robot 100 to follow the virtual path. While controlling the sixth and seventh driving units 136 and 137, an area within a safe distance from the virtual path is set as a movement restriction area, and a movement limiting command including data representing the movement restriction area is transmitted to the mobile electronic device. can transmit The mobile electronic device may operate to avoid the movement restriction area in response to the movement restriction command. Accordingly, while the interaction robot 100 moves close to the user, it is possible to prevent path obstruction by the mobile electronic device.

Of course, the interaction operations executable by the interaction robot 100 are not limited to the examples listed above. In addition, when the interaction robot 100 actively plays music or according to a user's request, the music files stored in the information storage unit 141 and the information detected by the information collection unit 110 are used as an aid to music playback. can For example, the information processing unit 140 is configured to recognize the peripheral space based on the profile of each of at least one music file (eg, singer, composer, lyricist, genre (rock, classical, etc.), BPM) and vision-based sensing data. Using information related to (e.g., indoor/outdoor status, room, living room, kitchen, bathroom, etc. if indoors), the sound effect profile and spatial profile to be applied to the music to be played are automatically determined and mapped to the situation perceived in the current space. The speaker 121 may be controlled to play music with the set value.

The present invention is not limited to the specific embodiments and application examples described above, and various modifications can be made by those skilled in the art without departing from the gist of the present invention claimed in the claims. Of course, these modified implementations should not be understood separately from the technical spirit or perspective of the present invention.

In particular, the components implementing the technical features of the present invention included in the block diagrams and flowcharts shown in the accompanying drawings represent the logical boundaries between the components. However, according to an embodiment of software or hardware, the illustrated components and their functions are executed in the form of independent software modules, monolithic software structures, codes, services, and combinations thereof, and may execute stored program codes, instructions, and the like. Since functions may be implemented by being stored in a medium executable on a computer having a processor, all of these embodiments should also be regarded as falling within the scope of the present invention.

Accordingly, although the accompanying drawings and description thereof describe technical features of the present invention, they should not be simply inferred unless a specific arrangement of software for implementing these technical features is explicitly mentioned. That is, various embodiments described above may exist, and since such embodiments may be partially modified while retaining the same technical characteristics as the present invention, this should also be regarded as falling within the scope of the present invention.

In addition, in the case of a flow chart, although operations are depicted in a specific order, this is shown in order to obtain the most desirable results, and such operations must be executed in a specific sequence or sequential order shown or all illustrated operations must be executed. It should not be understood as what it should be. In certain cases, multitasking and parallel processing can be advantageous. In addition, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems are generally integrated together in a single software product or in multiple software products. It should be understood that it can be packaged.

100: interaction robot
110: information collection unit 111 to 116: first to sixth sensing unit
211: wide-angle camera 212: telephoto camera
213: infrared sensor 221: microphone
231: touch sensor 232: strain gauge
241: air quality sensor 242: temperature sensor
243 Humidity sensor
120: information output unit 121: speaker
122: display 123: optical output
130: driving unit 131 to 137: first to seventh power joints
140: information processing unit 150: wireless communication unit
160: power storage unit 170: function port
180: heat dissipation member

Claims (12)

In the multimodal-based interaction robot,
an information collection unit including a plurality of sensing units associated with a plurality of different modalities on a one-to-one basis;
an information output unit outputting interaction feedback related to the motion executed by the interaction robot;
a driving unit controlling movement of the interaction robot;
A wireless communication unit for accessing a mobile electronic device registered as being controllable by IoT through a wireless communication network; and
An information processing unit configured to generate first context-aware information representing a recognized situation with respect to at least one of a user and the interaction robot based on sensing data from at least one of the plurality of sensing units,
The information processing unit,
Selecting at least one interaction action based on the first contextual information;
A first interaction action related to a local situation based on an object feature of a local object within a first reference distance from the interaction robot and a second interaction related to a far situation based on an object feature of a far object outside the first reference distance from the interaction robot When an action is selected, giving the first interaction action a higher priority than the second interaction action;
Controlling at least one of the information output unit and the driving unit to sequentially execute the first interaction operation and the second interaction operation according to the priority of each interaction operation,
When the short-distance object is the user, the first interaction operation is a following movement operation moving within a second reference distance from the user;
When the information processing unit executes the following movement operation,
creating a virtual path from the interaction robot to the user;
An area within a predetermined safety distance from the virtual path is set as a movement restriction area,
and controlling the wireless communication unit to transmit a movement limiting command for the mobile electronic device to avoid the movement restriction area to the mobile electronic device through the wireless communication network.
According to claim 1,
The plurality of sensing units,
A first sensing unit configured to generate first sensing data including at least one of a 2D image, a 3D image, and a brightness value around the interaction robot;
The information processing unit,
Based on the first sensing data, visual-based-situation data representing object characteristics for each object located around the interaction robot is generated;
An interaction robot that generates the first context-aware information based on the time-based context data.
According to claim 2,
The information processing unit,
Comparing images from two cameras of the first sensing unit to generate position data of at least one object around the interaction robot;
The time-based-situation data includes location data of the at least one object, the interaction robot.
According to claim 1,
The plurality of sensing units,
A second sensing unit configured to detect sounds around the interaction robot and generate second sensing data;
The information processing unit,
Based on the second sensing data, hearing-based context data around the interaction robot is generated;
Based on the auditory-based contextual data, the interaction robot for generating the first contextual information.
According to claim 4,
The information processing unit,
Comparing audio signals from at least two of the plurality of microphones included in the second sensing unit to generate position data of at least one sound source around the interaction robot,
The auditory-based context data includes location data of the at least one sound source.
According to claim 1,
The plurality of sensing units,
A third sensing unit generating third sensing data by detecting a contact event with respect to a predetermined area of the body of the interaction robot;
The information processing unit,
Based on the third sensing data, generating tactile based-situation data of the interaction robot;
Based on the tactile sense-based contextual data, the interaction robot for generating the first contextual information.
According to claim 1,
The plurality of sensing units,
A fourth sensing unit configured to sense a standby state around the interaction robot and generate fourth sensing data;
The information processing unit,
Based on the fourth sensing data, generating olfactory based-situation data of the interaction robot;
An interaction robot that generates the first context-aware information based on the olfactory-based context data.
According to claim 1,
a power storage unit supplying power to the information collection unit, the information output unit, the driving unit, and the information processing unit;
Including more,
The plurality of sensing units,
A sixth sensing unit configured to sense a state of the power storage unit and generate sixth sensing data;
The information processing unit,
An interaction robot generating the first contextual information based on the sixth sensing data.
According to claim 1,
The wireless communication unit,
Accessing at least one external information medium through the wireless communication network to collect second context-aware information representing an undetectable external situation through the information collection unit;
The information processing unit,
An interaction robot that selects the at least one interaction action further based on the second context awareness information.
According to claim 1,
a heat dissipation member mechanically coupled between an outer surface of the interaction robot and the information processing unit so that thermal energy generated by the information processing unit is transmitted to the body of the interaction robot;
Further comprising, interaction robot.
In the control method of the interaction robot according to any one of claims 1 to 10,
collecting sensing data from at least one sensing unit among the plurality of sensing units;
generating first context-aware information representing a recognized situation with respect to at least one of the user and the interaction robot, based on sensing data from the at least one sensing unit;
selecting at least one interaction action based on the first contextual information;
A first interaction action related to a local situation based on an object feature of a local object within a first reference distance from the interaction robot and a second interaction related to a far situation based on an object feature of a far object outside the first reference distance from the interaction robot if an action is selected, giving the first interaction action a higher priority than the second interaction action; and
Controlling at least one of the information output unit and the driving unit to sequentially execute the first interaction operation and the second interaction operation according to the priority of each interaction operation,
When the short-distance object is the user, the first interaction operation is a following movement operation moving within a second reference distance from the user;
In the step of executing the follow-up movement operation,
creating a virtual path from the interaction robot to the user;
setting an area within a predetermined safe distance from the virtual path as a movement restriction area; and
Controlling the wireless communication unit to transmit a movement limiting command for the mobile electronic device registered as IoT controllable to avoid the movement restriction area to the mobile electronic device through the wireless communication network. control method.
A computer program stored in a computer-readable storage medium to execute the control method of the interaction robot according to claim 11.

Images