KR101343860B1 - Robot avatar system using hybrid interface and command server, learning server, and sensory server therefor - Google Patents

Abstract

Provided is a robot avatar system equipped with a hybrid interface including a brain computer interface (BCI), a head-mounted display (HMD) or smart glasses, a motion sensor, a position sensor, a microphone, a speaker, a tactile sensor, and a haptic actuator. The robot avatar system of the present invention comprises: an interface including the BCI, the HMD or the smart glasses, the motion sensor, the position sensor, the microphone, the speaker, the tactile sensor, and the haptic actuator; a command server which generates a command to control the behavior of the robot based on BCI information and motion information received therein from the BCI and the motion sensor or the position senor and outputs the voice information of a person received from the microphone through the robot; a BCI-motion learning server which constructs a BCI-motion mapping database and a label determination model used in the command server based on the BCI information and the motion information received therein from the BCI and the motion sensor or the position sensor; and a sensory server which processes audio, video, and tactile information, which are obtained through the robot, and outputs the processed information through the speaker, the HMD or the smart glasses, and the haptic actuator. [Reference numerals] (110) BCI device;(120) HMD/Smart glasses;(130) Vision sensor;(140,620) Microphone;(160) Position sensor;(200) Command server;(400) BCI-motion learning server;(500) Sensory server;(610) Camera;(640) Speaker;(650) Tactile sensor;(AA) Motion command, voice;(BB) BCI data, vision/position data, voice;(CC) Humanoid;(DD) Person;(EE) Raw sensory data;(FF) Processed sensory data

Description

Robot avatar system using hybrid interface and command server, learning server, and sensory server therefor}

The present invention relates to a robot avatar system and a robot avatar system, and more particularly to a robot avatar system using a hybrid interface.

The concept of the avatar is recognized by the public through movies and the like, but the method for realizing the concept is not sufficiently studied.

Avatars depict themselves as computer users, two-dimensional models in computer games, two-dimensional icons (pictures) in Internet forums and other communities, and string structures in early systems such as mud games. In other words, it is an object that gives the user its own appearance (Source: Internet Wikipedia).

As a related art in this regard, there is a Patent Publication No. 10-2009-0113084, Patent Publication No. 10-2011-0047847.

In Patent Publication No. 10-2009-0113084, 3D image information including 2D image information and distance information of a user is recognized, and a first reference point and a second reference point are identified based on the 3D image information. A remote control device that calculates a change in the angle of the joint based on three-dimensional coordinates of the first reference point and the second reference point, and transmits a joint control signal through a wired / wireless network, and the joint received from the remote control device; Disclosed is a motion control system of a robot including a robot which grasps joint control data from a control signal and changes the angle of the joint to have a motion corresponding to the movement of the user.

In Korean Patent Laid-Open No. 10-2011-0047847, the robot part performing the main motion highly related to the motion commanded to the humanoid robot is controlled to move along the optimized motion trajectory generated by the motion optimization operation considering the robot dynamics. In addition, by controlling the robot part to perform the remaining motion that is not related to the commanded motion to move along a predetermined motion trajectory to correspond to the commanded motion, the humanoid robot can be simplified while optimizing the whole body motion optimization of the humanoid robot. Disclosed are a humanoid robot capable of performing an operation as closely as possible to an actual human action and a control method thereof.

The present invention has been made in the technical background as described above, by adjusting the humanoid robot, the humanoid can imitate human behavior and speech, and can transmit the visual, auditory, tactile information accepted by the humanoid to the human The task is to provide a robot avatar system.

Another object of the present invention is to provide a robot avatar system using a personalized hybrid interface to imitate human behavior and speech in a flexible and accurate manner.

In order to solve the above problems, in the present invention, a Brain Computer Interface (BCI), Head-Mounted Display (HMD) or Smart Glasses, Motion Sensor, Position Sensor, Microphone, A robot avatar system having a hybrid interface including a speaker, a tactile sensor, and a haptic actuator is provided.

Robot avatar system using a hybrid interface according to an aspect of the present invention, a brain computer interface (BCI), Head-Mounted Display (HMD) or smart glasses (Smart Glasses), motion sensors an interface including a motion sensor or a position sensor, a microphone, a speaker, a tactile sensor, and a haptic actuator; Generate a command for controlling the behavior of the robot based on the BCI and the BCI information and motion information received from the motion sensor or position sensor, and delivers the voice information of the person received from the microphone through the robot A command server; A BCI-motion learning server for constructing a BCI-motion mapping database and a label discrimination model used in the command server based on the BCI and BCI information and motion information received from the motion sensor or position sensor; And a perception server that processes audio, video, and tactile information obtained through the robot to be output through the speaker, the HMD or the smart glasses, and the haptic actuator, respectively.

The microphone and the speaker may be integrally formed with the HMD or the smart glasses, or the BCI may be integrally formed with the HMD or the smart glasses.

According to another aspect of the present invention, there is provided a command server for generating a command for controlling a robot in a robot avatar system using a hybrid interface, which includes a vision data preprocessor for receiving and preprocessing motion information obtained from a vision sensor; A position data preprocessor configured to receive and preprocess the motion information acquired from the position sensor; A real time motion fine control unit which receives the vision data and the position data from the vision sensor and the position sensor and extracts accurate motion data in real time by resolving and adjusting the inconsistency between the vision data and the position data; A BCI data preprocessor for preprocessing BCI information received from a Brain Computer Interface (BCI); A real-time BCI label discrimination unit for determining a label indicating what kind of movement the user currently thinks by analyzing the preprocessed BCI data in real time using a discrimination model stored in a database; A motion data extractor for extracting motion data from a BCI-motion mapping database based on the determined label; A hybrid motion generator configured to determine a motion intended by the user based on the first motion data finely adjusted by the real-time motion fine controller and the second motion data extracted by the motion data extractor; A person-robot motion mapping unit for mapping the movement of the user to the movement of the robot based on the motion determined by the hybrid motion generating unit and the information of the robot; And a motion command transmitter for transmitting a motion command.

The command server may include a voice processing unit configured to receive voice information by human speech or speech and process the voice information in a form that the robot can output; And a voice transmitter for transmitting the voice information processed by the voice processor to the robot.

When only the first motion data or the second motion data is input, the hybrid motion generation unit uses only the first motion data or the second motion data input, and the first motion data and the second motion. When all of the data is input, when the first motion data and the second motion data coincide, the first motion data and the second motion data are combined to be used as a more precise motion, and the first motion data and If the second motion data does not match, both motions are used when two motions designated by the first motion data and the second motion data can be simultaneously used, and the first motion data and the second motion are used. Two movements if the two movements specified by the movement data cannot be performed at the same time The predetermined movement of the predetermined high priority may be used to determine the movement intended by the user.

Preferably, the command server further includes a user recognition unit for recognizing the user so as to use a different BCI-motion mapping database for each user.

According to another aspect of the present invention, a brain-computer interface (BCI) -motion mapping database and a label discrimination model in a command server generating a motion command for controlling a robot in a robot avatar system using a hybrid interface Provided is a learning server for building a vision data pre-processing unit for receiving and pre-processing motion information obtained from the vision sensor; A position data preprocessor configured to receive and preprocess the motion information acquired from the position sensor; A motion fine control unit which receives the vision data and the position data from the vision sensor and the position sensor and extracts accurate motion data by resolving and adjusting the inconsistency between the vision data and the position data; A BCI data preprocessor for preprocessing the BCI data received from the BCI; A motion data modeling unit for converting the exact motion extracted by the motion fine control unit into a form suitable for database storage; A BCI data modeling unit for converting the BCI data into a form suitable for database storage; A BCI-motion mapping registration unit which stores the motion data and the BCI data in a BCI-motion mapping database; And a BCI-motion discrimination model generation unit for generating a model for determining a label indicating which BCI pattern corresponds to which idea using the BCI-motion mapping database.

The learning server may further include a user recognizing unit which checks the identification information of the user and transmits the identification information of the user who is currently learning to the BCI-motion mapping database so that the label can be individually determined for each user.

According to another aspect of the present invention, there is provided a perception server for transmitting information obtained through a robot to a person in a robot avatar system using a hybrid interface, which receives and processes video data from a camera included in the robot A video data processor; An audio data processor configured to receive and process audio data from a microphone included in the robot; And a tactile data processor configured to process the tactile information received from the tactile sensor included in the robot into a form for transmitting the tactile information to a person through a haptic actuator.

The perception server further includes a database unit for storing Augmented Reality (AR) information, wherein the video data processor and the audio data processor are configured to store the video data and the augmented reality information using the augmented reality information stored in the database unit. Audio data can be processed.

The tactile data processor may process the tactile information so that the tactile information does not exceed a predetermined maximum value, or may be processed to a strength adjusted by a user in a desired manner.

According to the robot avatar system using the hybrid interface of the present invention, it is possible not only to convey the senses that the robot sees, hears, and feels to a human being, but also to convey almost all the actions that a human can make through a hybrid interface to a humanoid at a remote location. have. As a result, the robot avatar concept can be realized for a wide range of applications, including medical, rehabilitation, special tasks, games, and military operations.

1 is a view showing the overall configuration of a robot avatar system using a hybrid interface according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of a command server used in a robot avatar system using a hybrid interface according to an exemplary embodiment of the present invention.
3 is a flowchart illustrating a process of generating a motion command by combining motion data and BCI data in a command server used in a robot avatar system using a hybrid interface according to an embodiment of the present invention.
4 is a block diagram illustrating a configuration of a BCI-motion learning server used in a robot avatar system using a hybrid interface according to an embodiment of the present invention.
5 is a block diagram illustrating a configuration of a sensory server used in a robot avatar system using a hybrid interface according to an exemplary embodiment of the present invention.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. It is to be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. It is noted that the terms "comprises" and / or "comprising" used in the specification are intended to be inclusive in a manner similar to the components, steps, operations, and / Or additions.

Hereinafter, a robot avatar system using a hybrid interface according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

1 is a view showing the overall configuration of a robot avatar system using a hybrid interface according to an embodiment of the present invention.

As shown in FIG. 1, a robot avatar system using a hybrid interface according to an exemplary embodiment of the present invention intends to control a humanoid 60, which is a robot in the form of a human, and has a human 10 and a humanoid 60. The human interface (60) shows the behavior of the person (10) obtained through the hybrid interface (110, 120, 130, 140, 150, 160) and the hybrid interface (110, 120, 130, 140, 150, 160). By reconfiguring the information perceived through the learning server 400, the humanoid 60 to generate the BCI-motion mapping data to be used by the command server 200 and the command server 200 to configure and deliver the command to control the It includes the perception server 500 to deliver to (10).

The hybrid interface includes a brain computer interface (BCI) 110, a head-mounted display (HMD) or smart glasses 120, a vision sensor 130. , A microphone 140, a haptic actuator 150, a position sensor 160, and a speaker (not shown). Two or more of the vision sensor 130, the microphone 140, the haptic actuator 150, and the position sensor 160 may be used.

The brain-computer interface 110 analyzes the activity of the brain and predicts the intention of the person 10. The brain-computer interface 110 receives brain waves through a device that recognizes brain wave stimuli, encodes them, and transfers them to the command server 200. do. The brain-computer interface 110 may be used as a separate device or may be integrated with the head mounted display 120. When the brain-computer interface 110 is used, the humanoid 60 may be adjusted according to the intention of the person 10 even when the sensory or physical behavior such as vision is incomplete.

The head mounted display 120 is a display device in which the person 10 can be mounted on the head or in the form of a helmet, and displays image data perceived by the humanoid 60 as the person 10 sees.

In addition, although not shown in the drawings, a speaker for outputting the voice data perceived from the humanoid 60 as a person 10 is included, the speaker may be implemented to be included in the head mounted display 120.

The vision sensor 130 and the location sensor 160 transmit vision information and location information for recognizing the movement of the person 10 to the command server 200. Since the humanoid 60 is designed to mimic the movement of the person 10, various sensors may be used to accurately recognize the movement of the person 10. The position sensor 160 refers to a sensor that measures the position of each part of the body by wearing or attaching to the body, such as a data globe, and using a motion sensor instead of the position sensor 160 or using both a motion sensor and a position sensor. Can also be used.

The microphone 140 receives a human voice and transmits the voice data to the command server 200. In the robot avatar system using the hybrid interface according to the embodiment of the present invention, the humanoid 60 mimics not only the action of the person 10 but also speech or utterance.

On the other hand, the microphone 140 may be implemented integrally with the head mounted display 120.

The haptic actuator 150 is used by the person 10 to recognize the tactile information perceived by the humanoid 60, and may be used more than two, and may be mounted on a required body part of the person 10.

In the robot avatar system using the hybrid interface according to the exemplary embodiment of the present invention, the behavior related information of the person 10 recognized through the hybrid interface is transmitted to the command server 200. More specifically, brain wave information recognized through the brain-computer interface 110, vision information obtained through the vision sensor 130, position information acquired by the position sensor 160, and input through the microphone 140. Voice information is transmitted to the command server 200.

In addition, the BCI-motion learning server 400 is a BCI-motion mapping database used in the command server 200 to generate a desired motion command based on the brain wave information recognized through the brain-computer interface 110. And a label discrimination model.

The command server 200 generates information for controlling the humanoid 60 using information transmitted through the hybrid interface and information constructed by the BCI-motion learning server 400, that is, motion command and voice, and generates the humanoid. Forward to 60.

2 is a block diagram illustrating a configuration of a command server used in a robot avatar system using a hybrid interface according to an exemplary embodiment of the present invention.

As shown in FIG. 2, the command server 200 determines a vision data preprocessor 210, a position data preprocessor 212, a real time motion fine control unit 214, a BCI data preprocessor 230, and a real time BCI label determination. Unit 232, user recognition unit 236, BCI-motion mapping database 238, motion data extraction unit 234, hybrid motion generation unit 216, human-robot motion mapping unit 218, motion transmitter 220, a voice data processor 240, and a voice transmitter 242.

The vision data preprocessor 210 and the position data preprocessor 212 process raw data received from the vision sensor 130 and the position sensor 160, respectively, and deliver the raw data to the real-time motion fine control unit 214. do. The real-time motion fine control unit 214 extracts accurate motion data in real time by receiving vision data and position data from various sensors to solve and adjust inconsistencies among the sensor data.

The BCI data preprocessor 230 processes the raw BCI data received from the BCI device 110 and transmits the raw BCI data to the real-time BCI label determination unit 232. The real-time BCI label determination unit 232 analyzes the pre-processed BCI data in real time using a database stored discrimination model to determine the label of what kind of movement a person currently thinks. With the label determined as such, the motion data extractor 234 extracts the corresponding motion data from the BCI-motion mapping database 238.

Now, by receiving the motion data of the person finely adjusted by the real-time motion fine control unit 214 and the motion data that is reflected in the head, the hybrid motion generation unit 216 finally determines the motion intended by the person.

The human-robot motion mapping unit 218 maps the motion of the person to the motion of the robot by receiving the determined motion and the information of the robot, and transmits the mapped motion to the humanoid 60 by the motion transmitter 220. do.

The command server 200 also includes a user recognizer 236 that recognizes the user so that each person can use a different BCI-motion mapping database 238.

On the other hand, in the robot avatar system using the hybrid interface according to an embodiment of the present invention, the human voice is not used to generate a command for controlling the behavior of the humanoid 60, the humanoid 60 is spoken by the human Or mimics speech. Accordingly, the voice information processor 240 receives voice information of the person 10 input through the microphone 140 and processes the humanoid 60 to output the voice information. The voice transmitter 260 processes the processed voice information. Transmit to humanoid (60).

In the robot avatar system using the hybrid interface according to the embodiment of the present invention, since the humanoid 60 mimics the movement of the person 10 as it is, the vision sensor 130 basically receives a command related to the movement of the humanoid 60. And based on the motion information acquired through the position sensor 160, the BCI information can be used as information for reinforcing it. However, if the sensory or physical behavior is not intact due to a disorder of the body, the information by the motion sensor 130 may not sufficiently reflect the intention of the person 10, and in this case, BCI information may be mainly used. .

That is, the hybrid motion generator 216 combines the information derived by the real-time motion fine control unit 214 and the motion data extractor 234 according to a preset priority, or by combining the information in various other ways. Can be used.

For example, the hybrid motion generation unit 216 (1) uses only motion data when there is no motion, and (2) uses only motion data from the database when thinking without motion, and (3) When moving simultaneously with thoughts (3-1) If the two match, combine the two data for more precise movement, or (3-2) If the two do not match, move the data Can work by combining At this time, no matter what thought it is, it must be recognized as a valid thought before it is registered in the database through the learning server, otherwise it is recognized as not thinking. In the example of (3-2), if the user thinks of jumping in the head while shaking the arm, the robot avatar is transmitted by shaking the arm. In addition, in the case of (3-2), if two kinds of movements cannot be performed at the same time, only one kind of movement is used according to the priority designated by the user. For example, if you are thinking about lowering your arm while raising your arm, only one of them will be used. However, this is unlikely to occur very often because it is difficult or rare to actually do the opposite of what humans think.

3 is a flowchart illustrating a process of generating a motion command by combining motion data and BCI data in a command server used in a robot avatar system using a hybrid interface according to an embodiment of the present invention.

As shown in FIG. 3, when only motion data is input first (YES in S310), the motion data is output as it is (S320), and conversely, when only BCI data is input (YES in S330), the corresponding BCI data is used. The motion data extracted from the database may be output as it is (S340).

If both data are input (NO in S330), it is determined whether the two data match within a range value (S350). When the two data coincide within the range value (YES in S350), the two pieces of data may be used together to generate more accurate motion data (S360).

 If the two data do not match within the range value (NO in S350), it is determined whether two motions can be performed simultaneously (S370).

If two motions can be performed simultaneously (Yes in S370), motion data combining the two motions is generated (S380). If two motions cannot be performed simultaneously (No in S370), Only motion data of one of the two motions is output according to the priority (S390).

Meanwhile, the BCI-motion learning server 400 is a BCI-motion mapping database used by the command server 200 to generate a desired motion command based on the brain wave information recognized through the brain-computer interface 110. And a label discrimination model.

4 is a block diagram illustrating a configuration of a BCI-motion learning server used in a robot avatar system using a hybrid interface according to an embodiment of the present invention.

As shown in FIG. 4, the BCI-motion learning server 400 includes a vision data preprocessor 410, a position data preprocessor 412, a motion fine controller 414, a motion data modeling unit 420, and BCI data. The preprocessor 430, the BCI data modeling unit 432, the BCI-motion mapping registration unit 434, the BCI-motion mapping database 438, the BCI-motion determination model generation unit 440, and the user recognition unit 436. Include.

The vision data preprocessor 410 and the position data preprocessor 412 process the raw data received from the vision sensor 130 and the position sensor 160, respectively, and deliver the raw data to the motion fine control unit 414. . The motion fine control unit 414 receives vision data and position data from various sensors, resolves and adjusts inconsistencies among the sensor data, and extracts and transmits the correct motion data to the motion data modeling unit 420, and the motion The data modeling unit 420 converts the fine and accurate movement into a form suitable for database storage and transmits the same to the BCI-motion mapping registration unit 434.

The BCI data preprocessor 430 processes the raw BCI data received from the BCI device 110 and delivers the processed BCI data to the BCI data modeling unit 432, and the BCI data modeling unit 432 is configured to transfer the BCI data into a database suitable for storage. In addition, the BCI-motion mapping registration unit 434 stores the motion data received from the motion data modeling unit 420 and the BCI data in the BCI-motion mapping database 438. The BCI-motion discrimination model generation unit 440 generates a discrimination model (classifier) so that the BCI-motion mapping database 438 may be used to determine which BCI pattern is a label.

In addition, the user recognition unit 436 checks the user ID and transmits the ID of the person currently learning to the database so that such labeling can be customized for each person.

A unique feature of the BCI-motion learning server used in a robot avatar system using a hybrid interface according to an embodiment of the present invention is to theoretically provide a function for discriminating BCI data having an infinite number of branches. This feature is very useful and important when implementing the robot avatar platform because the number of motions that a person can take is infinite. When using the existing supervised learning or classification methods, a person specifies a class label (eg, a walking motion) for each BCI data to be learned, and builds a classification model based on the set of <data, label> pairs thus prepared. do. Then, when new BCI data is input during actual operation, it is determined which class label belongs to, and a movement belonging to the corresponding label is transmitted as a command.

However, these existing methods have a problem in that an individual cannot determine how to take a walking motion specifically for a specific class label (eg, a walking motion). However, in the case of a learning server according to an embodiment of the present invention, when a person takes an action while thinking about an action, the action data itself is regarded as a class label and stored in a database along with the BCI data corresponding to the idea. Assume that Therefore, there is no limit on the number of kinds of class labels, and different operations may be stored as class labels for each individual. If you build a classification model in the BCI-Motion Discrimination Model Generator for a set of <data, label> pairs stored in the database, each time a new BCI data is entered when operating the command server, It discriminates and transmits the motion registered by each person by a command.

The robot used in the robot avatar system using the hybrid interface according to the embodiment of the present invention is preferably a humanoid that is a robot that implements a recognition function and a movement function such as a human.

The humanoid 60 mimics the behavior of the person 10 based on the information transmitted from the command server 200, and the information perceived through the humanoid 60 is transmitted to the person 10 again. To this end, the humanoid 60 includes a camera 610 for recognizing an image, a microphone 620 for recognizing audio information, a speaker 640 for speech and speech, and a tactile sensor for recognizing tactile information. 650 and the like, it is desirable to be able to mimic the movement of specific joints of the person 10 including bipedal walking.

However, the robot used in the robot avatar system using the hybrid interface according to the embodiment of the present invention does not necessarily have to be a humanoid, and the function and configuration of the robot avatar system using the hybrid interface according to the embodiment of the present invention may be properly adjusted. It can also be used as a system for adjusting a robot of a type, such as a wheel moving robot.

Perception information (raw sensory data) recognized by the humanoid 60 is transmitted to the perception server 500.

5 is a block diagram illustrating a configuration of a sensory server used in a robot avatar system using a hybrid interface according to an exemplary embodiment of the present invention.

As shown in FIG. 5, the perceptual server 500 includes a video data processor 510, an audio data processor 520, and a tactile data processor 550. If necessary, additional information may be added to the acquired information. It may further include an AR database 530 that stores Augmented Reality (AR) information for providing.

The video data processor 510 receives raw video data from the camera 610 included in the humanoid 60, and the audio data processor 520 from the microphone 620 included in the humanoid 60. Receive raw audio data, perform data processing, and transmit it to the head mounted display 120 or smart glasses of the person 10, respectively. In this case, AV data may be processed using the augmented reality information stored in the AR database 530.

The tactile data processor 550 processes the tactile information received from the tactile sensor 650 included in the humanoid into a form for delivering the tactile information to the person 10 through the haptic actuator 150. In addition to signal processing, in addition to signal processing, the tactile data processor 550 may process tactile data such as limiting the maximum range of the impact or adjusting the intensity to a calibrated intensity in a manner desired by the user in consideration of different characteristics of the humanoid and the human. It may be.

In the robot avatar system using the hybrid interface according to the embodiment of the present invention, the person 10 may control the humanoid 60 at a remote location.

To this end, the connection between the person 10, the command server 200, the learning server 400, the humanoid 60, and the perception server 500 may use a wireless communication method, and there is no particular limitation on the communication method. Of course, it is also possible to implement a specific connection via wired communication, if necessary.

While the invention has been described in terms of the preferred embodiments, it is to be understood that the invention is not limited to the disclosed embodiments, but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the invention. Accordingly, the appended claims are intended to embrace all such modifications and variations as fall within the true spirit of the invention.

Claims (12)

A robot avatar system using a hybrid interface,
Brain Computer Interface (BCI), Head-Mounted Display (HMD) or Smart Glasses, Motion Sensor or Position Sensor, Microphone, Speaker, Tactile Sensor An interface comprising a haptic actuator;
Generate a command for controlling the behavior of the robot based on the BCI and the BCI information and motion information received from the motion sensor or position sensor, and delivers the voice information of the person received from the microphone through the robot A command server;
A BCI-motion learning server for constructing a BCI-motion mapping database and a label discrimination model used in the command server based on the BCI and BCI information and motion information received from the motion sensor or position sensor; And
Perceptual server that processes the audio, video, and tactile information obtained through the robot to output through the speaker, HMD or smart glasses, haptic actuator, respectively
Robot avatar system using a hybrid interface comprising a. The method of claim 1,
The microphone and the speaker is a robot avatar system using a hybrid interface formed integrally with the HMD or the smart glasses. The method of claim 1,
The BCI robot avatar system using a hybrid interface formed integrally with the HMD or the smart glasses. A command server for generating a command for controlling a robot in a robot avatar system using a hybrid interface,
A vision data preprocessor configured to receive and preprocess the motion information acquired from the vision sensor;
A position data preprocessor configured to receive and preprocess the motion information acquired from the position sensor;
A real time motion fine control unit which receives the vision data and the position data from the vision sensor and the position sensor and extracts accurate motion data in real time by resolving and adjusting the inconsistency between the vision data and the position data;
A BCI data preprocessor for preprocessing BCI information received from a Brain Computer Interface (BCI);
A real-time BCI label discrimination unit for determining a label indicating what kind of movement the user currently thinks by analyzing the preprocessed BCI data in real time using a discrimination model stored in a database;
A motion data extractor for extracting motion data from a BCI-motion mapping database based on the determined label;
A hybrid motion generator configured to determine a motion intended by the user based on the first motion data finely adjusted by the real-time motion fine controller and the second motion data extracted by the motion data extractor;
A person-robot motion mapping unit for mapping the movement of the user to the movement of the robot based on the motion determined by the hybrid motion generating unit and the information of the robot; And
A motion command transmitter for transmitting a motion command
Command server comprising a. 5. The method of claim 4,
A voice processing unit which receives voice information by human speech or speech and processes the voice information into a form that the robot can output; And
And a voice transmitter for transmitting the voice information processed by the voice processor to the robot. The method of claim 4, wherein the hybrid motion generating unit,
When only the first motion data or the second motion data is input, only the input first motion data or the second motion data is used.
When both the first motion data and the second motion data are input,
When the first motion data and the second motion data coincide, the first motion data and the second motion data are combined and used as a more precise motion.
If the first motion data and the second motion data do not coincide, if both motions designated by the first motion data and the second motion data can be simultaneously performed, both motions are used, and the first motion data is used. If two motions specified by the motion data and the second motion data cannot be performed at the same time, a motion having a predetermined high priority among the two motions is used.
A command server for determining the movement intended by the user. 5. The method of claim 4,
And a user recognizer configured to recognize the user so as to use a different BCI-motion mapping database for each user. A learning server for constructing a brain computer interface (BCI) -motion mapping database and a label discrimination model in a command server generating a motion command for controlling a robot in a robot avatar system using a hybrid interface,
A vision data preprocessor configured to receive and preprocess the motion information acquired from the vision sensor;
A position data preprocessor configured to receive and preprocess the motion information acquired from the position sensor;
A motion fine control unit which receives the vision data and the position data from the vision sensor and the position sensor and extracts accurate motion data by resolving and adjusting the inconsistency between the vision data and the position data;
A BCI data preprocessor for preprocessing the BCI data received from the BCI;
A motion data modeling unit for converting the exact motion extracted by the motion fine control unit into a form suitable for database storage;
A BCI data modeling unit for converting the BCI data into a form suitable for database storage;
A BCI-motion mapping registration unit which stores the motion data and the BCI data in a BCI-motion mapping database;
And a BCI-motion discrimination model generation unit for generating a model for determining a label indicating which BCI pattern corresponds to what idea using the BCI-motion mapping database. 9. The method of claim 8,
And a user recognizing unit which checks the identification information of the user and transmits the identification information of the user currently being trained to the BCI-motion mapping database so that the label can be individually determined for each user. A perception server that delivers information acquired through a robot to a human in a robot avatar system using a hybrid interface,
A video data processor configured to receive and process video data from a camera included in the robot;
An audio data processor configured to receive and process audio data from a microphone included in the robot; And
And a tactile data processor configured to process the tactile information received from the tactile sensor included in the robot into a form for transmitting the tactile information to a person through a haptic actuator. The method of claim 10,
Further comprising a database unit for storing Augmented Reality (AR) information,
And the video data processing unit and the audio data processing unit process the video data and the audio data using augmented reality information stored in the database unit. The method of claim 10, wherein the tactile data processor,
Perceptual server is processed so that the tactile information does not exceed a predetermined maximum value, or processed to the intensity adjusted (calibrated) in a way desired by the user.

Images