KR20140139394A - Augmented reality implementation apparatus and method for interacting with robot for education - Google Patents

Abstract

An augmented reality implementing apparatus is disclosed. The augmented reality implementing apparatus comprises: a camera module which takes an image of an educational robot to generate image data, wherein a marker is attached to the educational robot; a metadata extracting unit which extracts first metadata recorded on the marker; a transceiving unit which connects to a content server using the first metadata and requests second metadata from the content server; a generating unit which generates at least one virtual object and an educational scenario for interacting with the educational robot based on the second metadata, when the second metadata is received from the content server; and a display unit which visualizes the at least one virtual object according to the education scenario, augments the image data by using the at least one virtual object, and displays the result.

Description

[0001] The present invention relates to an apparatus and method for realizing an augmented reality interacting with an educational robot,

The present invention relates to an augmented reality realizing device and method for interacting with an educational robot, and more particularly, to an augmenting reality realizing device capable of interacting with an educational robot and using the latest contents.

Augmented reality refers to a technology that fuses or replaces computer-generated virtual objects and information in the user's real world. It is also referred to as mixed reality (MR) because a virtual world with additional information in real time is added to the real world and displayed as a single image.

Since the start of research on augmented reality technology, studies have been carried out to apply this technology in various fields, and researches have been carried out in the field of education to incorporate augmented reality technology.

On the other hand, it is common to use marker recognition in order to realize the augmented reality technology. Specifically, a marker is recognized through a camera, and a virtual object corresponding to the recognized marker is mixed with an actual image. In this case, the content associated with the virtual object is programmed into the marker. If the content size is small, it can be programmed to the marker, but if the content size is large, programming to the marker is limited.

In particular, since the contents used in the education field are large in size and have various educational scenarios, it is difficult to realize an augmented reality through marker recognition. Also, it is difficult to maintain contents once programmed into the markers is difficult to update.

In order to solve the above-described problems, an embodiment of the present invention provides a smart learning environment for a learner by creating and enhancing virtual objects and educational scenarios to interact with the educational robot, thereby providing augmented reality And a method for implementing the same.

Another object of the present invention is to provide an apparatus and method for realizing an augmented reality in which a virtual object and a training scenario are generated by using metadata stored in a content server, will be.

According to another aspect of the present invention, there is provided an apparatus for realizing an augmented reality according to an embodiment of the present invention. The apparatus includes a camera module for photographing an educational robot with a marker and generating image data, A metadata extraction unit for extracting first metadata recorded in the recognized marker, a transmission / reception unit for accessing the content server using the first metadata and requesting the second metadata to the content server, A generating unit for generating at least one virtual object and an educational scenario for interaction with the educational robot based on the second metadata when the second metadata is received, Visualizing a virtual object, and enhancing the visualized at least one virtual object to the image data It includes a display for display over.

Here, the augmented reality implementing apparatus includes a first estimator for detecting a marker from the image data and estimating a posture of the camera module, a second estimator for estimating a posture of the educational robot by detecting the educational robot from the image data, Wherein the control unit controls the first estimating unit and the second estimating unit to estimate the attitude of the camera module and the attitude of the educational robot based on the attitude of the camera module and the estimated attitude of the educational robot, And a control unit for controlling the display unit to adaptively increase the virtual object of the image data on the image data.

Meanwhile, the controller may increase the virtual object on the image data, and then calculate a distance between the educational robot and the virtual object on the image data. If the distance is less than a preset reference value, A collision event is generated between the virtual object and the virtual object, and the collision event can be visualized using X3D (Extensible 3D).

The apparatus for realizing an augmented reality may further include a Bluetooth communication unit for performing Bluetooth communication with the educational robot.

The controller may control the Bluetooth communication unit to transmit the user input signal to the educational robot when a user input signal for controlling the operation of the educational robot is inputted through the user input unit.

Meanwhile, when a collision event occurs between the training robot and the virtual object, the control unit generates a control signal for the training robot to process the collision event, and transmits the generated control signal to the training robot The Bluetooth communication unit can be controlled.

Meanwhile, when the educational robot is located in a predetermined area while the educational robot is controlled according to the user input signal, the controller may control the learning robot based on the user information input signal, It is possible to control the Bluetooth communication unit such that the automatic control of the robot is prioritized and the event information for controlling the educational robot is transmitted to the educational robot.

The first meta data is content meta data including content server information and content related information including storage location information of the second meta data, and the second meta data includes detailed meta data for realizing an augmented reality Information on storage locations of media files used to execute the contents, information on three-dimensional geometry of the educational robot, interaction information with the educational robot in the virtual environment, education scenario information, And specification specification information for enhancement.

Meanwhile, the generation unit may generate the content, the media files, the educational robot extension module, the three-dimensional geometric information of the educational robot, the interaction information with the educational robot in the virtual environment, Information and specification description information for the enhancement, and generate the training scenario and the virtual object.

The marker may be a QR code, and the QR code may include a data area in which the first meta data is recorded, a position detection pattern, a direction detection pattern, and a rectangular pattern formed along an outer periphery of the QR code.

A method for realizing an augmented reality for interaction with an educational robot according to an embodiment of the present invention includes the steps of generating image data by photographing an educational robot to which a marker is attached, recognizing the marker from the image data, Accessing a content server using the first metadata and requesting the second metadata from the content server, receiving the second metadata from the content server Generating at least one virtual object and an educational scenario for interaction with the educational robot based on the second metadata and visualizing the at least one virtual object according to the educational scenario, Augmenting and displaying at least one virtual object on the image data, The.

Here, the method for realizing an augmented reality may further include estimating a posture of the camera module by detecting a marker from the image data, and estimating a posture of the educational robot by detecting the educational robot from the image data, Wherein the step of enhancing and displaying at least one virtual object visualized on the image data includes adaptively arranging the at least one virtual object on the image data according to the estimated attitude of the camera module and the attitude of the educational robot The display can be enhanced.

The method for realizing the augmented reality includes the steps of augmenting the virtual object on the image data, and then calculating a separation distance between the training robot and the virtual object on the image data, and when the distance is less than a preset reference value , Generating a collision event between the training robot and the virtual object, and visualizing the collision event using X3D (Extensible 3D).

The method for realizing the augmented reality further includes transmitting the user input signal to the educational robot using Bluetooth communication when a user input signal for controlling the operation of the educational robot is inputted through the user input means You may.

The method for realizing the augmented reality includes generating a control signal for the training robot to process the collision event when a collision event occurs between the training robot and the virtual object, And transmitting a signal to the educational robot.

Meanwhile, the method of realizing the augmented reality may further include the step of, when the training robot is positioned in a predetermined area while the training robot is controlled according to the user input signal, And transmitting the event information for controlling the educational robot to the educational robot using the Bluetooth communication with priority given to automatic control of the educational robot.

The first meta data is content meta data including content server information and content related information including storage location information of the second meta data, and the second meta data includes detailed meta data for realizing an augmented reality Information on storage locations of media files used to execute the contents, information on three-dimensional geometry of the educational robot, interaction information with the educational robot in the virtual environment, education scenario information, And specification specification information for enhancement.

Meanwhile, the step of generating at least one virtual object and a training scenario for interaction with the educational robot may further include the steps of: using the second metadata, the content, the media files, the educational robot extension module, Dimensional geometry information of the educational robot, interaction information with the educational robot in the virtual environment, education scenario information, and specification description information for the enhancement, and generate the education scenario and the virtual object.

The apparatus and method for realizing the augmented reality according to the embodiment can increase the utilization in the education field by creating and enhancing virtual objects and training scenarios to interact with the educational robot.

In addition, since the metadata is received by accessing the content server through the marker recognition, there is no need to consider the maintenance related to updating of the content, the latest contents can be always used, and the virtual objects and education Since the scenario is created, the size of the marker can be minimized.

1 is a diagram illustrating an augmented reality system according to an embodiment of the present invention.
2 is a block diagram illustrating a configuration of an augmented reality realization apparatus according to an embodiment of the present invention.
3A and 3B are views for explaining a camera position estimation method using a QR code according to an embodiment of the present invention.
4 is a flowchart illustrating a method for implementing an augmented reality according to an embodiment of the present invention.
5 to 7 are views showing an embodiment of an augmented reality according to various embodiments of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. Also, terminologies used herein are terms used to properly represent preferred embodiments of the present invention, which may vary depending on the user, intent of the operator, or custom in the field to which the present invention belongs. Therefore, the definitions of these terms should be based on the contents throughout this specification. Like reference numerals in the drawings denote like elements.

1 is a diagram illustrating an augmented reality system according to an embodiment.

Referring to FIG. 1, an augmented reality system 1000 includes an augmented reality implementation apparatus 100, a content server 200, and an educational robot 300.

The augmented reality realization device 100 interacts with the educational robot 300 using an augmented reality technology, and can be a display device such as a TV or a monitor. A laptop, a tablet PC, or a smart Phone or the like. Further, the display device can be a display device having or having a two-dimensional or three-dimensional display function. However, the augmented reality realization device 100 is not limited thereto, and may be other devices capable of providing augmented reality technology through a display configuration.

The augmented reality realization device 100 may include a camera module 110 for photographing the educational robot 300. If the augmented reality realization device 100 does not include a camera module, the camera 10 may be connected to the augmented reality realization device 100.

The educational robot 300 may be a robot for use in learning. This educational robot 300 is a robot that actually produced a robot corresponding to a virtual object in order to enhance the educational experience of students and to generate interest in learning.

The educational robot 300 may have a marker 310 attached thereto. For example, the marker 310 may be a QR code, and the marker 310 may be recorded with the first metadata.

Meanwhile, the camera module 110 can photograph the education robot with the marker 310 attached thereto to generate image data.

The augmented reality implementing apparatus 100 can recognize the marker 310 from the image data generated by the camera module 110 and extract the first metadata recorded in the marker 310. [

Here, the first metadata may be content metadata including content server information and content related information. The content server information may be IP address information or URL information of the content server 200 in which the second meta data is stored, that is, storage location information of the second meta data. The content-related information may include a content type, a content name, a first meta data creation date, a content author, and the like.

The augmented reality implementing apparatus 100 can access the content server 200 using the extracted first meta data and request the content server 200 for the second meta data. This can be performed using the content server information included in the first meta data.

The second meta data is detailed meta data for implementing the augmented reality, and may include content, and may include storage location information of media files used to execute the content. In addition, the second metadata may further include an educational robot extension module, three-dimensional geometric information of the educational robot, interaction information with the educational robot in the virtual environment, educational scenario information, and specification technical information for the enhancement.

The augmented reality implementing apparatus 100 can generate a virtual object and a training scenario to interact with the educational robot 300 using the second metadata.

The augmented reality implementing apparatus 100 can implement a virtual object according to an educational scenario, and display the augmented virtual object on the image data. The augmented reality implementing apparatus 100 estimates the attitude of the camera 10 and the attitude of the educational robot 300 and estimates the attitude of the educational robot 300 based on the attitude of the camera 10 and the attitude of the educational robot 300, As shown in FIG.

In addition, the augmented reality implementing apparatus 100 can display a training scenario indicating that a virtual object is to be augmented in a certain order, a screen, a method, etc. according to a user's request. Accordingly, a user or a learner can check the training scenario to control the attitude of the educational robot 300 or to control the reinforcement of virtual objects.

The augmented reality system 1000 according to the present embodiment as described above connects to the content server 200 using the first metadata recorded in the marker 310. When the actual content is not recorded in the marker 310 The size of the non-markers 310 can be reduced.

In addition, the augmented reality system 1000 accesses the content server 200 to receive the second meta data, and uses the second meta data to generate training scenarios and virtual objects necessary for the augmented reality, And it is not necessary for the user to directly manage the content.

Here, the augmented reality implementing apparatus 100 generates the training scenario and the virtual object using the second meta data, but the present invention is not limited thereto. Specifically, the content server 200 may store and manage training scenarios and virtual objects created using the second metadata. Accordingly, the augmented reality implementing apparatus 100 may access the content server 200 to request the second metadata, and acquire information related to the training scenario and the virtual object related to the second metadata. In this case, the second metadata, the educational scenario, the virtual object, and the like may be information periodically updated by the administrator of the content server 200. Therefore, the augmented reality implementing apparatus 100 can always implement the augmented reality using the latest information.

Meanwhile, since the augmented reality system 1000 enhances virtual objects in a form interacting with the educational robot 300, it can improve the immersion and concentration of the learners and provide a smart educational environment.

2 is a block diagram illustrating a configuration of an augmented reality realization apparatus according to an embodiment of the present invention. 1, the augmented reality realization apparatus 100 includes a user input unit 101, a camera module 110, a metadata extraction unit 120, a transmission / reception unit (not shown) A first estimator 150, a second estimator 160, a display unit 170, a Bluetooth communication unit 180, and a controller 190. The first estimator 150, the second estimator 160,

The user input unit 101 may be a keyboard, a keypad, a mouse, a touch screen, or the like for user input. For example, when a user performs an input using a mouse, the user input unit 101 may generate a user input signal corresponding to the input.

The camera module 110 photographs the educational robot 300 to which the marker 310 is attached and generates image data.

The metadata extraction unit 120 can recognize the marker 310 from the image data and extract the first metadata recorded in the recognized marker 310. [

The first metadata may be content metadata including content server information and content related information, and may have a small data capacity. Therefore, considering that the size of the marker 310 is increased in proportion to the capacity and complexity of the programmed or recorded data, the marker 310 recording the first metadata may have a relatively small size. In particular, considering the fact that the marker 310 is attached to an object, the smaller the size of the marker 310, the more the range of use can be increased since the marker 310 can be attached to a small object.

In addition, the first metadata may be content metadata including content server information and content related information. The content server information may be IP address information or URL information of the content server 200 in which the second meta data is stored, that is, storage location information of the second meta data. The content-related information may include a content type, a content name, a first meta data creation date, a content author, and the like.

The first metadata has an XML (Extensible Markup Language) structure, and can be represented in Table 1 below.

TAG ATTRIBUTE DESCRIPTION <? xml version = "1.2" encoding = "utf-8"?> XML metadata version INFORMATION Start / End SUBJECT 3D MODEL Content type URL http://vr.knu.ac.kr/~jppark/Contents/Information/butterfly.txt </ URL> The URL of the content server in which the second metadata is stored NAME BUTTERFLY Content name DATE 2013-03-01 Date of first metadata generation FROM JPPARK Content author

The transceiver 130 may access the content server 200 using the first meta data and request and receive the second meta data to the content server 200. [

When the second metadata is received from the content server 200, the generation unit 140 updates the existing content with the second metadata, and generates the interaction with the educational robot 300 based on the second metadata A virtual object and a training scenario can be generated. The generated virtual object and the training scenario may be stored in a memory (not shown).

The second meta data is detailed meta data for implementing the augmented reality, and may include content, and may include storage location information of media files used to execute the content. For example, if the second metadata includes content related to "butterfly" and storage location information of media files associated with "butterfly "

{
"Model": "http://vr.knu.ac.kr/~jppark/Contents/3DModel/butterfly/butterfly.3ds",
"MarkerName": [
"tree1.patt"
"tree2.patt"
],
"MarkerResources": [
"http://vr.knu.ac.kr/~jppark/Contents/3DModel/butterfly/tree1.3ds"
"http://vr.knu.ac.kr/~jppark/Contents/3DModel/butterfly/tree2.3ds"
],
"MarkerTexture": [
"http://vr.knu.ac.kr/~jppark/Contents/3DModel/butterfly/tree1.png"
"http://vr.knu.ac.kr/~jppark/Contents/3DModel/butterfly/tree2.png"
],
"Material": "http://vr.knu.ac.kr/~jppark/Contents/3DModel/butterfly/butterfly.material",
"Animation": "false"
}

In order to enhance the content related to the "butterfly " and to implement it to interact with the educational robot, it is necessary to create content related to the" butterfly " as a virtual object.

Accordingly, the generating unit 140 may generate a virtual object using a model, a marker resource, a marker texture, a material, and an animation tag included in the second metadata .

For example, if the contents of the second metadata are acquired from the photographs of the "butterfly", the growth process of the "butterfly", the everyday life of the "butterfly", and the like, a training scenario and a virtual object .

In addition, the second metadata may further include an educational robot extension module, three-dimensional geometric information of the educational robot, interaction information with the educational robot in the virtual environment, educational scenario information, and specification technical information for the enhancement.

On the other hand, when a virtual object is augmented on the image data, the posture of the camera 10 must be considered because it must be accurately aligned with the three-dimensional topographic information. In addition, the posture of the educational robot 300 must be considered in order to accurately enhance the virtual objects in the image data including the educational robot 300. [ Here, the posture may be one in which both the position and the direction are considered.

The first estimator 150 may estimate the posture of the camera module 110 by detecting the marker 310 from the image data.

As described above, the QR code corresponding to the marker 310 may include a data area in which the first meta data is recorded, a position detection pattern, a direction detection pattern, and a rectangular pattern. Accordingly, the first estimating unit 150 can estimate the attitude of the camera module 110 using another method depending on whether the position detection pattern, the direction detection pattern, and the square pattern are detected in the QR code.

Specifically, when the position detection pattern and the direction detection pattern are detected in the QR code, the first estimation unit 150 determines the direction of the rectangular pattern using the position detection pattern and the direction detection pattern, Therefore, the direction of the QR code can be determined. The first homography between the 3-dimensional plane where the QR code is located and the first image frame of the image data can be calculated using the position and the size of the QR code on the measured 3-dimensional space.

The initial posture of the camera module 110 can be estimated from the first homography, and the posture according to the movement of the camera module 110 can be estimated from the second image frame of the image data.

On the other hand, when the position detection pattern and the direction detection pattern are not detected in the QR code, the first estimation unit 150 can estimate the posture of the camera module 110 based on the feature points.

When various patterns included in the QR code are used, the posture of the camera module 110 can be estimated quickly and accurately. However, if the distance between the camera module 110 and the QR code is too long, or if some or all of the QR code is damaged in the image data, the position detection pattern, the direction detection pattern, the square pattern, have. Accordingly, in this case, the posture of the camera module 110 can be estimated by detecting the feature points from the image frames constituting the image data.

Specifically, the first estimator 150 may detect the feature points on two consecutive image frames in the image frames constituting the image data, and may match the feature points.

The first estimator 150 may detect feature points using a Feature-Accelerated Segment Test (FAST) detector, and may match the feature points using a sum of squared difference (SSD) algorithm.

The first estimator 150 may calculate the second homography using the matching feature points between two consecutive image frames and estimate the attitude of the camera module 110 from the second homography.

The second estimator 160 may detect the training robot 300 from the image data and estimate the posture of the training robot 300. [ Specifically, the second estimator 160 detects the educational robot 300 in each of the image frames constituting the image data, searches for the position and direction of the educational robot 300 on two consecutive image frames, The attitude of the educational robot 300 can be estimated.

The detection of the educational robot 300 in each of the image frames may use an edge detection method or other known detection methods.

The Bluetooth communication unit 180 can communicate with the educational robot 300 through Bluetooth communication.

The controller 190 controls the overall operation of the augmented reality implementing apparatus 100. [

The control unit 190 checks each image frame constituting the image data and outputs the first estimation unit 150 and the second estimation unit 160 to estimate the attitude of the camera module 110 and the attitude of the educational robot 300 Can be controlled. The posture of the camera module 110 and the posture of the education robot 300 can be estimated in a predetermined time period or in real time.

The control unit 190 may control the display unit 170 to visualize the virtual object according to the estimated posture of the camera module 110 and the education robot 300, to enhance the virtual object on the image data, and display the virtual object. In this case, the control unit 190 can adaptively increase the virtual object according to the motion of the educational robot 300 on the image data. Accordingly, the display unit 170 can enhance and display the virtual object in a form interacting with the educational robot 300. [

The control unit 190 controls the Bluetooth communication unit 180 to transmit the user input signal to the training robot 300 when a user input signal for controlling the operation of the educational robot 300 is inputted through the user input unit 101 can do. Therefore, the user can control the educational robot 300.

In addition, the controller 190 may calculate the distance between the training robot 300 and the virtual object on the image data after the virtual object is augmented on the image data. The control unit 190 generates a collision event between the educational robot 300 and the virtual object when the separation distance is equal to or less than a predetermined reference value and displays the collision event through the display unit 170 using X3D (Extensible 3D) . Here, the distance between the educational robot 300 and the virtual object may be a two-dimensional distance or a three-dimensional distance.

For example, when the display unit 170 provides a two-dimensional display function, the distance between the educational robot 300 and the virtual object may be a two-dimensional distance, and the display unit 170 may provide a three- The distance between the educational robot 300 and the virtual object may be a three-dimensional distance.

When a collision event occurs between the training robot 300 and the virtual object, the control unit 190 generates a control signal for the training robot 300 to process the collision event and transmits the control signal to the training robot 300 The Bluetooth communication unit 180 can be controlled to transmit the data to the Bluetooth communication unit 180. [ Accordingly, the educational robot 300 can process the collision event with the virtual object by changing the posture according to the received control signal.

If the training robot 300 is positioned in a predetermined area while the training robot 300 is controlled according to a user input signal, the controller 190 controls the robot 300 according to the user input signal, The control of the educational robot 300 using the event information can be given priority. Here, the event information is generated by the augmented reality implementing apparatus 100 itself to control the educational robot 300, and may be information corresponding to the event.

The control unit 190 can automatically control the operation of the educational robot 300 by controlling the Bluetooth communication unit 180 to transmit event information for controlling the educational robot 300 to the educational robot 300. [

Since the augmented reality implementing apparatus 100 according to the present embodiment enhances virtual objects in a form interacting with the educational robot 300, it can improve the immersion feeling and concentration of the learners, .

3A and 3B are views for explaining a camera position estimation method using a QR code according to an embodiment. The camera posture estimation method may be performed by the augmented reality realization apparatus 100 shown in Figs.

As shown in FIG. 3A, the marker 310 used in the present invention may be a QR code. The QR code 310 may have a size of 20 mm in width and 20 mm in length for general purpose. The QR code 310 may include a data area in which the first metadata is recorded, a position detection pattern, a direction detection pattern, And may include a formed square pattern.

The square pattern formed along the outer periphery of the QR code 310 is for efficiently recognizing the QR code 310 and can secure 6 pixel spaces for the QR code 310 recognition.

When a virtual object is augmented on the image data, it must be accurately aligned with the 3D terrain information. Therefore, the attitude of the camera module 110 included in the augmented reality implementing apparatus 100 must be considered. Accordingly, it is possible to perform the process of estimating the posture of the camera module 110.

When both the position detection pattern, the direction detection pattern and the rectangular pattern are detected in the QR code 310, the augmented reality implementing apparatus 100 can estimate the posture of the camera module 110 based on the QR code 310 .

Specifically, the augmented reality implementing apparatus 100 can determine the direction of the rectangular pattern using the position detection pattern and the direction detection pattern, and determine the direction of the QR code according to the direction of the rectangular pattern.

The first homography between the 3-dimensional plane where the QR code 310 is located and the first image frame of the image data can be calculated using the position and size of the QR code 310 on the measured three-dimensional space. The initial posture of the camera module 110 can be estimated from the first homography, and the posture according to the movement of the camera module 110 can be estimated from the second image frame of the image data.

In this case, the attitude of the camera module 110 can be estimated using the first homography under the assumption that the three-dimensional space in which the QR code 310 is located is flat. Specifically, it can be estimated using the following equation (1).

(1) where H is the homography, K is the internal parameter of the camera module 110, E is the external parameter of the camera, r ₁ and r ₂ are rotational transformation vectors for the X and Y axes, It is a vector.

As described above, since it is assumed that the three-dimensional space in which the QR code 310 is located is a plane, the rotation about the Z axis, which is unknown through the first homography, can be calculated externally of the X and Y axis rotations have. Specifically, it can be calculated using the following equation (2).

In Equation (2), P denotes a 3x4 projection transformation matrix, and the projection transformation matrix can be used to detect rectangle sets on the QR code 310. [ This result is shown in Fig. 3B.

On the other hand, when at least one of the position detection pattern, the direction detection pattern, and the rectangular pattern is not detected in the QR code 310, the augmented reality implementing apparatus 100 estimates the posture of the camera module 110 based on the feature points .

The augmented reality implementing apparatus 100 can detect the feature points on two consecutive image frames in the image frames constituting the image data and match the feature points. A feature point is a point featured in image frames, and may be an edge, a contact, a point, or the like of an object included in an image frame.

The augmented reality implementing apparatus 100 can calculate the second homography using the matching feature points between two consecutive image frames and estimate the posture of the camera module 110 from the second homography.

On the other hand, since the feature point extraction and the feature point matching are accompanied by a large amount of computation, an image pyramid can be generated to reduce the calculation amount. Specifically, the feature points can be extracted from the image pyramid reduced to 25% of the original image frame. In this process, in order to match feature points between successive image frames, the entire area of the image frame can be divided into a grid pattern and the motion for the entire area can be tracked. At this time, feature points can be extracted from the surrounding region of the motion representative point in the entire region of the image frame. This is for selecting the number of matching points to be matched to a certain number or less, and it is possible not only to reduce the calculation amount of the minutiae point extraction and the minutia matching, but also to reduce the time required for the calculation.

The augmented reality implementing apparatus 100 calculates a second homography using matching feature points between two consecutive image frames and applies a second homography to Equations 1 and 2 to determine a posture of the camera module 110 Can be calculated. In this process, an outlier among the matched minutiae points can be removed by using the RANSAC algorithm (RANdom SAmple Consensus). The second homography can be more robust by removing the outlier.

According to the above-described method of estimating the posture of the camera module 110 according to the present embodiment, the posture of the camera module 110 can be more accurately estimated, and even when various patterns included in the QR code are not detected, It is possible to more accurately estimate the posture of the body 110.

4 is a flowchart illustrating a method for implementing an augmented reality according to an embodiment of the present invention. The method of implementing an augmented reality shown in Fig. 4 can be implemented by the augmented reality implementation apparatus shown in Figs.

The augmented reality implementing apparatus 100 photographs an educational robot 300 having a marker 310 such as a QR code and generates image data in operation 410.

After that, the augmented reality implementing apparatus 100 recognizes the marker 310 from the image data (step 420), and extracts the first metadata recorded in the recognized marker 310 (step 430).

Meanwhile, the augmented reality implementing apparatus 100 accesses the content server 200 using the first meta data and requests the second meta data (step 440), and receives the second meta data from the content server 200 (Step 450).

The augmented reality implementing apparatus 100 generates a virtual object and a training scenario for interaction with the educational robot 300 based on the second metadata 460 and visualizes the virtual object according to the training scenario 470 step). In particular, the second metadata may be used to obtain various content and media files,

The second meta data is detailed meta data for implementing the augmented reality, and may include content, and may include storage location information of media files used to execute the content. In addition, the second metadata may further include an educational robot extension module, three-dimensional geometric information of the educational robot, interaction information with the educational robot in the virtual environment, educational scenario information, and specification technical information for the enhancement. The educational scenarios and virtual objects configurable through the contents and media files and other information can be generated. In addition, this virtual object can be visualized so that it can be implemented in accordance with a training scenario.

Then, the augmented reality implementing apparatus 100 estimates the attitude of the camera module 110 and the attitude of the education robot 300 for more accurate reinforcement of the virtual object (step 480). The posture of the camera module 110 and the posture of the educational robot 300 can be estimated in a predetermined time period or in real time.

The augmented reality implementing apparatus 100 adaptively increases and displays the virtual object on the image data according to the posture of the camera module 110 and the posture of the education robot 300 in operation 490.

Specifically, the augmented reality implementing apparatus 100 generates a collision event between the training robot 300 and the virtual object when it is determined that the distance between the training robot 300 and the virtual object is equal to or less than a preset reference value, (Extensible 3D) can be used to visualize collision events.

When a collision event occurs between the training robot 300 and the virtual object, the augmented reality implementing apparatus 100 generates a control signal for the training robot 300 and transmits the control signal to the training robot 300 via the Bluetooth communication. (300). Therefore, it is possible to change the posture of the educational robot 300 so as to process the collision with the virtual object.

In addition, for other reasons, when the attitude of the educational robot 300 is to be changed, the user can operate the user input means provided in the augmented reality implementing apparatus 100. [ Accordingly, when a user input signal is generated by the user input means, the user input signal can be transmitted to the educational robot 300 using the Bluetooth communication.

According to the present embodiment as described above, the first metadata recorded in the marker 310 can be used to connect to the content server 200, receive the second metadata from the content server 200, It is possible to use the latest second metadata managed by the management server 200. Therefore, there is no need for the user to directly perform management (e.g., update) of the content. In addition, the virtual object is augmented so that the interaction between the educational robot 300 and the virtual object can be enhanced, thereby improving the learner's concentration and concentration.

The augmented reality implementation method as described above may be implemented as a program including an executable algorithm that can be executed in a computer, and the program may be stored in a non-transitory computer readable medium .

Here, the non-transitory readable recording medium is not a medium for storing data for a short time such as a register, a cache, a memory, etc., but means a medium that semi-permanently stores data and can be read by a device. Specifically, the above-described programs can be stored in non-volatile readable recording media such as CD, DVD, hard disk, Blu-ray disk, USB, memory card, ROM, and the like.

5 to 7 are views showing an embodiment of an augmented reality according to various embodiments of the present invention, illustrating an augmented reality implementation mode using an interaction between a training robot and a virtual object.

FIG. 5 is an implementation example of an augmented reality for the growth process of a long beetle, and implements a training scenario for augmenting a virtual object corresponding to the growth process of the beetle according to the position of the marker 510. FIG. Here, since the marker 510 can be attached to an educational robot (not shown), the position of the marker 510 can be the position of the educational robot.

Referring to FIG. 5, a photograph and description corresponding to "egg " correspond to a first virtual object 521, a photograph corresponding to" larvae " and a description corresponding to a second virtual object 522, The description corresponding to the third virtual object 523 and the picture and description corresponding to the "adult" may be the fourth virtual object 524.

The training scenario also augments the first virtual object 521 when the marker 510 is recognized at the first position P1 and augments the second virtual object 522 when it is recognized at the second position P2, The third virtual object 523 is recognized at the third position P3 and the fourth most object 524 is recognized at the fourth position P4.

The first to fourth virtual objects 521 to 524 are augmented according to the educational scenario to display the process of growing the long beetle into egg, caterpillar, pupa and adult on the display screen 500 of the augmented reality realization device can do.

FIG. 6 is an implementation example of an augmented reality for everyday life of a butterfly, which implements a training scenario in which a virtual object corresponding to the daily life of a butterfly is augmented according to the position of the educational robot 610. FIG.

6, a virtual object corresponding to the everyday life of a butterfly includes a first virtual object 621 corresponding to a "butterfly", a second virtual object 622 corresponding to a "peripheral butterfly", a "first flower field" A fourth virtual object 624 corresponding to the "first tree ", a fifth virtual object 625 corresponding to the" second tree ", a third virtual object 623 corresponding to the " A sixth virtual object 626, a seventh virtual object 627 corresponding to the "second flower field " and an eighth virtual object 628 corresponding to the" third tree ".

In addition, the training scenario may include a first virtual object 621 corresponding to a "butterfly " to match the training robot 610 to augment the second virtual object 622 to a second virtual object 628 at a predetermined position, The scenario may be a scenario in which the first virtual object 621 reacts or acts when the robot 610 approaches at least one of the second through eighth virtual objects 622 628.

Additionally, the training scenario may further include scenarios that augment auditory virtual objects in addition to visual virtual objects. For example, when the educational robot 610 moves to the third virtual object 623 corresponding to the "first flower field ", a scenario in which an auditory virtual object such as a swallowing of honey is picked up .

By enhancing the first to eighth virtual objects 621 to 628 in accordance with the training scenario, it is possible to display the everyday life of the butterfly on the display screen 600 of the augmented reality implementing apparatus.

7 is an implementation example of an augmented reality for everyday life of a butterfly, which implements the occurrence of a collision event between the educational robot 710 and a virtual object.

The virtual object corresponding to the everyday life of the butterfly is composed of a first virtual object 721 corresponding to "a peripheral butterfly", a second virtual object 722 corresponding to "first tree", a third virtual object 722 corresponding to " A virtual object 723, a fourth virtual object 724 corresponding to the "second tree ", and the like.

Referring to Fig. 7, the educational scenarios related to the everyday life of the butterfly can be similar to Fig. That is, in the training scenario, the virtual object corresponding to the "butterfly " is augmented to match the educational robot 710, the first to fourth virtual objects 721 to 724 are augmented to a predetermined position, and the educational robot 710 Approaching at least one of the first to fourth virtual objects 721 to 724 can be a scenario for enhancing how the "butterfly" augmented with the educational robot 710 reacts or behaves.

During the display of the everyday life of the butterfly by enhancing the first to fourth virtual objects 721 to 724 in accordance with the educational scenarios described above, a collision between the educational robot 710 and the first to fourth virtual objects 721 to 724 An event 730 may be generated.

For example, the distance between the educational robot 710 and the first to fourth virtual objects 721 to 724 can be calculated on the display screen 700 of the augmented reality implementing apparatus. If the distance between the educational robot 710 and the third virtual object 723 is less than a preset reference value, a collision event 730 may be generated. In this case, the collision event 730 may be visualized using X3D (Extensible 3D).

When a collision event 730 occurs, the augmented reality implementing device generates a control signal for the training robot 710 to process the collision event 730, and transmits the control signal to the training robot 710 Lt; / RTI &gt;

Alternatively, when a collision event 730 occurs, the augmented reality implementing device may receive a user input signal for controlling the operation of the educational robot 710, and may transmit the user input signal to the training robot 710 have.

Accordingly, the educational robot 710 can process the collision event 730 by moving the distance between the educational robot 710 and the third virtual object 723 according to the control signal or the user input signal to be equal to or greater than a preset reference value.

Meanwhile, although not shown in the drawings, the training robot 710 can be controlled in operation in accordance with a user input signal transmitted from the augmented reality implementing device in a general situation. However, if the educational robot 710 is located in a predetermined area while the educational robot 710 is controlled according to the user input signal, the augmented reality realizing device can not control the augmented reality realizing device itself Automatic control can be implemented first. In this case, the augmented reality implementing apparatus can read event information for automatic control of the educational robot 710 and transmit it to the education robot 710. [

7, when the educational robot 710 moves to a predetermined first region a, the augmented reality implementing apparatus moves the educational robot 710 out of the first region a, for example, The training robot 710 can automatically control the robot 710 itself.

Event information according to a predetermined area of the educational robot 710 may be preset and stored in the augmented reality realizing device, and event information may be extracted as the position of the educational robot 710 is sensed.

The apparatus for realizing an augmented reality according to the present invention can adaptively increase a virtual object according to the movement of the educational robot 710 and implement the interaction between the educational robot 710 and the virtual object.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those of ordinary skill in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. This is possible. Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

110: camera module 120: metadata extraction unit
130: Transmitting / receiving unit 140:
150: first estimating unit 160: second estimating unit
170: display unit 180: Bluetooth communication unit
190:

Claims (17)

An apparatus for realizing an augmented reality for interaction with an educational robot,
A camera module for photographing an educational robot with a marker and generating image data;
A metadata extracting unit that recognizes the marker from the image data and extracts first metadata recorded in the recognized marker;
Generating from the content server second metadata corresponding to the first metadata and generating at least one virtual object and an educational scenario for interaction with the educational robot based on the received second metadata, part; And
And a controller for enhancing the at least one virtual object to the image data according to the generated training scenario. The method according to claim 1,
A first estimator for detecting a marker from the image data and estimating a posture of the camera module;
A second estimator for detecting the educational robot from the image data and estimating a posture of the educational robot; And
And a display unit,
Wherein,
Wherein the control unit controls the first estimating unit and the second estimating unit to estimate the attitude of the camera module and the attitude of the educational robot and calculates at least one of the at least one virtual camera and the at least one virtual camera based on the estimated attitude of the camera module and the estimated attitude of the educational robot. Wherein the object is adaptively enhanced on the image data and displayed through the display unit. The method according to claim 1,
Wherein,
And generates a control signal for processing the collision event and transmits the control signal to the educational robot when a collision event occurs between the educational robot and the virtual object. The method of claim 3,
Wherein,
A distance between the educational robot and the virtual object is calculated on the image data after the virtual object is augmented on the image data, and when the distance is less than a preset reference value, And generates a collision event. The method of claim 3,
Wherein,
Wherein when the collision event occurs between the training robot and the virtual object, the collision event is visualized using X3D (Extensible 3D). The method according to claim 1,
A Bluetooth communication unit for Bluetooth communication with the educational robot; And
And a transmission / reception unit connected to the content server and receiving the second meta data. The method according to claim 1,
Wherein,
Wherein the training robot is controlled by transmitting the user input signal to the training robot when a user input signal for controlling the operation of the training robot is inputted. 8. The method of claim 7,
Wherein,
And transmits the event information for controlling the educational robot to the educational robot when the educational robot is positioned in a predetermined area while the educational robot is controlled according to the user input signal. . The method according to claim 1,
Wherein the first metadata includes:
Content metadata including content-server information and content-related information including storage location information of the second metadata,
Wherein the second metadata includes:
A detailed description of the contents of the metadata, storage location information of media files used to execute the contents, information about the educational robot extension module, three-dimensional geometric information of the educational robot, The interaction information, the training scenario information, and the specification description information for the enhancement. The method according to claim 1,
The marker is a QR code,
The QR code includes:
A data pattern in which the first metadata is recorded, a position detection pattern, a direction detection pattern, and a rectangular pattern formed along an outer periphery of the QR code. A method for implementing an augmented reality for interaction with an educational robot,
Capturing an educational robot having a marker attached thereto through a camera module to generate image data;
Recognizing the marker from the image data and extracting first metadata recorded in the recognized marker;
Receiving second metadata corresponding to the first metadata from a content server;
Generating at least one virtual object and a training scenario for interaction with the educational robot based on the received second metadata; And
And augmenting the at least one virtual object to the image data according to the generated training scenario. 12. The method of claim 11,
Detecting a marker from the image data and estimating a posture of the camera module; And
Further comprising the step of estimating a posture of the educational robot by detecting the educational robot from the image data,
Wherein the step of enhancing the at least one virtual object to the image data comprises:
Wherein the at least one virtual object is adaptively enhanced and displayed on the image data according to the estimated camera module posture and the estimated posture of the educational robot. 12. The method of claim 11,
Further comprising: generating a control signal for processing the collision event when the collision event occurs between the training robot and the virtual object, and transmitting the control signal to the training robot. 14. The method of claim 13,
Generating a control signal for the education robot and transmitting the control signal to the education robot,
Augmenting the virtual object on the image data, and calculating a distance between the educational robot and the virtual object on the image data;
Generating a collision event between the educational robot and the virtual object when the distance is less than a preset reference value; And
Further comprising: visualizing the collision event using X3D (Extensible 3D). 12. The method of claim 11,
And transmitting the user input signal to the educational robot to control the educational robot when a user input signal for controlling the operation of the educational robot is inputted. 16. The method of claim 15,
And transmitting event information for controlling the educational robot to the educational robot when the educational robot is positioned in a predetermined area while the educational robot is controlled according to the user input signal Augmented reality. 12. The method of claim 11,
Wherein the first metadata includes:
Content metadata including content-server information and content-related information including storage location information of the second metadata,
Wherein the second metadata includes:
A detailed description of the contents of the metadata, storage location information of media files used to execute the contents, information about the educational robot extension module, three-dimensional geometric information of the educational robot, The interaction information, the training scenario information, and the specification description information for the enhancement.

Images