KR101788248B1 - On-line learning system and method using virtual reality and augmented reality - Google Patents

Abstract

The present invention relates to an online learning system and method using a virtual reality and an augmented reality, and more specifically, to a learning object such as a child or a student in a lower grade by learning character The virtual reality and the augmented reality which provide the learning by interlocking the virtual reality and the augmented reality on the basis of the photographed video of the student and the virtual reality based education image including the character, And an online learning system and method using the same. The present invention synthesizes a photographed image of an actual student photographed from a student terminal with the virtual reality-based image information, and supports an item used in the virtual reality to be applied to the photographed image, thereby enabling a virtual reality and an augmented reality to be simultaneously experienced However, if a character interacting with the teacher moves an event to deliver the item to the student, the process in which the item is transferred from the virtual reality-based image to the shot image or vice versa is displayed visually So that it can support interactive interactions between virtual reality and augmented reality so that experiential education based on items between teachers and students can be achieved. There is an effect to provide.

Description

[0001] The present invention relates to an on-line learning system and method using virtual reality and augmented reality,

The present invention relates to an online learning system and method using a virtual reality and an augmented reality, and more specifically, to a learning object such as a child or a student in a lower grade by learning character The virtual reality and the augmented reality which provide the learning by interlocking the virtual reality and the augmented reality on the basis of the photographed video of the student and the virtual reality based education image including the character, And an online learning system and method using the same.

In addition to the development of the Internet and various communication devices supporting it, a variety of online learning systems are provided for the students and infants of the lower grades, so that the teachers can remotely provide various learning programs to the subjects .

However, existing online learning systems targeting younger students or young children are mostly unilateral deliveries of simple programs or only application or web-based interactive learning services, which makes it difficult to obtain continuous attention from educated individuals who are distracted. There is a lower limit of effectiveness.

In addition, there exists an online video learning service technology in which a real teacher teaches and watches real-time students while online, but this is merely a conversion of offline learning to online, and supplementary services are used as an auxiliary learning tool This is also difficult to keep up with the attention of infants and younger students, and it is difficult to keep attention of distracted students.

In other words, the existing online learning system is limited to one-way communication of information to the teacher, which causes students to feel fatigue easily and lose interest in learning.

Korean Patent Publication No. 10-2008-0006063

The present invention adopts a configuration in which a character performs real-time learning so that infants or younger students can concentrate, thereby helping to increase attention and concentration of a target person through personalized lessons in a new virtual world that can not be provided offline It has its purpose.

According to another aspect of the present invention, there is provided a virtual reality system, comprising: a virtual reality-based training image including a character responsive to a movement of a teacher in a virtual reality; The purpose of this study is to support students to make experiential education based on movement between teachers and students through interactive online interaction between virtual reality and augmented reality. have.

An online learning system using a virtual reality and an augmented reality includes a teacher's voice information and motion information according to a teacher's motion, the teacher's terminal according to an embodiment of the present invention and an education server communicating with one or more student terminals through a communication network. And transmitting the photographed image photographed by the student to the education server, the student terminal receiving the education image from the education server, and the motion image information received from the teacher terminal in a preset background image, Based on the video generated by synthesizing the characters whose operation is changed according to the voice and the voice information, synthesizes the photographed image received from the student terminal in a predetermined area of the virtual reality-based video And a training server for generating and transmitting a training image, The server includes an image processing unit for moving an item displayed in the training image to an area corresponding to the photographed image according to the motion information and transmitting position information and item information to the student terminal upon reaching the corresponding area, And the terminal includes an application unit for receiving the item information and the positional information from the education server and displaying the items in an interactive manner in accordance with the operation of the object to be imaged in the captured image.

In one embodiment of the present invention, the image processing unit displays the item on the training image when receiving usage information for use of the item from the teacher terminal, and displays the item on the training image when the preset event related to the item is generated based on the motion information. And moving the item to an area of the photographed image corresponding to the event in the training image.

According to an embodiment of the present invention, the application unit recognizes an object to which an item is applied according to the item information in the captured image, and determines, based on the item information and the positional information, Displays the item, identifies an application position of the item according to the item information in the object recognized in the captured image from the initial position, and moves the item to the applicable position and displays the moved item, The item is dynamically changed so that the item is displayed on the display unit, and the taken image displayed with the item is transmitted to the education server.

In one embodiment of the present invention, the item information includes object information to which the item is applied, and application position information on an area or a position to which the item is applied in the object.

According to an embodiment of the present invention, the application unit may change at least one of a state, a size, a direction, an attitude, and a position of the item according to movement of the object in the captured image.

As an example related to the present invention, the training server may classify each of a plurality of shot images received from each of the plurality of student terminals in communication with a plurality of different student terminals, And displaying the information in one.

In one embodiment of the present invention, the education server transmits the collection information to the student terminal when receiving the collection information for collection of the items from the teacher terminal, and the application unit receives the collection information And transfers the final position information on the final position immediately before departure from the photographed image to the education server when the photographed image leaves the photographed image, And displays the item corresponding to the count information at a predetermined position in the character related area of the training image from the final position in the captured image according to the final position information included in the item.

According to an embodiment of the present invention, the image processor calculates a moving direction and a route of the item according to the event, and from the character included in the training image, To the control unit.

According to an embodiment of the present invention, the application unit may change the state of the item according to at least one of a color and a brightness of the photographed image.

In the online learning method using the virtual reality and the augmented reality according to the embodiment of the present invention, the teacher terminal generates motion information according to the teacher's voice information and the motion of the teacher, and transmits the generated motion information to the preset education server. And transmitting the captured image to the training server. The training server transmits the captured image to a predetermined background image, Generating a virtual reality-based image based on the virtual reality-based image, synthesizing the photographed image received from the student terminal in a predetermined area of the virtual reality-based image, and transmitting the generated educational image to the student terminal and the teacher terminal; Wherein the training server is configured to perform the training Moving an item displayed in an image to an area corresponding to the photographed image according to the motion information and delivering position information and item information to the student terminal when reaching the corresponding area; Receiving the information and the positional information, displaying the item in an interactive manner in accordance with the operation of the object to be imaged in the imaged image, and transmitting the imaged image in which the item is displayed to the education server.

The present invention synthesizes a photographed image of an actual student photographed from a student terminal with the virtual reality-based image information, and supports an item used in the virtual reality to be applied to the photographed image, thereby enabling a virtual reality and an augmented reality to be simultaneously experienced However, if a character interacting with the teacher moves an event to deliver the item to the student, the process in which the item is transferred from the virtual reality-based image to the shot image or vice versa is displayed visually So that it can support interactive interactions between virtual reality and augmented reality so that experiential education based on items between teachers and students can be achieved. There is an effect to provide.

In addition, the present invention is characterized in that the item applied to the virtual reality-based image is displayed so as to continuously move to the captured image of the real student synthesized in the virtual reality-based image according to the item delivery operation of the teacher, Is applied to the student in the photographed image based on the augmented reality based on the photographed image of the actual student, the virtual reality and the augmented reality are interlinked, It enables students to realize online communication between students, and at the same time supports students to use the items instead of physical parish through online education. Education To be greatly improved.

1 and 2 are block diagrams of an online learning system using a virtual reality and an augmented reality according to an embodiment of the present invention.
FIG. 3 and FIG. 4 are diagrams illustrating an operation example of an item delivery process of an online learning system using a virtual reality and an augmented reality according to an embodiment of the present invention;
5 is a diagram illustrating an example of a continuous item transfer process between a virtual reality and an augmented reality in an online learning system using a virtual reality and an augmented reality according to an embodiment of the present invention;
6 is a diagram illustrating an operation for displaying an item to interact with a student's movement in an online learning system using a virtual reality and an augmented reality according to an embodiment of the present invention;
FIG. 7 is a flowchart illustrating an online learning method using a virtual reality and an augmented reality according to an embodiment of the present invention.

Hereinafter, detailed embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of an online learning system using a virtual reality and an augmented reality according to an embodiment of the present invention. As shown in FIG. 1, one or more student terminals 20 corresponding to a student such as a child or a younger child, And a training server 100 that communicates with the student terminal 20 and the teacher terminal 10 through a communication network.

The teacher terminal 10 and the student terminal 20 may be a smart phone having a communication function, a portable terminal, a mobile terminal, a personal digital assistant (PDA) A notebook computer, a slate PC, a tablet PC, an ultrabook, and the like, and may be configured as a portable terminal, The communication network can be applied to various widely known wired and wireless communication methods.

The training server 100 establishes a virtual classroom for a class in which the teacher terminal 10 and one or more student terminals 20 participate in response to a request from the teacher terminal 10, The server terminal 100 can access and store one or more student terminals 20 participating in the virtual classroom of the teacher corresponding to the teacher terminal 10 with the teacher terminal 10 and store them.

2, the education server 100 includes a member DB 102 and an education-related contents (contents) in which the member information of the teacher corresponding to the teacher terminal 10 and the member information of the student are stored And a stored content DB 101.

Accordingly, the education server 100 matches the teacher terminal 10 and the student terminal 20 participating in the virtual classroom with each other to match the information received from the teacher terminal 10 and the student terminal 20 And provide the training image generated based on the bases to the teacher terminal 10 and the student terminal 20 corresponding to the virtual classroom.

The training server 100 extracts a background image selected by the teacher terminal 10 from the content DB 101 and displays the motion of the teacher received from the teacher terminal 10 The virtual reality-based image information can be generated by generating a character that interacts with the motion according to the motion information related to the virtual reality, To the student terminal (20).

Accordingly, the education server 100 supports the learning of the student through the character responding to the movement of the teacher in the virtual reality on behalf of the teacher, so that the attention can be concentrated on the student This can increase the learning effect.

However, it is difficult to expect that the learning effect will continue to be maintained when the student is accustomed to the character, although the student may be temporarily interested in the character operating under the control of the teacher in the virtual reality.

Accordingly, the present invention synthesizes a photographed image photographed by a real student on the virtual reality-based image information, supports an item used in the virtual reality to be applicable to the photographed image, If an interactive event is performed by a character interacting with a teacher according to a teacher's movement, the item is transferred from the virtual reality-based image to the captured image or vice versa. Visual augmentation system to support interactive interactions between virtual reality and augmented reality, thereby enabling experiential education based on items between teachers and students. .

Hereinafter, detailed embodiments of the present invention will be described with reference to the drawings based on the above-described configuration.

2 is a block diagram of an online learning system using a virtual reality and an augmented reality according to an embodiment of the present invention. As shown in the figure, the teacher terminal 10 includes a first camera unit 11, And may include a first display unit 12, a first input unit 13, a first communication unit 15, and a first control unit 14, but the present invention is not limited thereto.

The first camera unit 11 provides an image of the teacher to the first controller 14 and the first controller 14 controls the first camera unit 11 based on the image received from the first camera unit 11. [ And transmits the motion information to the education server 100 through the first communication unit 15. [0050]

For this, the first control unit 14 includes an image analysis module for motion recognition, and the first camera unit 11 may be a three-dimensional camera that provides depth information for recognizing the motion of a teacher , The first controller 14 can recognize the motion of the teacher based on the depth information and generate the motion information.

The first input unit 13 may include a microphone to generate a voice signal corresponding to the voice input of the teacher and provide the voice signal to the first control unit 14. The first control unit 14, May generate audio information based on the audio signal, and may transmit the audio information to the education server 100 together with the motion information.

At this time, the first controller 14 may synchronize the motion information and the voice information, and may transmit the synchronized motion information and the voice information to the education server 100. [

The student terminal 20 includes a second camera unit 21, a second display unit 22, a second input unit 23, a second communication unit 25, and an application unit 24 And may further include various components.

At this time, the application unit 24 configured in the student terminal 20 is connected to the second camera unit 21, the second display unit 22, the second input unit 23, and the second communication unit 25), or may be composed of application-related data stored in a storage unit, which is executed by the control unit and is configured in the student terminal, and the application unit (24) It is possible to perform various operations described in the present invention.

Accordingly, the second camera unit 21 provides the photographed image generated by photographing the actual student to the application unit 24, and the application unit 24 transmits the photographed image to the application unit 24 through the second communication unit 25, And transmits the image to the education server 100.

The second input unit 23 may include a microphone to provide a voice signal to the application unit 24 according to the voice input of the student and the application unit 24 may receive the voice signal And transmits the voice information generated according to the signal to the education server 100.

In this case, the application unit 24 can synchronize the photographed image and audio information to the education server 100.

The training server 100 may include a third communication unit 110 and a second control unit 120. The second control unit 120 may include a video synthesis unit 121, And an image processing unit 122. [

At this time, the image processing unit 122 and the image synthesizing unit 121 may be configured as one component.

The second controller 120 may further include components other than the image combining unit 121 and the image processing unit 122. For example, the image combining unit 121 and the image processing unit 122 And the operation unit may be configured in any one of the image synthesis unit 121 and the image processing unit 122. [

First, the third communication unit 110 can receive motion information and audio information from the teacher terminal 10, receive the photographed image and audio information from the student terminal 20, ).

In addition, the education server 100 may include a contents DB 101 storing various contents, and the contents may include a plurality of different background images (or background images), a character related graphic graphic information, and item information on items available in the virtual reality. At this time, the item may also be composed of graphic information.

In addition, the second controller 120 may transmit content related list information stored in the content DB 101 to the teacher terminal 10, and the first controller 14 configured in the teacher terminal 10 may transmit And displays the selected information on the selected content according to the input of the teacher through the first input unit 13 to the education server 100 through the first communication unit 15. [ ).

Accordingly, the second control unit 120 of the education server 100 interacts with the preset background image corresponding to the content selected based on the selection information received from the teacher terminal 10 and the movement of the teacher Related graphic information from the content DB 101 and provide the character related graphic information and the motion information to the image processing unit 122. [

Accordingly, the image processing unit 122 can identify a motion and a posture according to the motion information based on the character-related graphic information and the motion information, synchronize the motion of the character with the motion information, Related graphic information to the image synthesizing unit 121. The image synthesizing unit 121 synthesizes the character-

Also, the second controller 120 of the education server 100 may receive the photographed image from the student terminal 20, and may provide the photographed image to the image synthesizer 121.

The image synthesis unit 121 synthesizes the background image extracted from the content DB 101 and the character-related graphic information synchronized with the motion information through the image processing unit 122 to the background image, Based image information. In this case, the image synthesizing unit 121 may add the character to the background image in an overlay manner.

In addition, the image synthesizing unit 121 may synthesize the photographed image received from the student terminal 20 on the virtual reality-based image information into a predetermined area of the virtual reality-based image information to generate a training image, The second control unit 120 may transmit the training image generated through the image synthesis unit 121 to the teacher terminal 10 and the student terminal 20 through the third communication unit 110. [

At this time, the education server 100 may communicate with a plurality of different student terminals 20 to receive photographed images from each of the plurality of student terminals 20, Based on the virtual reality-based image information, a plurality of different photographed images received from the plurality of student terminals 20 from the plurality of student terminals 120, respectively, into one of a plurality of different preset regions .

The first control unit 14 of the teacher terminal 10 and the application unit 24 of the student terminal 20 transmit the training image received from the training server 100 to the first Can be displayed through the display unit 12 or the second display unit 22 of the student terminal 20. [

Accordingly, the education server 100 can receive the motion information of the teacher's motion in real time and process the motion of the character moving in the virtual reality in real time according to the motion information. Accordingly, Based video information and the training image generated by combining the photographed image received from the student terminal 20 and the virtual reality based video information is transmitted to the teacher terminal 10 and the student terminal 20 ).

At this time, the second controller 120 synchronizes the voice information received from the teacher terminal 10 with the character-related graphic information synchronized with the motion information, and synchronizes the voice information with the character- Based video information including the audio information to the teacher terminal 10 and the student terminal 20 to generate a video image and a video image corresponding to the virtual reality- And the voice information may be output from the teacher terminal 10 and the student terminal 20 together with the voice information.

The second control unit 120 may receive voice information from the student terminal 20 and may transmit the voice information to the teacher terminal 10 and the student terminal 20 by including the voice information in the education image, So that the teacher terminal 10 and the student terminal 20 can output the audio information corresponding to the student.

To this end, each of the teacher terminal 10 and the student terminal 20 may be constituted by a voice output unit for outputting the voice information.

Through the above-described configuration, the education server 100 can display the actions and voices of a teacher who guides a student to a student through a character, so that a student is interested in the character, The educational efficiency can be greatly improved.

In addition, the second controller 120 of the education server 100 may acquire teaching material related contents corresponding to the selection information received from the teacher terminal 10 among the various teaching material related contents for the student education stored in the contents DB 101 Can be applied to the virtual reality-based image information, so that more effective student education can be achieved.

At this time, the textbook-related contents may include text, images, images, graphics, and the like.

The training server 100 may display the item-related content used in the virtual reality in the virtual reality-based image information, and may display the item in the virtual reality-based image information in response to a specific event according to the movement of the teacher. The virtual reality and the augmented reality can be interlinked so that the experiential exchange between the teacher and the student can be performed in the virtual reality, Which will be described in detail below with reference to the drawings.

3 and 4, the education server 100 synthesizes one or more photographed images with the virtual reality-based image information to generate a virtual reality-based image corresponding to the virtual reality-based image information, A training image including an image can be generated.

At this time, the second control unit 120 may transmit the item list information for the list of items available in the virtual reality according to the virtual reality-based image information to the teacher terminal 10, It is possible to receive usage information for use of the item selected in the item list information and extract the item related content corresponding to the usage information from the content DB 101. [

In addition, the image synthesizing unit 121 of the second controller 120 generates an item A corresponding to the item-related content extracted according to the usage information, and displays the item A on the character B displayed on the virtual reality- And the item (A) may be combined with the virtual reality-based image and displayed.

In addition, the image processing unit 122 interlocks with the image synthesizing unit 121 to determine the movement of the teacher based on the motion information, and when the preset event related to the item A is generated according to the determination result, To move the item (A) on the virtual reality-based image according to the virtual reality-based image information.

For example, when one of the at least one shot images included in the training image is selected based on the motion information, the image processing unit 122 generates a virtual reality-based image corresponding to the character B in the virtual- The item (A) displayed in the region can be moved in the training image with the taken image (taken image 3) selected in accordance with the motion information from at least one of the taken images included in the training image.

At this time, the image processing unit 122 generates animation data for movement of the item A in association with the image synthesis unit 121, synthesizes (or displays) the virtual reality-based image in real time, Based on the virtual reality-based image obtained by synthesizing the animation data, the captured image received from the student terminal 20 can be synthesized in real time to generate a training image in real time, and the student terminal 20, (10).

In another example, the image processing unit 122 may determine that the character B displayed on the virtual reality-based image included in the training image based on the motion information matches the item A with a specific shooting image (Photographed image 3) corresponding to the motion information in accordance with the result of the calculation, and when a throwing event occurs in the direction of the photographed image (photographed image 3) The user can move the item A to the selected shot image (shot image 3).

At this time, the image processing unit 122 generates an education image including an animation of the movement of the item A in association with the image composition unit 121, and transmits the education image including the animation to the teacher terminal 10 and the student terminal 20).

In addition, in the above-described configuration, the image synthesizing unit 121 may combine the item A with an item corresponding to the character B, based on the item information related to the item A stored in the content DB 101 The player can display the item A in an area corresponding to the hand of the character B so that the character B can throw the item A. For example,

Accordingly, as shown in FIGS. 4 and 5A, the image processing unit 122 moves the item A from the virtual reality-based image included in the training image to synthesize the virtual reality- (A) related item information from the content DB 101 when the item A reaches the area corresponding to the photographed image (shot image 3) selected according to the motion information among the at least one shot image , Position information (or coordinate information) for the coordinates reached by the item A in an area corresponding to the selected shot image (shot image 3) can be generated.

Also, the second control unit 120 identifies the specific student terminal 20 that has transmitted the selected shot image (shot image 3) corresponding to the motion information, generates the item information and the item information through the image processing unit 122 To the specific student terminal (20).

At this time, the second control unit 120 can receive the identification information of the student terminal 20 together with the photographed image through the third communication unit 110, and transmits the photographed image to the student The terminal 20 can be identified.

On the other hand, the application unit 24 configured in the specific student terminal 20 receives the item information and the position information, and acquires the item information (image information) from the photographed image (photographed image 3) photographed through the second camera unit 21 The object to which the item information is applied can be recognized.

At this time, the item information may include information on an object corresponding to the shooting subject, to which the item information is applied, and application position information on an area or position to which the item A is applied in the object.

For example, as shown in FIG. 5B, the application unit 24 recognizes a face of a student, which is an object corresponding to the item to be photographed and an object to be photographed, according to the item information in the photographed image (photographed image 3) And can recognize an area or a position corresponding to the eye to which the item A is applied in the object corresponding to the face based on the application position information included in the item information.

The application unit 24 determines the initial position of the item A on the captured image (captured image 3) based on the position information (or the initial position information) received from the training server 100, (A) according to the item information at the initial position of the photographed image, and after confirming the application position of the item on the basis of the item information, the item (A) A) to the eye area of the object which is the application position (or application area, wear position) of the object.

6, the application unit 24 recognizes the motion of the object corresponding to the object to be imaged in the captured image and determines the state, size, posture, direction (direction) of the object A according to the motion of the object And the position of the item (A) in response to the movement of the object so that the item (A) interacts with the item.

In other words, the application unit 24 recognizes an object corresponding to an object to be imaged in the captured image, changes at least one of the state, size, posture, direction, and position of the item A according to the motion of the object, (A) is displayed in the photographed image while following the object, so that the item (A) can be displayed as if it is actually worn by the student.

At this time, the item A may be various items such as glasses, a mask, a hat, and the like.

For example, when the position of the eye is changed in the photographed image according to the rotation of the face corresponding to the object according to the movement of the object, the application unit 24 adjusts the posture of the item A And / or < / RTI >

As another example, the second control unit 120 of the education server 100 may provide a plurality of items for controlling the item A, such as application of the dynamic effect associated with the item A, to the teacher terminal 10, The first control unit 14 of the teacher terminal 10 may provide function related control information corresponding to the function information selected by the teacher in the interface related information, To the server (100).

Accordingly, the second control unit 120 of the education server 100 transmits the function-related control information to the student terminal 20, and the application unit 24 of the student terminal 20 transmits the control information The user can change the state of the item A applied to the photographed image on the basis of, for example, an animation such as popping the item A, shaking the item A, or flashing the item A .

In addition, the application unit 24 may execute an animation for changing the color of the item displayed on the photographed image based on at least one of brightness and hue of the photographed image.

In addition, the application unit 24 generates a shot image (shot image 3) in which an item A interacting according to the real-time movement of the item A and the movement of the object is displayed, Lt; / RTI >

At this time, the application unit 24 may generate the shot image (shot image 3) according to the execution of the animation related to the item and transmit it to the education server 100.

Meanwhile, the second controller 120 of the education server 100 generates an education image by synthesizing the photographed image in which the item is displayed through the image synthesis unit 121, with the virtual reality-based image information, Based on the position of the character B displayed on the virtual reality-based image, and transmits the virtual reality-based image to the virtual reality-based image The training image can be provided so that continuous animation of continuous movement of the item applied to the photographed image of the specific student terminal 20 can be confirmed.

Through the above-described configuration, the image processing unit 122 of the education server 100 tracks the operation of the character in response to the movement of the teacher, and responds to the event when an event occurs such as an operation in which the teacher delivers the item to the student Based image included in the training image and transmitting the item information and the position information to the student terminal 20 when the user moves the item from the virtual reality based image included in the training image and reaches the photographed image related area corresponding to the event, To the application unit 24 of the student terminal 20 and the application unit 24 of the student terminal 20 receives the control authority based on the item information and the position information, By applying the item to the object to which the item is applied according to the item information, the teacher actually delivers the item to the student The same experience can be implemented online.

For example, if the item is an eyeglass item, the image processing unit 122 of the education server 100 moves the eyeglass item to the shot image if the character throws the eyeglass item toward the specific shot image in the training image, The application unit 24 of the student terminal 20 recognizes the portion corresponding to the face of the student in the photographed image corresponding to the eyeglass item, The user can place the eyeglass item in the eye area to which the user applies the virtual eyeglass item.

In addition, the application unit 24 of the student terminal 20 moves to the face of the student displayed on the photographed image, and transmits the photographed image including animation worn on the eyes of the student to the education server 100 The training server 100 generates the training image based on the photographed image including the animation and the virtual reality based image information, and then transmits the training image to the teacher terminal 10 and a plurality of different student terminals participating in the education And the teacher and many students participating in the education are provided to confirm the process of delivering the corresponding item.

In addition, the image processing unit 122 of the education server 100 may transmit an item such as a bubble to the student terminal 20 in the above-described manner, and the application unit 24 of the student terminal 20 Executing an animation in which an item state change such as popping up the bubble related item at the time of receiving the touch input through a touch display configured on the student terminal 20, The training server 100 generates a training image based on the photographed image and the virtual reality based image received from the student terminal 20 and transmits the training image to the teacher terminal 10 and a plurality of To the student terminal (20) of the student terminal (20), thereby providing an educational image in which an interactive virtual experience between the teacher and the student is performed. At this time, the second display unit 22 and the second input unit 23 of the student terminal 20 may be configured as the touch display.

In the above-described configuration, the first control unit 14 of the teacher terminal 10, through the interface displayed on the teacher terminal 10 according to the interface-related information received from the education server 100, The second control unit 120 of the education server 100 may transmit the collection information to the student terminal 100 to which the item is delivered, (20).

Accordingly, the application unit 24 of the student terminal 20 moves the item applied to the object at the time of receiving the collection information, thereby causing the item to be removed from the captured image, (Final coordinates) to the training server 100. [0050] FIG.

At this time, the application unit 24 of the student terminal 20 transmits the shot image including the animation of the dropout process of the item to the education server 100, So that the training image can be generated.

In addition, the image processor 122 of the education server 100 may identify the final position in the captured image included in the training image based on the final position information in the training image when receiving the final position information, Displaying an item corresponding to the number-of-times information at the final position identified in the shot image, displaying an item corresponding to the number-of-times information at a predetermined position among the character-related areas included in the training image, And transmits the item related education image moving from the photographed image to the virtual reality based image to the teacher terminal 10 and the student terminal 20 based on the collection information, And can be displayed on the teacher terminal 10 and the student terminal 20.

Accordingly, the education server 100 can not only provide training images for delivering items continuously moving from a virtual reality to an augmented reality, but also provide educational information on the collection of items that are continuously moved from the augmented reality to the virtual reality Can provide images.

According to the above-described configuration, the present invention is characterized in that the item applied to the virtual reality-based image is continuously displayed continuously on the photographed image photographed by the actual student synthesized on the virtual reality-based image according to the item delivering operation of the teacher At the same time, since the item moved to the region of the photographed image is applied to the student displayed on the photographed image based on the augmented reality in the photographed image of the actual student, it is difficult to realize in the existing online education through the interlocking of the virtual reality and the augmented reality It enables teachers to communicate online with teachers through the use of a parish, and at the same time supports students to make use of items instead of physical teaching materials through online education. Experiential Education Assistance can greatly improve the training effect.

That is, the present invention can support delivery of items between a virtual reality and an augmented reality, thereby enabling online experience education between a teacher and a student to be supported.

7 is a flowchart illustrating an online learning method using a virtual reality and an augmented reality according to an embodiment of the present invention.

As shown in the figure, the teacher terminal 10 can generate the motion information according to the teacher's voice information and the motion of the teacher and transmit the motion information to the preset education server 100 (S1).

In addition, the student terminal 20 may transmit the photographed image of the student to the education server 100 (S2).

Thereafter, the education server 100 generates a virtual reality-based video based on the video generated by synthesizing the character whose operation is changed according to the motion information received from the teacher terminal 10, Generates a training image by synthesizing the photographed image received from the student terminal 20 in a predetermined area of the virtual reality-based image, and transmits the training image to the student terminal 20 and the teacher terminal 10 (S3).

In addition, the education server 100 moves an item displayed in the training image to a region corresponding to the photographed image according to the motion information when a predetermined event is generated (S4) based on the motion information, The location information and the item information to the student terminal 20 (S5).

When the student terminal 20 receives the item information and the position information from the education server 100, the student terminal 20 displays the items in an interactive manner in accordance with the operation of the subject to be photographed in the photographed image, To the education server 100 (S6).

Accordingly, the education server 100 generates a training image displayed on the shooting target of the photographed image based on the photographed image in which the item is displayed and the virtual reality-based image, and then transmits the training image to the teacher terminal 10 and the student terminal 20).

The various devices and components described herein may be implemented by hardware circuitry (e.g., CMOS-based logic circuitry), firmware, software, or a combination thereof. For example, it can be implemented utilizing transistors, logic gates, and electronic circuits in the form of various electrical structures.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or essential characteristics thereof. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the scope of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

10: Teacher terminal 20: Student terminal
100: Training Server

Claims (10)

An online learning system using virtual reality and augmented reality, which includes a teacher terminal and an education server communicating with one or more student terminals via a communication network,
A teacher terminal for generating and transmitting motion information according to a teacher's voice information and a teacher's motion;
A student terminal for transmitting a photographed image of a student to the education server and receiving an education image from the education server; And
Generating a virtual reality-based video based on the video generated by synthesizing a character whose operation is changed according to the motion information received from the teacher terminal on a preset background video and the audio information, And a training server for generating the training image and transmitting the synthesized training image,
Wherein the training server displays on the training image an item upon receiving usage information for use of the item selected from the item list information from the teacher terminal and displays the item on the training image when a predetermined event related to the item is generated based on the motion information And an image processing unit for moving the displayed item to an area of the photographed image corresponding to the event and transmitting positional information and item information to the student terminal when reaching the corresponding area,
Wherein the student terminal includes an application unit for receiving the item information and the positional information from the education server and displaying the items so as to interact with each other in accordance with the operation of the object to be imaged in the captured image,
Wherein the application unit recognizes an object to which an item is applied according to the item information in the captured image and displays the item at an initial position according to the position information in the captured image based on the item information and the position information, Based on the position of the character displayed in the virtual reality-based image and recognizing the continuous animation of the continuous movement of the item applied to the photographed image synthesized in the virtual reality-based image, Identifying an application position of the item according to the item information in the object, moving the item to the applicable position and displaying the item, and dynamically changing the item to interact with movement of the object, The above- Online learning system using virtual reality and augmented reality, characterized in that the transfer to.
delete delete The method according to claim 1,
Wherein the item information includes object information to which the item is applied, and application position information on an area or position to which the item is applied in the object.
The method according to claim 1,
Wherein the application unit changes at least one of a state, a size, a direction, an attitude, and a position of the item according to movement of the object in the captured image.
The method according to claim 1,
Wherein the education server displays each of a plurality of photographed images received from each of the plurality of student terminals in one of a plurality of different preset areas in the training image when communicating with a plurality of different student terminals An online learning system using virtual reality and augmented reality.
The method according to claim 1,
The education server transmits the collection information to the student terminal when receiving the collection information for collection of the item from the teacher terminal,
Wherein the application unit moves the item applied to the object when the collection information is received and separates the captured image from the captured image, transmits final position information on the final position immediately before the departure from the captured image to the education server when leaving the captured image,
Wherein the image processing unit displays an item corresponding to the count information from a final position in the captured image according to the final position information included in the training image at a preset position in the character related area of the training image upon receiving the final position information And the virtual reality and the augmented reality.
The method according to claim 1,
Wherein the image processing unit calculates a moving direction and a path of the item according to the event and moves the item from the character included in the training image to an area of the photographing image corresponding to the event, An online learning system using reality and augmented reality.
The method according to claim 1,
Wherein the application unit changes the state of the item according to at least one of color and brightness of the photographed image.
Generating teacher's voice information and motion information according to a teacher's motion and transmitting the generated motion information to a preset education server;
Transmitting a photographed image of the student terminal taken by the student terminal to the education server;
Wherein the training server generates a virtual reality-based image based on the generated video and the audio information by compositing characters whose operation is changed according to the motion information received from the teacher terminal on a preset background image, Generating a training image by synthesizing the photographed image received in a predetermined region of the virtual reality-based image, and transmitting the training image to the student terminal and the teacher terminal;
Wherein the training server moves an item displayed in the training image to a region corresponding to the photographed image according to the motion information when the preset event is generated based on the motion information, To a terminal; And
When the student terminal receives the item information and the position information from the education server, displays the items so that the items interact with each other in accordance with the operation of the subject to be photographed, and transmits the photographed image displaying the items to the education server / RTI >
Wherein the step of transmitting the location information and the item information to the student terminal comprises:
Displaying the item on the training image when the education server receives usage information for use of the item selected from the item list information from the teacher terminal; And
Moving the item displayed in the training image to the region of the photographed image corresponding to the event and transmitting the position information and item information to the student terminal when the training server reaches the corresponding region,
Wherein the step of transmitting the photographed image displaying the item to the education server comprises:
A step of the student terminal recognizing an object to which an item is applied according to the item information in the captured image;
Displaying the item at an initial position according to the position information in the shot image based on the item information and the position information;
And the student terminal moves from the position of the character displayed in the virtual reality-based image to the student terminal so that the continuous animation of the continuous movement of the item applied to the photographed image synthesized in the virtual reality- Identifying an application position of the item according to the item information in the object recognized in the captured image, moving the item to the applicable position, and displaying the moved item;
Dynamically changing the item so that the student terminal interacts with the movement of the object; And
And transmitting the photographed image, in which the item is displayed, to the education server, using the virtual reality and the augmented reality.

Images