KR101638550B1 - Virtual Reality System using of Mixed reality, and thereof implementation method - Google Patents

Abstract

The present invention relates to a virtual reality system using mixed reality and a method of implementing the same, and more particularly, to a chromatic key screen. An operation recognition camera positioned at one side of the chroma key screen to recognize the operation of the user; An input / output embedded device for processing input or output to a mixed reality in which a virtual environment and a real environment are mixed; An HMD worn on the user head and connected to the input / output embedded device to display the mixed reality; And a virtual reality realization PC connected to the input / output embedded device and implemented by the HMD device on the chroma key screen to implement the virtual reality in the HMD device, wherein the world coordinates through the chroma key screen and the user's HMD The mixed reality including the virtual reality and the motion of the user is implemented in the HMD and displayed by matching and rendering the motion recognition information obtained by the motion recognition camera.
The present invention provides a system and method for mixing a virtual reality and a real environment so that a user can provide a virtual experience with a high sense of immersion and operation.

Description

Technical Field [0001] The present invention relates to a virtual reality system using mixed reality,

The present invention relates to a virtual reality system using mixed reality, and more particularly, to a virtual reality system using a mixed reality that provides a user with immersive virtual experience by mixing a virtual environment and a real environment, and an implementation method thereof will be.

2. Description of the Related Art [0002] Wearable glasses (see Fig. 2 (a)), which can be viewed anytime and anywhere in recent digital information, are reduced in weight and utilized to provide digital information (e.g., local information, photographs, , Restaurants, wikipedia, etc.) on the spot are immediately attracting much attention. This can serve as a new media that can replace portable Internet terminals (eg, smart phones, notebooks) used as media for acquiring existing information.

In other words, it is possible to instantly provide information of interest to users in the field, and it is possible to instantly perform life logging services such as existing Facebook / Twitter / YouTube, and to acquire various information easily. This will have various ripple effects in social / cultural / economic / industrial aspects.

(1) augmented reality authoring interface

Conventional augmented reality authoring methods can be generally classified into programming based low dimensional authoring methods and non - programming based high dimensional authoring methods. Programming-based authoring methods (such as ARToolKit, osgART, and Goblin XNA) have the advantage of being able to optimize performance through programming, but for technical developers who have expertise in computer vision and 3D graphics, It is difficult to do.

The non-programming based authoring method is based on graphical user interface (GUI) -based visual programming, and can be used by relatively unprofessional (ComposAR, CATOMIR, Metaio, etc.). However, it is difficult to apply the existing GUI based 2D interface to 3D space manipulation discussed in this study.

On the other hand, a tangible user interface (TUI) based authoring method enables a more abstract and easy authoring than a GUI method, and can be operated in three dimensions (iaTAR, ARtalet, etc.). However, TUI is mainly useful for manipulating 3D objects in close proximity.

(2) Matching method for augmented reality authoring

The simplest matching method is to match the mirror world based on GPS / compass sensor and is used in most mobile augmented reality applications (Layar, Wikitude AR, Junaio, Sekai camera). It is possible to operate in outdoor environment, and accuracy of matching is very low due to error of GPS / compass sensor information.

For more precise matching, there is a way to use 2D objects. Perform camera tracking and match virtual objects (ARToolKit, ARToolKit Plus, 2D barcode markers) using square markers with clear color contrast. Alternatively, there is a method of matching a 2D object of a natural image to a target (BazAR, Qualcomm AR SDK). You can also apply augmented reality plug-ins to commercial software (Virtools, Max / Maya, Unity3D, RhinoAR, Autodesk 3DS Max, Flash) to match software 3D models to 2D objects. However, this method has a disadvantage in that the object to be matched is limited to a two-dimensional plane.

(3) Manipulation of a 3D object based on a mobile input device in a wearable augmented reality environment The simplest method of manipulating a 3D object using a mobile input device is a method of pressing a button device of a mobile device. However, such discrete user input is not possible to perform high dimensional simultaneous manipulation (three dimensional movement, three dimensional rotation) in 3D space and time delay. As an extension of this, it is also impossible to perform operations with a high degree of freedom in a two-degree-of-freedom input method using a joystick or touch-based input. Using the built-in sensors of the mobile input device (eg, tilt, accelerometer, gyro, compass sensor, etc.) allows manipulation of 3D degrees of freedom and allows for gyroscopes, game controllers, menu selection, text input, scroll gestures, Respectively. However, they also can not be manipulated freely (movement / rotation of 6 degrees of freedom) in 3D space. Generally, an external tracking device must recognize / track the position / rotation of the input device (3D game controller, uWand, VR Wand, etc.) for 3D object manipulation of 6 degrees of freedom.

Most augmented reality (technology for inserting virtual environment on virtual environment) is aimed at getting help through virtual reality in actual environment, so experience virtual reality (such as putting it in completely different environment, ). In most virtual reality interactions, a method of reconstructing hands and feet and inserting objects corresponding to the hands (or feet, etc.) of a virtual environment is applied. However, this has the problem that the immersion feeling is low and the present reconstruction technique is also inferior to the use of chroma key in the real screen in accuracy.

As described above, it is difficult to implement a system and an apparatus for manipulating a 3D object in augmented reality in which a real environment and a virtual environment are mixed, and the system used in the past is difficult to implement in augmented reality, The 3D environment of the virtual environment is manipulated by the virtual environment and the 3D environment of the virtual environment is manipulated so that the actual environment and the virtual environment are visually distinguished and the environment is hardly matched with the relative motion. There is a problem of falling very much.

Korean Patent Laid-Open No. 10-2014-0001167 (published on January 06, 2014) Korean Patent Publication No. 10-2013-0137692 (public date: December 17, 2013)

An object of the present invention is to provide a system and a method for providing a virtual experience with a high sense of immersion and operation by mixing a virtual reality and a real environment.

A first aspect of the present invention to solve the above problems is a chroma key screen. An operation recognition camera positioned at one side of the chroma key screen to recognize the operation of the user; An input / output embedded device for processing input or output to a mixed reality in which a virtual environment and a real environment are mixed; An HMD worn on the user head and connected to the input / output embedded device to display the mixed reality; And a virtual reality realization PC connected to the input / output embedded device and implemented by the HMD device on the chroma key screen to implement the virtual reality in the HMD device, wherein the world coordinates through the chroma key screen and the user's HMD The mixed reality including the virtual reality and the user's motion is displayed on the HMD by matching and rendering the motion recognition information acquired by the motion recognition camera.

Preferably, the motion recognition camera is a depth camera capable of recognizing the motion of the user, and the HMD is equipped with a position tracking sensor, a gyro sensor, and a stereo scoping RGB camera.

Alternatively, the position tracking sensor may be a sensor capable of tracking the position of the HMD using an accelerometer. The gyro sensor may be a six-axis gyro sensor, And may further include an illumination system.

In addition, the embedded input / output device may be any one of a mobile PC, a smart phone, a tablet PC, and a PDA.

According to a second aspect of the present invention, there is provided a virtual reality realization method using the system of the first aspect, comprising: (a) correcting coordinates and setting up a system by the input / output embedded device; (b) the input / output embedded device receiving depth information of the HMD from the motion recognition camera and inputting depth information in the virtual reality; (c) extracting HMD coordinate information including HMD three-dimensional position coordinates and direction coordinates from the information extracted by the sensor installed in the HMD of the input / output embedded device; (d) rendering the virtual reality on the HMD based on the HMD coordinate information; And (e) displaying the rendered image through an output device of the HMD.

In the step (c), it is preferable to set the 3D position coordinates of the HMD based on the depth information and the position information of the HMD acquired by the sensor installed in the HMD. And the relative position in the world coordinate system of the virtual expression room is relatively set.

Preferably, the motion data of the user acquired by the motion recognition camera is set such that a direction and a position are relatively calculated in the world coordinate system and reacted with an object in the virtual reality.

The present invention can be applied to a virtual reality environment in which an object is interacted with a virtual object through the body (hands, feet, legs, etc.) , The feeling of immersion can be increased, and the interacting body can be viewed with the user's own eyes to enhance the operation feeling, and the system construction cost for implementing the system is very low.

In addition, when the specification of the camera in the real environment and the specification of the camera in the virtual reality (FOV, eye offset, etc.) are configured to be the same (Rendering time of virtual reality, Overlap processing after Chromakey is also processed in almost real time) There is a great advantage that the user can feel the reality and immersion feeling that he is in reality.

FIG. 1 is a block diagram of a virtual reality system using mixed reality according to an embodiment of the present invention, and FIG.
FIG. 2 is a usage diagram of a virtual reality system using mixed reality according to an embodiment of the present invention,
3 is a flowchart illustrating a method for implementing a virtual reality using a virtual reality system using mixed reality according to another embodiment of the present invention,
4 is a flowchart illustrating a method of implementing a coordinate system in a virtual reality realizing method using mixed reality according to an embodiment of the present invention.

BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention, and how to accomplish it, will be described with reference to the embodiments described in detail below with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. The embodiments are provided so that those skilled in the art can easily carry out the technical idea of the present invention to those skilled in the art.

In the drawings, embodiments of the present invention are not limited to the specific forms shown and are exaggerated for clarity. Also, the same reference numerals denote the same components throughout the specification.

The expression "and / or" is used herein to mean including at least one of the elements listed before and after. Also, singular forms include plural forms unless the context clearly dictates otherwise. Also, components, steps, operations and elements referred to in the specification as " comprises "or" comprising " refer to the presence or addition of one or more other components, steps, operations, elements, and / or devices.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

1 is a block diagram of a virtual reality system using mixed reality according to an embodiment of the present invention. As shown in FIG. 1, a virtual reality system using mixed reality according to an embodiment of the present invention includes a chroma key screen 100; An operation recognition camera 200 located at one side of the chroma key screen 100 and recognizing the operation of the user; An input / output embedded device 450 for processing input or output to a mixed reality in which a virtual environment is mixed with a real environment; An HMD mounted on the user head and connected to the input / output embedded device 450 to display the mixed reality; And a PC (400) connected to the input / output embedded device (450) for realizing the virtual reality by being projected on the chroma key screen (100) by the HMD device and implementing the virtual reality on the HMD device, Matching the HMD direction coordinate of the user with the motion recognition information obtained by the motion recognition camera 200 and outputting the mixed reality including the virtual reality and the motion of the user to the HMD And displayed.

The virtual reality system using the mixed reality according to this embodiment of the present invention is not an augmented reality system that adds a virtual environment to a real environment, but implements a real environment in a virtual environment, and includes a motion recognition camera 200, And provides a system capable of enhancing the relative operation feeling of an object by matching with a world index system in a virtual reality by using an implementation PC 400 or the like, and enhancing a user's experience immersion degree.

Here, the chroma key screen 100 is a screen that forms a blue screen zone. A person or an object is placed in front of a light blue screen, which is a chroma key, and is photographed with a color camera, ) To a very different screen. It is not easy to use a blue system color for a subject to be fitted, but it can also produce a special effect by consciously using a blue system color on the contrary. This technology process is carried out in an electronic circuit (electronic circuit), so that it can be instantaneously synthesized only by a switch operation.

As described above, the chromakey screen 100 applied in the embodiment of the present invention is a space in which all the walls, the ceiling, and the floor are painted with a uniform blue color, and a space where a person or an object can take motion within the space . It is also preferable to provide an illumination system for forming illumination light so that colors can uniformly enter the camera around the chroma key screen 100.

The motion recognition camera 200 means a depth camera capable of recognizing the motion of an object to be photographed in front. The Kinect device detects, for example, a body and a voice of a person without a separate controller And is reflected in the monitor screen as it is. In the embodiment of the present invention, the motion of the user wearing the HMD is recognized and visualized in three dimensions through the motion recognition camera 200 to provide coordinate data that can be matched with the world coordinate system in the virtual reality.

An HMD (Head Mounted Display) refers to a display device worn on a person's head. In the embodiment of the present invention, mixed reality worn on a user's head and mixed with a real environment and a virtual reality is displayed.

In addition, it is preferable that a position tracking sensor, a gyro sensor, and a stereoscopic RGB camera are installed in the HMD device. The position tracking sensor can track the position of the current HMD using an accelerometer, And the front image of the wearer of the HMD can be captured and acquired by the stereoscopic RGB camera.

As described above, according to the embodiment of the present invention, the world coordinate through the chroma key screen 100, the HMD direction coordinate of the user, and the motion recognition information acquired by the motion recognition camera 200 are matched and rendered, The mixed reality including the virtual reality and the user's motion is implemented in the HMD and displayed.

2 is a schematic diagram illustrating a use of a virtual reality system using mixed reality according to an embodiment of the present invention. 2, when the system according to the embodiment of the present invention is used, the hand movement of the hand is detected through the motion recognition camera 200 by the user's hand (body) through the stereoscopic RGB camera attached to the front of the HMD, In the virtual environment. It is possible to interact with a virtual object and actually provide an operation experience (touch, drag, etc.) that interacts with the position of the hand obtained through the motion recognition camera 200 and the position of the object in the virtual environment .

3 is a flowchart illustrating a method for implementing a virtual reality using a virtual reality system using mixed reality according to another embodiment of the present invention. As shown in FIG. 3, the virtual reality realizing method according to the embodiment of the present invention includes: (a) a step S100 of correcting coordinates and setting up a system by the input / output embedded device, ; (b) receiving the depth information of the HMD from the motion recognition camera (200) and inputting depth information of the HMD to the input / output embedded device (S200); (c) extracting HMD coordinate information including the HMD three-dimensional position coordinates and direction coordinates from the information extracted by the sensor installed in the HMD of the input / output embedded device (S300); (d) rendering virtual reality on the HMD based on the HMD coordinate information (S400); And (e) displaying the rendered image through an output device of the HMD (S500).

As described above, the embodiment of the present invention realizes a virtual reality by using the virtual reality system using the mixed reality, and is a method for realizing the virtual reality by using the three-dimensional position coordinates of the HMD acquired by the motion recognition camera 200, And the world coordinates of the virtual reality extracted from the virtual reality realization PC 400, and displays the mixed reality image mixed with the real environment and the virtual reality through the HMD .

As shown in FIG. 3, when the program is started, the operation of the motion recognition camera 200 is controlled by the virtual reality realization PC 400, and the system is set up through the calibration process and prepared for program execution. The motion recognition information (Depth Map) photographed by the motion recognition camera 200 located at the upper end of the chroma key screen 100 is transmitted to the virtual reality implementation PC 400, And processes the input in the virtual reality.

The input / output embedded device extracts HMD coordinate information including HMD three-dimensional position coordinates and direction coordinates from the information extracted by the sensor installed in the HMD, and the virtual reality implementing PC 400 extracts, based on the HMD coordinate information The virtual reality is rendered in the camera viewport of the HMD.

Then, the rendered image is output to the input / output embedded device, and the rendered image is displayed by the HMD device connected to the input / output embedded device.

As described above, the virtual reality realizing method according to the embodiment of the present invention is a virtual reality realizing method in which virtual reality is created through virtual (virtual) So that it is possible to increase the feeling of immersion and to enhance the feeling of operation by allowing the user to see his or her own body interacting with his or her own eyes in order to interact with the user, This is very cheap.

In addition, by providing the method of realizing the virtual reality according to the embodiment of the present invention, the specifications of the camera in the real environment and the specifications (FOV, eye offset, etc.) of the cameras in the virtual reality are configured to be the same (Rendering time of the virtual reality, Overlap processing is almost real-time processing), there is a great advantage that the user can feel realistic feeling and immersion feeling that my body is in the virtual reality.

4 is a flowchart illustrating a method of implementing a coordinate system in a virtual reality realizing method using mixed reality according to an embodiment of the present invention. As shown in FIGS. 2 and 4, the origin of all the actions is the center axis of the experimenter's chest, tracking the position of the hand (the interacting body) with the relative coordinates thereof, It is also found in relative coordinates with the chest as the central axis. The center axis of the user is obtained relatively in the world coordinate system in the virtual environment.

The hand (the interacting body) is calculated by the position of the world coordinate system again, reacts with the objects in the virtual environment, and the direction coordinate of the HMD is converted into the world coordinate system, Direction. That is, in the test bed (TestBed), the HMD direction, position, and body coordinates are input in the TestBed basic coordinates independently of the world coordinate system.

The advantage of the above system and implementation method is that it guarantees the independence of head rotation (related to the direction of movement of the body) of the avatar (the user's virtual environment figure) in the virtual environment and the output screen and interaction Ensuring independence (that is, it is possible to pick up or touch something without looking at it)

Also, when the interacting body (hand) is chroma-keyed with the same setting value of the real / virtual camera and output to the HMD, the virtual environment and heterogeneity can be reduced. In other words, when the real camera and the camera for virtual reality rendering are the same and the chroma key processing is performed fast enough, the processing within the virtual reality frame rate becomes possible.

Referring to FIG. 4, the virtual reality PC 400 extracts a chroma key vector based on an image of a camera installed in the HMD in the system setup step, processes the extracted chroma key vector, A real environment image frame is extracted from a camera frame image, and a virtual reality image and the extracted real environment image are mixed to generate a mixed reality frame image.

The generated mixed reality frame image is transmitted to the HMD device through the input / output embedded device 450, and the HMD device displays the mixed reality image. Here, the hybrid reality frame image analyzes coordinate data calculated by a camera installed in the HMD and various sensors together with three-dimensional position coordinates and direction coordinates of the HMD, and obtains three-dimensional position coordinate and wander coordinate corresponding to the world coordinate system in the virtual reality Transformed, or computed coordinates.

While the invention has been shown and described with respect to the specific embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the invention as defined by the appended claims. Anyone with it will know easily.

100: chroma key screen, 200: motion recognition camera, 300: HMD,
450: Input / output embedded device, 400: Virtual reality PC

Claims (11)

Chromakey screen;
An illumination system for illuminating the chroma key screen with illumination light;
An operation recognition camera positioned at one side of the chroma key screen to recognize the operation of the user;
An input / output embedded device for processing input or output to a mixed reality in which a virtual environment and a real environment are mixed;
An HMD that is worn on a user's head and is connected to the input / output embedded device to display the mixed reality;
And a virtual reality PC connected to the input / output embedded device and configured to implement the virtual reality in the HMD by being projected on the chroma key screen by the HMD,
The mixed reality including the virtual reality and the motion of the user is implemented in the HMD by matching and rendering the world coordinate through the chroma key screen, the user's HMD direction coordinate, and the motion recognition information acquired by the motion recognition camera And a virtual reality system using the mixed reality.
The method according to claim 1,
The motion recognition camera comprises:
And a depth camera for recognizing the motion of the user.
The method according to claim 1,
In the HMD,
A position sensor, a gyro sensor, and a stereoscopic RGB camera.
The method of claim 3,
Wherein the position tracking sensor is a sensor capable of tracking the position of the HMD using an accelerometer.
The method of claim 3,
Wherein the gyro sensor is a six-axis gyro sensor.
The method according to claim 1,
The input / output embedded device includes:
A mobile PC, a smart phone, a tablet PC, and a personal digital assistant (PDA).
A method for implementing a virtual reality using the system of claim 1,
(a) the input / output embedded device corrects coordinates and sets up a system;
(b) the input / output embedded device receives the depth information of the HMD from the motion recognition camera and inputs the depth information in the virtual reality;
(c) extracting HMD coordinate information including HMD three-dimensional position coordinates and direction coordinates from the information extracted by the sensor installed in the HMD, by the input / output embedded device;
(d) rendering the virtual reality on the HMD based on the HMD coordinate information; And
(e) displaying the rendered image through an output device of the HMD,
Wherein the 3D position coordinates of the HMD are set based on the depth information and the position information of the HMD acquired by the sensor installed in the HMD in the step (c).
8. The method of claim 7,
Wherein the position coordinates of the HMD are calculated relative to a direction and a position with respect to a center axis of the user's chest and are relatively set in a world coordinate system in the virtual reality.
9. The method of claim 8,
Wherein the motion data of the user acquired by the motion recognition camera is set so that a direction and a position are relatively calculated in the world coordinate system and reacted with an object in the virtual reality.
delete delete

Images