KR102010023B1 - Method and system for providing mixed reality service - Google Patents

Abstract

The present invention provides a method for providing a mixed reality service of a server, the method comprising: receiving tracking information from an HMD as a mixed reality service is driven, based on the received tracking information, at least one of three-dimensional modeling data and a user interface Generating a virtual image of the virtual image and transmitting the virtual image to the HMD, wherein the tracking information includes a real image photographed by the HMD and identification information about a real space in the real image; The virtual image relates to a method and a system for performing the same, wherein the virtual image is synthesized with the real image in the HMD and displayed as a mixed reality image.

Description

Method and system for providing mixed reality service

The present invention relates to a method and system for providing a mixed reality service, and more particularly, to provide a user interface for processing 3D modeling data through a mixed reality image, and to process 3D modeling data through a hand gesture input. A method and system for providing a mixed reality service.

Recently, as 3D image processing technology has been rapidly developed, services using virtual reality (VR) or augmented reality (AR) have been provided to various fields such as movies, games, and interiors. Mixed reality (MR), which appeared after augmented reality, synthesizes and displays virtual images embodied with images of reality and 3D modeling data, thereby providing a user with an image without a boundary between reality and virtual images.

On the other hand, in the field of building design, a technology has been developed that enables the design data to be three-dimensional modeled data to grasp the structure of the building in a more three-dimensional sense.

However, the application of mixed reality in the field of architectural design to date is only to place virtual small-scale interior structures such as furniture in the real space through the screen of a smartphone.

In the design of interiors or buildings, the user's desire to create and edit spaces more realistically is increasing, and thus a method for providing efficient interaction between the device and the user is required.

The present invention relates to a method and system for providing a mixed reality service for providing a user interface for processing three-dimensional modeling data represented by a mixed reality image.

The present invention also relates to a method and system for providing a mixed reality service for processing three-dimensional modeling data represented by a mixed reality image through a hand gesture input.

The mixed reality service providing method according to the present invention for solving the above problems is a method for providing a mixed reality service of the server, receiving tracking information from the HMD as the mixed reality service is driven, the received tracking Generating a virtual image of at least one of 3D modeling data and a user interface based on the information, and transmitting the virtual image to the HMD, wherein the tracking information includes: a reality image captured by the HMD; It may include identification information for the real space in the reality image, wherein the virtual image is characterized in that the composite image and the composite image is synthesized with the real image in the HMD.

The generating of the virtual image may include identifying a specific area within the real image and when the specific area is identified within the real image, a user interface corresponding to the identified specific area may be identified. Generating the virtual image to be displayed on an area.

The specific area may be a part of a human body or an arbitrary object in the 3D modeling data.

The method may further include performing processing on the 3D modeling data in response to the user input when the user input based on the user's hand gesture input is identified, and generating a virtual image of the processed 3D modeling data. The method may further include regenerating and transmitting the regenerated virtual image to the HMD.

The hand gesture input may be one of an input for generating or processing a virtual object in real space or an input for processing a parasitic object in the 3D modeling data.

The mixed reality service providing system according to the present invention for solving the above problems is based on a head mounted display (HMD) for collecting and transmitting tracking information while using the mixed reality service and based on the tracking information received from the HMD. And a server configured to generate at least one virtual image of dimensional modeling data and a user interface and transmit the generated virtual image to the HMD, wherein the tracking information includes identification information about a reality image photographed by the HMD and a reality space within the reality image. The virtual image may be synthesized with the reality image in the HMD and displayed as a mixed reality image.

The server may be further configured to generate the virtual image such that a user interface corresponding to the identified specific area is displayed on the identified specific area when a specific area is identified within the real image.

The specific area may be a part of a human body or an arbitrary object in the 3D modeling data.

In addition, when a user input based on a user's hand gesture input is identified, the server processes the 3D modeling data in response to the user input, and regenerates a virtual image of the processed 3D modeling data to generate the HMD. It may be characterized in that the transmission to.

The hand gesture input may be one of an input for generating or processing a virtual object in real space or an input for processing a parasitic object in the 3D modeling data.

The mixed reality service providing method and system according to the present invention increases the convenience of a user using the mixed reality service by dynamically providing a user interface for processing 3D modeling data according to a user's needs.

In addition, the method and system for providing a mixed reality service according to the present invention process 3D modeling data through a hand gesture input, thereby providing a more realistic mixed reality service through interaction between a device and a user.

1 is a diagram illustrating a network structure of a mixed reality service providing system according to the present invention.
2 is a view showing the structure of a server according to the present invention.
3 and 4 illustrate the structure of a head mounted display according to the present invention.
5 is a flowchart illustrating a method of providing a mixed reality service according to the present invention.
6 is a diagram illustrating an example of a mixed reality image according to the present invention.
7 is a diagram illustrating an example of a user interface according to the present invention.
8 to 10 illustrate various embodiments of a hand gesture input and an operation according to the present invention.

In the following description of embodiments of the present disclosure, when it is determined that a detailed description of a related well-known configuration or function may obscure the gist of the present disclosure, the detailed description may be omitted.

As used herein, "includes," "can include." And the like refer to the existence of the corresponding function, operation, component, etc. disclosed, and do not limit one or more additional functions, operations, components, and the like. Also, in this specification, "includes." Or "have." And the like are intended to indicate that there is a feature, number, step, action, component, part, or combination thereof described in the specification, and that one or more other features or numbers, step, action, component, part, or It should be understood that they do not preclude the presence or possibility of adding these in advance.

As used herein, the singular forms "a", "an" and "the" include plural forms unless the context clearly indicates otherwise.

The mixed reality service providing method and system according to the present invention may be provided for building design. More specifically, the method and system for providing a mixed reality service according to the present invention display the design data of a building as mixed reality to a plurality of user terminals entering the mixed reality conference room as 3D modeling data, and manage the design information (addition / It can be configured to allow for modification / deletion, etc.).

Hereinafter, with reference to the accompanying drawings will be described the present invention.

1 is a diagram illustrating a network structure of a mixed reality service providing system according to the present invention.

Referring to FIG. 1, a mixed reality service providing system 1 according to an exemplary embodiment of the present invention may include a server 10 providing a mixed reality service and a head mounted display (HMD) 20 provided with a mixed reality service. It may be configured to include.

The server 10 is provided to provide the mixed reality service according to the present invention, and may be a network server, an application server, a domain server, or the like operated by a provider of the mixed reality service.

When the server 10 requests to provide the mixed reality service from the HMD 20, the server 10 generates a virtual image from the 3D modeling data and provides the generated virtual image to the HMD 20. The virtual image provided to the HMD 20 may be displayed as a mixed reality image by being synthesized with the reality image photographed by the HMD 20.

The server 10 may also provide the HMD 20 with various user interfaces as virtual images for performing additional processing on the three-dimensional modeling data. In addition, the server 10 may provide a user interface for processing the 3D modeling data as a virtual image or process the 3D modeling data according to a user input received through the user interface.

In various embodiments of the present disclosure, the processing of the 3D modeling data may include creating, editing, and removing arbitrary objects for the 3D modeling data, managing, storing, and deleting the 3D modeling data. Here, the arbitrary objects may include various objects necessary for the design of the building, such as walls, ceilings, floors, doors, windows, furniture, lighting, and electric / gas installations.

The HMD 20 performs data communication with the server 10 to receive a mixed reality service. The HMD 20 generates a mixed image by synthesizing the virtual image received from the server 10 with the real image photographed by the HMD 20, and displaying the mixed image as a left eye and right eye image, thereby allowing the user to three-dimensionally display the real image. Make your modeling data realistic.

In various embodiments of the present disclosure, the 3D modeling data may be modeling data for building design. In particular, the 3D modeling data may be Industry Foundation Classes (IFC) data as building information modeling (BIM) data, but is not limited thereto. The 3D modeling data for building design may include information related to the structure, form, dimensions, materials, materials, colors, patterns, and facilities of the building.

The server 10 and the HMD 20 are connected to each other through a network to perform data communication, and perform a control operation for a mixed reality service. This will be described later in more detail.

2 is a view showing the structure of a server according to the present invention.

Referring to FIG. 2, the server 10 may include a communication unit 11, a control unit 12, and a storage unit 13.

The communication unit 11 may transmit and receive data with the outside via a network. For example, the communication unit 11 receives a request corresponding to a user input and information necessary for generating a virtual image from the HMD 20, and is synthesized on the real image by the HMD 20 under the control of the controller 12. You can send a virtual image.

The control unit 12 is configured to control the respective components of the server 10 to provide the mixed reality service according to the present invention. More specifically, the controller 12 may include a mixed reality service provider 121 and a virtual image generator 122.

The mixed reality service provider 121 performs various control operations for providing the mixed reality service to the HMD 20.

In one embodiment, the mixed reality service provider 121 may create and manage a user account of the HMD 20 that receives the mixed reality service. The mixed reality service provider 121 may store and manage data generated or acquired through the mixed reality service in association with a user account. For example, the mixed reality service provider 121 stores and manages an ID, a password, and the like as identification information for identifying a user account, and stores modeling data related to the user account corresponding to the user account. Can be stored and managed in the data directory.

As the mixed reality service is driven at the request of the HMD 20, the mixed reality service provider 121 receives information necessary for generating a virtual image, for example, tracking information, from the HMD 20, and receives the received information. May be transferred to the virtual image generator 122. In addition, the mixed reality service provider 121 transmits the virtual image generated from the virtual image generator 122 to the HMD 20, and the virtual image is synthesized with the reality image in the HMD 20 to be composed of the mixed reality image. To be possible.

According to various embodiments of the present disclosure, the mixed reality service provider 121 may transmit a user interface to the HMD 20 as a virtual image generated from the virtual image generator 122. The user interface may be provided for processing three-dimensional modeling data. The virtual image for the user interface may be generated to be displayed on the identified specific area on the real image based on the tracking information. Here, the specific area may be an upper part, a lower part, and one side part on the screen. Alternatively, in various embodiments of the present disclosure, the particular area may be a part of the human body, for example, an area identified as a hand (palm, hand, etc.).

While the mixed reality service is being provided, the mixed reality service provider 121 may identify a user input from tracking information received from the HMD 20, or may receive a request corresponding to the user input identified at the HMD 20. Can be. The mixed reality service provider 121 may process 3D modeling data in response to the identified user input.

In one embodiment, the user input may be a hand gesture input. The mixed reality service provider 121 may identify a user's hand gesture input in the reality image of the tracking information, or may receive a request corresponding to the identified hand gesture input from the HMD 20.

The hand gesture input may be, for example, an input corresponding to a user interface displayed as a mixed reality image. The hand gesture input may be a gesture corresponding to an action of holding a pen, a gesture corresponding to an operation of drawing a trajectory, and a back (or a palm) of the back of the hand (a direction away from the HMD` 20) or a rearward (a direction of approaching the HMD 20). Gestures to move). The hand gesture input may consist of one gesture as described above or a combination of various gestures.

The mixed reality service provider 121 may create, move / edit, or remove an object in response to a user input. The mixed reality service provider 121 stores the processed result as 3D modeling data and transfers the processed 3D modeling data to the virtual image generator 122 to generate a virtual image corresponding to the processed result. Make sure

The virtual image generator 122 may generate a virtual image at the request of the mixed reality service provider 121. The virtual image generator 122 may generate a virtual image of the 3D modeling data and / or a virtual image of the user interface based on the tracking information.

Components of the above-described control unit 12 may be implemented as physically separate devices in the control unit 12, but the technical spirit of the present invention is not limited thereto. That is, the above-described components of the controller 12 may be modularized or programmed in one physical processor. In addition, the above-described components of the controller 12 are merely divided into operational aspects of the controller 12, and at least one or all of the above-described components may be integrated into one component.

The storage unit 13 may store an operating system, a program, software, etc. necessary for the operation of the server 10. In various embodiments of the present disclosure, the storage unit 13 may store and manage at least one 3D modeling data. In one embodiment, the at least one three-dimensional modeling data may be stored in association with the user account and geographic location information.

3 and 4 are views showing the structure of the HMD according to the present invention.

3 and 4, the HMD 20 according to the present invention may be configured as a frame 21. The frame 21 may be formed of a flexible material so as to be easily worn on the user's head or the like, for example, may be formed in a spectacles to be worn on the user's face. The frame 21 may be referred to as a main body portion or a body portion. The frame 21 may be provided with components described below, for example, a camera unit 22, a sensor unit 23, an input unit 24, a display unit 25, and an output unit 26.

The camera unit 22 includes at least one camera module so as to capture an image in front of the user. In various embodiments of the present disclosure, the camera unit 22 may include a depth camera that can identify the shape and depth of the real space (subject) of the user. In addition, the camera unit 22 may include an infrared camera or the like for taking a user's hand gesture or the like.

The sensor unit 23 may include at least one sensor capable of sensing various information about the operation of the HMD 20. In various embodiments of the present disclosure, the sensor unit 23 may include a GPS sensor for detecting a geographical location of the user. In addition, the sensor unit 23 may include a gyro sensor, a speed sensor, an acceleration sensor, and the like, for detecting a movement such as three-dimensional rotation, tilt, and speed of the HMD 20.

The input unit 24 receives various inputs of a user. The input unit 24 may include a touch pad, a keypad, a jog dial, a dome switch, a button, and the like. Alternatively, in one embodiment, when receiving a user's hand gesture through the camera unit 22 as an input, the input unit 24 may be configured to perform an operation of identifying a hand gesture taken through the camera unit 22. Can be. In addition, in various embodiments, the input unit 24 may further include a microphone that receives a sound, such as a user's voice, and processes the sound into electrical voice data.

The display unit 25 may visually display various information processed by the HMD 20. In the HMD 20, the display unit 25 may include a left and right eye display unit 25 corresponding to the left and right eyes, respectively, in a state worn by the user. Accordingly, the HMD 20 allows the user to feel a depth sense corresponding to the parallax of the left eye image and the right eye image, and to experience a more realistic mixed reality space.

The output unit 26 is configured to output information processed by the HMD 20 in various forms such as sound, vibration, and light. To this end, the output unit 26 may include a speaker, a haptic module, an LED lamp, and the like.

The HMD 20 according to the present invention may further include a communication unit 27, a control unit 28, and a storage unit 29 as shown in FIG. 3.

The communication unit 27 may transmit and receive data with the outside through a network. For example, the communication unit 27 transmits various requests and / or information for receiving the mixed reality service to the server 10 through the control of the controller 28, and transmits the 3D modeling data and the server 10 from the server 10. And / or receive virtual images for various objects.

The controller 28 is configured to control each component of the HMD 20 in order to receive the mixed reality service according to the present invention. More specifically, the controller 28 may include the mixed reality service manager 281 and the mixed reality generator 282.

The mixed reality service manager 281 controls an operation related to the mixed reality service provided through the server 10.

In detail, the mixed reality service manager 281 may transmit a request to create / manage a user account or log in to the user account in response to a user input. The mixed reality service manager 281 may receive an ID, a password, etc. from the user as identification information for creating a user account or logging in to the user account to the server 10, and transmit the same to the server 10.

The mixed reality service manager 281 may transmit a request for driving the mixed reality service to the server 10 in response to a user input. As the mixed reality service is driven, the mixed reality service manager 281 may collect tracking information through the camera unit 22 and / or the sensor unit 23, and transmit the collected tracking information to the server 10. .

The mixed reality service manager 281 may receive a virtual image generated based on the tracking information from the server 10. The virtual image received from the server 10 may include a virtual image of a user interface for processing 3D modeling data. The virtual image of the user interface may be generated to be displayed on a specific area on the real image. For example, the virtual image of the user interface may be generated to be displayed on an area identified by a user's hand (palm, hand, etc.) of the real image collected through the camera unit 22.

The mixed reality service manager 281 may transfer the received virtual image to the mixed reality generator 282 to generate a mixed reality image by synthesizing the virtual image and the virtual image photographed through the camera unit 22. .

The mixed reality service manager 281 may identify a user input while the mixed reality service is provided. In one embodiment, the user input may be a hand gesture input of the user. The mixed reality service manager 281 may identify a hand gesture input in the reality image captured by the camera unit 22, and transmit a request corresponding to the identified hand gesture input to the server 10. In one embodiment, the identification of the hand gesture input is performed by the server 10 through the tracking information provided by the mixed reality service manager 281 to the server 10 (more specifically, the reality image and / or spatial identification information in the tracking information). ) May be performed.

The mixed reality service manager 281 may transmit a request for creating, moving / editing, or removing an object to the server 10 in response to the identified user input.

The mixed reality generator 282 generates a mixed reality image by synthesizing the virtual image received from the server 10 with the reality image photographed through the camera unit 22. The generated mixed reality image may be displayed through the display unit 25.

In one embodiment, the mixed reality generator 282 may generate a left eye image and a right eye image of the mixed reality image. The left eye image and the right eye image may be displayed on the left eye display unit and the right eye display unit of the HMD 20, respectively.

The storage unit 28 may store an operating system, a program, software, etc. necessary for the operation of the HMD 20.

In various embodiments of the present disclosure, the structure of the HMD 20 is not limited to that shown in FIG. 4 and may have various structures and shapes.

Hereinafter, a method of providing the mixed reality service through the mixed reality service providing system 1 according to the present invention will be described in more detail. The mixed reality service providing method described below may be performed through an application, a program, or software installed in the HMD 20, or may be performed through an HTTP-based web service. However, the technical idea of the present invention is not limited thereto, and the mixed reality service providing method according to the present invention may be executed through various methods.

5 is a flowchart illustrating a method of providing a mixed reality service according to the present invention.

Referring to FIG. 5, the HMD 20 may receive a user input for driving a mixed reality service (501). The user input for driving the mixed reality service may be received through, for example, driving an application, a program, or a software for providing the mixed reality service, or moving to a web page that provides the mixed reality service.

The HMD 20 may transmit a request for driving the mixed reality service to the server 10 in response to the user input (502).

The server 10 may transmit a driving response to the driving request of the HMD 20 to the HMD 20 (503). In various embodiments, the server 10 performs device authentication and / or security authentication for the HMD 20 to determine whether to provide the mixed reality service to the HMD 20 and to drive the HMD 20. You can send a response.

In various embodiments, the authentication process as described above may not be performed separately.

According to various embodiments of the present disclosure, after initial execution of the mixed reality service, the server 10 and the HMD 20 may perform an operation for creating a user account and / or logging in a user account. In detail, the HMD 20 may transmit an account creation request or an account login request to the server 10 according to a user input. The account creation request or the account login request may include, for example, an ID and a password as identification information of the user. The server 10 may store the identification information of the user included in the account creation request, and may set and load the data directory so that data related to the user may be stored in association with the identification information of the user. Alternatively, the server 10 searches for previously stored user identification information matching the identification information of the user included in the account login request, and if the matching user identification information exists, the server 10 stores the data directory corresponding to the retrieved user identification information. Can be loaded.

In some embodiments, if the server 10 does not separately provide a service related to a user account, the above process may not be performed.

When the driving response of the mixed reality service is received, the HMD 20 may collect tracking information in real time (504). In various embodiments, the HMD 20 may collect tracking information through the camera unit 22 and / or the sensor unit 23. The tracking information collected by the HMD 20 may include a real image, identification information (mesh network information) about a real space in the real image, geographic location information, 3D rotation information, speed information, and the like.

For example, the HMD 20 may determine the current geographical position of the HMD 20 through the GPS sensor.

In addition, the HMD 20 may capture a real image through the camera unit 22 and collect the real image as tracking information. The HMD 20 may identify the shape, depth, and the like of the real space by analyzing an image of the real space in the real image photographed by the camera unit 22. Through the space identification, the HMD 20 may determine the shape, size, location, etc. of the ground of a real space located in front of the user, a building, and the like. The HMD 20 may collect the image analysis result of the real space as tracking information.

In addition, the HMD 20 may detect movement such as three-dimensional rotation, tilt, and speed (movement) of the HMD 20 through the sensor unit 23.

Tracking information collected by the HMD 20 in the present invention is not limited to the above-described spatial identification information and motion information, various information required for generating a virtual image, for example marker recognition information, hand gesture identification Information and the like.

The HMD 20 may transmit the collected tracking information to the server 10 in real time (505).

The server 10 may generate a virtual image based on the received tracking information (506).

In one embodiment, the server 10 may generate a virtual image for any loaded three-dimensional modeling data. In addition, the server 10 may generate a virtual image of a user interface for processing the loaded 3D modeling data. In one embodiment, the server 10 may generate a virtual image of a user interface for an object identified in the real image, for example, as a virtual image corresponding to the real image in the tracking information.

The virtual image of the 3D modeling data may be configured as an exterior of the building or a region within the building according to the geographical position of the HMD 20. Here, an area inside the building may correspond to a front display area (display direction) of the HMD 20 extracted based on tracking information, for example, 3D rotation information, from an exterior or internal image of the building.

The user interface for the processing of 3D modeling data includes a user interface for adding, moving, editing, and removing structures (forms), shapes, dimensions, materials, materials, colors, patterns, and facilities of arbitrary objects of the displayed building. It may include. However, the type of user interface is not limited to the above.

The virtual image of the user interface may be generated such that the user interface is displayed on the corresponding object. The server 10 may identify the object from the real image and place a user interface corresponding to the identified object on the identified object to generate the virtual image.

The user interface may be generated to be displayed on a specific area of the real image. The server 10 may generate a virtual image by arranging a user interface in a specific area on the screen of the HMD 20, for example, an upper part, a lower part, and one side. Alternatively, the server 10 identifies a specific area, for example, a part of the human body (hand, palm, back of the hand, finger, arm, etc.) from the real image, and displays a user interface corresponding to the identified specific area on the identified specific area. Can be created in a virtual image.

The server 10 may transmit the generated virtual image to the HMD 20 (507).

The HMD 20 may generate a mixed reality image by synthesizing the received virtual image with the reality image captured by the camera unit 22 (508). The mixed reality image generated as described above may be an image in which a building corresponding to the 3D modeling data is disposed in the real space where the HMD 20 is located. In addition, the generated mixed reality image may be an image in which at least one user interface capable of performing processing on any object of the reality image for the building or the real space corresponding to the 3D modeling data is disposed.

The HMD 20 may display the generated mixed reality image through the display unit 25 (509). An example of the mixed reality image thus displayed is illustrated in FIGS. 6 and 7. 6 and 7 illustrate examples in which virtual images 602, 603, and 702 of the user interface are displayed on the real images 601 and 701 captured by the camera unit 22 of the HMD 20.

Referring to FIG. 6, the mixed reality image may include a virtual image 602 of a user interface displayed on the identified object in the reality image 601. In addition, the mixed reality image may include a virtual image 603 of a user interface displayed on a specific area of the screen, for example, one side.

Meanwhile, referring to FIG. 7, the mixed reality image may include a virtual image 701 of a user interface displayed on a specific area on the screen, that is, an identified back area of the user. As shown in the embodiment shown in Figure 7, the present invention, when a part of the human body is identified in the reality image, by displaying a virtual image of the various user interface on the identified human body, a more realistic mixed reality image to the user It provides a and to experience the interaction between the user and the device (HMD 20).

While the mixed reality image is displayed, the HMD 20 continues to collect tracking information in real time and send it to the server 10. In addition, the server 10 continuously generates a virtual image based on the received tracking information and transmits it to the HMD 20. The HMD 20 may display the mixed reality image to the user by synthesizing the virtual image that changes according to the real time tracking information to the reality image.

In various embodiments of the present disclosure, the HMD 20 may identify a user input while collecting tracking information (510).

The HMD 20 may detect a user's hand gesture input in the real image photographed by the camera unit 22. In detail, the HMD 20 may identify a shape, a size, a moving direction, a moving distance, and the like of the user's hand. The HMD 20 may sense a hand gesture input that consists of a single gesture or a combination of at least one of various gestures.

In addition, the HMD 20 may determine an operation corresponding to the identified hand gesture input. An operation corresponding to the hand gesture input may be preset and stored in the storage 29 of the HMD 20. According to various embodiments of the present disclosure, the user input identified through the hand gesture input may be for generating and processing a virtual object in the real space, as well as processing for the parasitic object in the 3D modeling data.

For example, as illustrated in FIG. 8, when a hand gesture input 801 that detects a specific location with a finger is detected, the HMD 20 executes a user interface displayed at a location pointed by the finger or points the finger. User input for selecting an object corresponding to a location may be identified. Alternatively, as shown in FIG. 8, when a combination of a hand gesture input 801 for pointing a specific position with a finger and a hand gesture input 802 for forming an arbitrary trajectory is detected, the HMD 20 may determine that a finger is not present. Move the object or user interface displayed at the pointed position corresponding to the direction of the trajectory, or expand the object or user interface displayed at the position pointed by the finger corresponding to the direction of the trajectory, or the position and trajectory of the finger pointing. A user input for displaying at least one user interface, generating a wall corresponding to the direction of the trajectory, or identifying an electric / gas facility may be identified. In the present invention, the trajectory of the hand gesture may include a trajectory moving in an arbitrary direction, a trajectory forming a circle, a triangle, a square, a polygon, and the like, a trajectory forming a character, and the like.

For example, as illustrated in FIG. 9, when a hand gesture input 901 in a form of holding a pen is detected, the HMD 20 may identify a user input for entering an object generation mode or a memo generation mode. have. Alternatively, as shown in FIG. 9, when a combination of the hand gesture input 901 in the form of holding a pen and the hand gesture input 902 forming an arbitrary trajectory is detected, the HMD 20 corresponds to the trajectory. The user input may be displayed or a user input for generating an object such as a wall, a ceiling, a floor, a door, a window, a furniture, a lighting, an electric / gas installation, and the like may be identified in response to the trajectory.

For example, as illustrated in FIG. 10, when the gesture input 1001 having the fingers spread out is detected, the HMD 20 selects an object corresponding to the position of the hand or identifies the hand as described above. A user input for displaying a user interface corresponding to the specific area may be identified. Alternatively, as shown in FIG. 10, when the hand gesture input 1001 having the fingers spread out and the hand gesture input 1002 moving forward (or backward) are detected, the HMD 20 is located at the position of the hand. A user input for zooming, moving, editing, or displaying a user interface corresponding to an object may be identified.

For example, after a combination of a hand gesture input 901 in the form of holding a pen and a hand gesture input 902 forming an arbitrary trajectory is detected as shown in FIG. 9, then as shown in FIG. 10. When the gesture input 1001 in which the fingers are spread out and the hand gesture input 1002 moving forward (or backward) are detected, the HMD 20 generates an object such as a wall corresponding to the trajectory, or protrudes / Identify user input to depress.

An embodiment of such a hand gesture input is to create three-dimensional modeling data for an object such as a wall, a ceiling, or a floor that exists outside the visible area of the camera unit 22 of the HMD 20 and cannot be identified by the HMD 20. It may be possible for the user to create it manually. Accordingly, the producer of the three-dimensional modeling data can be used to create the three-dimensional modeling data to be effectively generated through manual input even in buildings or objects within the building that are too large or too far to be identified by the HMD 20.

In addition, various hand gestures may be used, such as a hand gesture for pinching or spreading a finger, a hand gesture for specifying the number of extended fingers, and a hand gesture using both hands. In the present invention, the user input corresponding to the user's hand gesture is not particularly limited in kind or form.

The HMD 20 may send the identified user input to the server 10 (511). Alternatively, the HMD 20 may transmit a request corresponding to the identified user input to the server 10.

In another embodiment of the present invention, the identification of the user input may be performed by the server 10. That is, the server 10 may identify a user input through tracking information received in real time from the HMD 20 (512). Since the operation of identifying the user input by the server 10 is similar to the operation of the user input by the HMD 20, detailed description thereof will be omitted.

The server 10 may process 3D modeling data in response to the identified user input (513).

The server 10 may generate an object for the 3D modeling data, move an existing object, or edit or remove an object's color, material, pattern, shape, etc. in response to a user input.

The server 10 may perform management and storage operations on the 3D modeling data, such as storing the processed 3D modeling data, or storing the 3D modeling data before being processed for backup.

Thereafter, the server 10 may generate a virtual image based on the edited 3D modeling data (514) and transmit the generated virtual image to the HMD 20 (515). The HMD 20 may generate a mixed reality image based on the virtual image transmitted from the server 10 (516) and display the mixed reality image (517), thereby confirming the processing result of the 3D modeling data in real time.

Those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential features of the present invention. In addition, the embodiments disclosed in the specification and the drawings merely present specific examples to easily explain and easily understand the contents of the present invention, and are not intended to limit the scope of the present invention. Therefore, the scope of the present invention should be construed that all changes or modifications derived based on the technical spirit of the present invention are included in the scope of the present invention in addition to the embodiments disclosed herein.

1: Mixed Reality Service Delivery System
10: server 11: communication unit
12: control unit 121: mixed reality service providing unit
122: virtual image generation unit 13: storage unit
20: HMD 21: frame
22: camera portion 23: sensor portion
24: input unit 25: display unit
26: output unit 27: communication unit
28: control unit 281: mixed reality service management unit
282: mixed reality generator 29: storage unit

Claims (10)

As a way to provide mixed reality services of servers,
Receiving tracking information from the HMD as the mixed reality service is driven;
Generating at least one virtual image of 3D modeling data and a user interface based on the received tracking information; And
Transmitting the virtual image to the HMD;
The tracking information,
A real image photographed by the HMD and identification information about a real space in the real image,
The virtual image,
The HMD is synthesized with the reality image and displayed as a mixed reality image.
Generating the virtual image,
Identifying a portion of a user's body within the reality image; And
And when the portion of the user's body is identified within the reality image, generating the virtual image such that the user interface is overlaid and displayed on the identified portion of the user's body. delete delete The method of claim 1,
If a user input based on a hand gesture input is identified through the identified part of the user's body, performing processing on the 3D modeling data in response to the user input;
Regenerating a virtual image of the processed three-dimensional modeling data; And
Sending the regenerated virtual image to the HMD. The method of claim 4, wherein the hand gesture input is performed.
And an input for generating or processing a virtual object in the real space or an input for processing a parasitic object in the 3D modeling data. A system that provides mixed reality services
A head mounted display (HMD) for collecting and transmitting tracking information while using the mixed reality service; And
A server for generating at least one virtual image of 3D modeling data and a user interface based on the tracking information received from the HMD, and transmitting the generated virtual image to the HMD;
The tracking information,
A real image photographed by the HMD and identification information about a real space in the real image,
The virtual image,
The HMD is synthesized with the reality image and displayed as a mixed reality image.
The server,
And when a portion of the user's body is identified within the reality image, generating the virtual image such that the user interface is superimposed and displayed on the identified portion of the user's body. delete delete The method of claim 6, wherein the server,
When a user input based on a hand gesture input is identified through the identified part of the user's human body, the 3D modeling data is processed in response to the user input, and the virtual image of the processed 3D modeling data is regenerated to reproduce the virtual input. System characterized in that the transmission to the HMD. The method of claim 9, wherein the hand gesture input is
And an input for generating or processing a virtual object in the real space or an input for processing a parasitic object in the 3D modeling data.

Images