KR102467017B1 - Method for augmented reality communication between multiple users - Google Patents

Abstract

An augmented reality communication method between multiple users according to an embodiment of the present invention is an augmented reality communication method between multiple users performed in a main manager computing device, comprising the steps of performing a communication connection with a plurality of on-site user computing devices; obtaining and displaying captured images from the plurality of on-site user computing devices; providing a sub-manager-site user matching interface matching a sub-manager computing device and the site user computing device; and performing augmented reality communication between a sub-manager computing device selected according to the matching interface and a field user computing device.

Description

Augmented reality communication method between multiple users {METHOD FOR AUGMENTED REALITY COMMUNICATION BETWEEN MULTIPLE USERS}

The present invention relates to an augmented reality (AR) communication method between multiple users. More specifically, it relates to an augmented reality communication method between multiple users in which augmented reality communication between multiple users is performed in parallel and shared in real time.

Recently, with the development of information and communication technology (ICT), information provision technology that provides information on various fields in real time to a plurality of subscribers through at least one host server through a long-distance data communication network is active. are being developed

In addition, in recent years, interest in virtual reality (VR), augmented reality (AR), and mixed reality (MR) is rapidly increasing, and with the addition of extended reality (XR), The market for services and devices that provide users with an immersive experience is developing day by day.

Thanks to this trend, in recent various industrial fields, various augmented reality (AR) or mixed reality (MR) data related to the corresponding industrial field are created and managed, and when necessary, based on wired / wireless networks It is performing communication exchanging generated data.

However, in conventional communication, various augmented reality or mixed reality data can be shared in parallel in real time between workers and managers in a one-to-many (1:n) or many-to-many (n:m) manner. The current state of technology development is insufficient.

On the other hand, in general, various facilities for producing a specific product are arranged in an industrial plant, and a corresponding worker is arranged in each facility to sequentially perform tasks according to a process required for product production.

At this time, the workers who are in charge of each field facility part in the industrial plant have a great influence on the efficiency and productivity of the work according to the proficiency of the assembly production process they are in charge of.

In particular, in the case of an unskilled worker, it may be difficult to assemble the process in the facility part in charge of the worker, which leads to a decrease in productivity of the entire product production.

In addition, even in the case of skilled workers, there is a problem of not immediately recognizing the change in work instruction contents, and in particular, there is a problem of time delay as the changed work instruction contents are directly instructed through a manager every time.

In particular, workers wear various items such as noise generation and work helmets and dustproof clothing in the work plant, making it difficult to communicate with workers, which is a major cause of lowering productivity.

Therefore, in existing industrial plants, a display was attached to each field equipment part, and output was output at all times to manage the state of the work process.

However, due to the complex system of each field equipment part in an industrial plant, organic process management was problematic.

In other words, the limitations of management due to the increase in increasingly complex systems in industrial plants, the need for a lot of time to prepare documents for sharing information, the difficulty in grasping the progress of real-time process work, and the lack of communication between workers and managers. There were limitations in management according to problems, untracked work, and change history of work process.

KR 10-1397908 B1

An object of the present invention is to provide an augmented reality communication (AR communication) method between multiple users in which augmented reality communication between multiple users is performed in parallel and shared in real time.

However, the technical problems to be achieved by the present invention and the embodiments of the present invention are not limited to the technical problems described above, and other technical problems may exist.

An augmented reality communication method between multiple users according to an embodiment of the present invention is an augmented reality communication method between multiple users performed in a main manager computing device, comprising the steps of performing a communication connection with a plurality of on-site user computing devices; obtaining and displaying captured images from the plurality of on-site user computing devices; providing a sub-manager-site user matching interface matching a sub-manager computing device and the site user computing device; and performing augmented reality communication between a sub-manager computing device selected according to the matching interface and a field user computing device.

At this time, the performing of the communication connection includes performing the communication connection with a field user computing device selected based on a 3D site map, which is a map formed by 3D modeling of a physical space of a work site.

The method may further include providing an input/output interface for generating virtual content matching a virtual location of the 3D site map and transmitting the virtual content as communication data.

In addition, when the field user computing device is detected in a predetermined area of the actual location of the work site that matches one virtual location of the 3D site map, the virtual content is transmitted to the detected field user computing device. Include more steps.

In addition, the step of displaying a virtual object representing the identification information of the field user on the virtual location of the 3D site map matched to the actual location of the field user located at the work site, and selecting the virtual object on the 3D site map. The method further includes providing an interface for selecting the on-site user computing device to be connected for communication.

In addition, an augmented reality communication method between multiple users according to an embodiment of the present invention is an augmented reality communication method between multiple users performed in a field user computing device, and multilateral augmentation between at least one other field user computing device and a main manager computing device. executing reality communication; figuring out the location of the on-site user computing device in the on-site physical space and surrounding spatial information; providing an interface for generating and displaying an augmented reality image by inputting virtual content matched to spatial coordinates based on the identified spatial location; and generating first virtual content matched to the first spatial coordinates through the interface, and transmitting the generated first virtual content to the other on-site user computing device or main manager computing device.

In addition, the step of receiving second virtual content matched to the second spatial coordinates from the other site user computing device, determining the second spatial coordinates based on the identified surrounding spatial information, and determining the second spatial coordinates on the determined second spatial coordinates The method may further include overlaying and displaying the second virtual content.

An augmented reality communication method between multiple users according to an embodiment of the present invention performs remote augmented reality communication between multiple users in parallel and shares them in real time, so that at least one remote manager can use at least one site. There is an effect of performing data communication with the user at the same time.

In addition, in the augmented reality communication method between multiple users according to an embodiment of the present invention, at least one or more remote managers perform real-time parallel data communication with at least one or more field users, thereby providing guidance for a plurality of field users collectively. There is an effect that can be processed and provided.

In addition, the augmented reality communication method between multiple users according to an embodiment of the present invention provides a table top system that effectively provides work guidance for the field remotely based on virtual content, thereby providing a realistic screen through virtual content. Easy and clear work guidance can be provided to remote field users in real time, and through this, the operator can perform the work more intuitively.

However, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood from the description below.

1 is a conceptual diagram of an augmented reality communication system between multiple users according to an embodiment of the present invention.
2 and 3 are conceptual diagrams for experiencing an augmented reality environment through a field user computing device according to an embodiment of the present invention.
4 is an example of creating virtual content through a manager computing device according to an embodiment of the present invention.
5 is an internal block diagram of a wearable type computing device according to an embodiment of the present invention.
6 is an internal block diagram of a mobile type computing device according to an embodiment of the present invention.
7 is an internal block diagram of a desktop type computing device according to an embodiment of the present invention.
8 is an internal block diagram of a table top type computing device according to an embodiment of the present invention.
9 is a flowchart illustrating an augmented reality communication method between multiple users according to an embodiment of the present invention.
10 is a diagram for explaining a method of performing a communication connection with a plurality of field user computing devices according to an embodiment of the present invention.
11 is an example of a 3D site map according to an embodiment of the present invention.
12 is an example of an interface providing augmented reality communication according to an embodiment of the present invention.
13 is an example of providing an on-site navigation interface according to an embodiment of the present invention.
14 is an example of a conceptual diagram illustrating how augmented reality communication between multiple users is performed according to an embodiment of the present invention.

Since the present invention can apply various transformations and have various embodiments, specific embodiments will be illustrated in the drawings and described in detail in the detailed description. Effects and features of the present invention, and methods for achieving them will become clear with reference to the embodiments described later in detail together with the drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various forms. In the following embodiments, terms such as first and second are used for the purpose of distinguishing one component from another component without limiting meaning. Also, expressions in the singular number include plural expressions unless the context clearly dictates otherwise. In addition, terms such as include or have mean that features or elements described in the specification exist, and do not preclude the possibility that one or more other features or elements may be added. In addition, in the drawings, the size of components may be exaggerated or reduced for convenience of description. For example, since the size and thickness of each component shown in the drawings are arbitrarily shown for convenience of description, the present invention is not necessarily limited to the illustrated bar.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, and when describing with reference to the drawings, the same or corresponding components are assigned the same reference numerals, and overlapping descriptions thereof will be omitted. .

- Augmented reality communication system between multiple users

An augmented reality communication system (hereinafter referred to as “communication system”) between multiple users is a multi-user augmented reality communication system for effectively providing at least one remote manager a work guide within an augmented reality environment to at least one or more on-site users.

Here, the augmented reality environment refers to an environment created by inserting virtual content related to a physical space around a user or a communication counterpart, and may include an augmented reality (AR) environment and a mixed reality environment (MR). The virtual content is virtual content generated by a computing device, and includes labels, text information, images, drawing objects, and 3D entities, and is augmented/mixed generated in response to the physical space or an image captured in the physical space. It is content.

In addition, XR Communication refers to exchange of communication data for mutual communication between different users through a computing device based on immersive media (eg, video, audio) including the virtual content. means activity.

That is, the communication system according to the embodiment can support multilateral communication within an augmented reality environment by communicating virtual content related to a physical space or a captured image as an additional medium in addition to voice and image, which are communication media of general communication.

Communication data may include audio, video, and/or virtual content exchanged between computing devices of different users (eg, a remote manager and an on-site user) over a network. In embodiments, communication data may be transmitted and received on a network basis by a computing device and/or server, and the like.

In an embodiment, the communication system may provide an augmented reality communication environment optimized for the communication environment by providing different communication data communication methods according to the communication environment of the user's computing device.

1 is a conceptual diagram of an augmented reality communication system between multiple users according to an embodiment of the present invention.

Meanwhile, referring to FIG. 1 , in an embodiment of the present invention, such a communication system may include a computing device 100, a server system 500, and a network.

In detail, the computing device 100 and the server system 500 of the communication system may be connected through a network.

Here, the network means a connection structure capable of exchanging information between nodes such as the computing device 100 and the server system 500, and examples of such networks include a 3rd Generation Partnership Project (3GPP) network, LTE (Long Term Evolution) network, WIMAX (World Interoperability for Microwave Access) network, Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network) ), a Bluetooth network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc., but are not limited thereto.

- Server system

In an embodiment, the server system 500 may perform a series of processes for providing an augmented reality communication service between multiple users.

In detail, the server system 500 may relay exchange of communication data transmitted and received on a network basis between the user computing devices 100 and the communication repeater.

In an embodiment, the server system 500 transmits communication data including captured images, virtual contents, and/or voice data taken at a work site to a plurality of computing devices 100 (manager computing devices 2000 and/or Alternatively, multi-party communication may be implemented by relaying exchanges between the on-site user computing devices 1000 .

The server system 500 may include a data relay server, a data processing server, a virtual content database, and a spatial information database.

In detail, the data relay server may include a communication facility for relaying data, and may relay communication data to be transmitted and received between the computing devices 100 through a wired/wireless communication network.

For example, the data relay server may perform a relaying role of providing communication data transmitted and received between the manager computing device 2000 and the field computing device 100 according to the network communication speed.

In addition, the data processing server may process communication data differently depending on the network speed.

Also, virtual content data for implementing an augmented reality environment or a mixed reality environment may be stored in the virtual content database. The virtual content database may store the virtual content as virtual content by matching the virtual content to a real object (eg, a marker) or spatial coordinates.

Further, the virtual content database may serve as a virtual content source that delivers matched virtual content to a physical space around the computing device 100 upon request from the computing device 100 .

In addition, the spatial information database may scan the physical space of a specific area or store information data on the physical space by modeling the 3D space. In addition, feature information obtained by image learning of real objects, markers, etc. in the physical space may be further stored, and the feature information may be matched with spatial information and stored.

That is, the server system 500 transmits both virtual content data and space information data for the physical space around the computing device 100 to provide an augmented reality environment or an augmented reality environment through the computing device 100. can

The server system 500 may include at least one computing server and a database server, and may include processors for data processing and memories for storing instructions for providing a communication relay service.

- computing device

In an embodiment of the present invention, the computing device 100 provides an augmented reality environment to the user, and augments communication with other users by using voice, image, and virtual content corresponding to real objects in the augmented reality environment. You can run real applications.

Hereinafter, for convenience of description, the role will be described by classifying according to the subject, environment, and/or purpose of using the computing device 100 .

In the embodiment, the computing device 100 may be divided into a manager computing device 2000 used by a remote manager and a field user computing device 1000 used by field users at a work site.

Here, the remote manager may be a person who instructs or assists the work of a user at a location remote from the site where the work is performed and provides work guidance in the form of virtual content.

In an embodiment of the present invention, the remote manager may include a main manager capable of overall directing field work, and a sub manager capable of directing field work under the direction of the main manager.

In detail, the main manager can manage at least one or more sub-administrators and at least one or more on-site users, and the sub-administrator can manage at least one or more on-site users while being managed by the main manager.

In addition, the field user may be a person who performs actual work based on the augmented reality environment provided through work guidance received from the work site.

In this case, the manager computing device 2000 may include a main manager computing device and a sub manager computing device.

In detail, the main manager computing device is the main manager's computing device 100 located remotely from the work site, and can transmit/receive data with the sub manager computing device and/or the field user computing device 1000 based on a network. .

In addition, the sub manager computing device, as a sub manager computing device 100 located remotely from the work site, may transmit and receive data with the main manager computing device and/or the field user computing device 1000 based on a network. .

In the following description, the main and / or sub manager computing device (manager computing device 2000) provides an augmented reality environment based on a photographed image received from the field computing device 100 of a field user, and the provided augmented reality interface environment It will be described based on the remote manager's computing device 100 generating communication data to convey intentions to on-site users according to the main manager's input within.

2 and 3 are conceptual diagrams for experiencing an augmented reality environment through the field user computing device 1000 according to an embodiment of the present invention.

In more detail, referring to FIGS. 2 and 3 , the augmented reality application of the on-site user computing device 1000 in an embodiment of the present invention transmits a photographed image of a physical space around the user to the manager computing device 2000. can

In addition, the augmented reality application of the field user computing device 1000 may acquire virtual content generated based on the captured image from the manager computing device 2000 .

In addition, the augmented reality application of the field user computing device 1000 may provide an augmented image providing an augmented reality environment in which virtual content is overlaid on the captured image.

In an embodiment, an augmented image providing an augmented reality environment may include virtual content (labels, text information, images, drawing objects, 3D entities and manuals, etc.) and/or voice data.

The field user computing device 1000 providing such an augmented reality application may be implemented as a wearable type or mobile type computing device according to an embodiment. A detailed description of this will be described later.

4 is an example of creating virtual content through the manager computing device 2000 according to an embodiment of the present invention.

Also, referring to FIG. 4 , in an embodiment of the present invention, the augmented reality application of the manager computing device 2000 may share captured images obtained from a plurality of on-site user computing devices 1000 in real time.

In addition, the augmented reality application of the manager computing device 2000 may provide an interface for generating an augmented image based on a shared captured image.

In addition, the augmented reality application of the manager computing device 2000 may transmit the augmented image generated based on the provided interface to the field user computing device 1000 to provide an augmented reality environment based on the captured image.

The manager computing device 2000 providing such an augmented reality application may be implemented as a desktop or table top type computing device according to an embodiment. A detailed description of this will be described later.

Meanwhile, the above computing device 100 may include various types (eg, wearable type, mobile type, desktop type or table top type) of computing devices 100 in which augmented reality applications are installed.

1. Wearable type computing device (101, 102)

2 is a conceptual diagram of experiencing an augmented reality environment through wearable type computing devices 101 and 102 according to an embodiment of the present invention, and FIG. 5 is a wearable type computing device 101 and 102 according to an embodiment of the present invention. ) is the internal block diagram.

In an embodiment of the present invention, the wearable type computing devices 101 and 102 may be used as the field user computing device 1000 .

Computing devices 101 and 102 according to embodiments may include wearable type computing devices 101 and 102 such as smart glasses display or head mounted display (HMD).

The smart glasses-type computing device 101 is a display system including glasses that display virtual content (eg, virtual object images) on the user's field of view while transmitting light so that the user can see the surrounding physical space while being worn. can include

In particular, an embodiment computing device 101 may include a transparent glass display that transmits light from the surrounding physical space to reach the user's eyes while simultaneously reflecting virtual content displayed by the display system towards the user's eyes. can

For example, referring to FIG. 2 , the augmented reality application of the computing device 101 may recognize an image of a real object RO and a marker MK within the real object in the surrounding physical space 20, and recognize It is possible to control to display the virtual content VC1 in the field of view of the user corresponding to the marked marker MK.

In addition, the augmented reality application can recognize the learned real object, and can control to display virtual content VC2 in the user's field of view corresponding to the position of the recognized real object.

In addition, the augmented reality application may recognize the physical space through the learned real object or marker (MK), and display virtual content matched to a specific position in the recognized space to correspond to the user's field of view.

Such virtual content may include visual content such as an image or video that may be displayed in a portion of a user's field of view on the computing device 101 . For example, virtual content may include virtual object images overlaying various parts of a physical space. These virtual object images can be rendered as 2D images or 3D images.

A computing device 102 of the head mounted display type may block light to the surrounding physical space so that displayed images can only be viewed by the display system. The head-mounted display-type computing device 102 may output a 3D image by outputting different images with a parallax offset to the left and right eye displays, respectively, in order to recognize a 3D scene.

In addition, the head-mounted display type computing device 102 may also provide an augmented reality environment by outputting an image of a surrounding physical space and virtual content generated based on the captured image as a 3D image.

Hereinafter, specific components will be described focusing on the smart glasses type computing device 101 among these wearable type devices.

Referring to FIG. 5 , a computing device 101 according to an exemplary implementation includes a memory 1000 including an augmented reality application, a processor assembly 2000, a communication module 130, an interface module 140, an input system ( 150), a sensor system 160 and a display system 170. Also, the components may be implemented to be included in the housing of the computing device 101 .

The memory 1000 stores an augmented reality application 111, and the augmented reality application 111 may include virtual content, an image buffer, a location engine, a virtual content display engine, and the like for providing an augmented reality environment. That is, memory 1000 may store commands and data that may be used to create an augmented reality environment.

In an embodiment, the augmented reality application 111 may include a communication application for performing communication based on an augmented reality environment. The communication application may include various applications, engines, data, and instructions for providing an augmented reality communication service between multiple users.

In addition, the memory 1000 may include at least one or more non-transitory computer-readable storage media and temporary computer-readable storage media. For example, the memory 1000 may be various storage devices such as ROM, EPROM, flash drive, hard drive, etc., and web storage that performs the storage function of the memory 1000 on the Internet can include

The processor assembly 2000 may include at least one or more processors capable of executing commands of the augmented reality application 111 stored in the memory 1000 in order to perform various tasks for creating an augmented reality environment.

In an embodiment, the processor assembly 2000 may control overall operations of components through the augmented reality application 111 of the memory 1000 in order to provide an augmented reality communication service between multiple users.

For example, the processor assembly 2000 may recognize a real object from an image acquired based on an image sensor, and generate and display an augmented reality image matching virtual content to the recognized real object (computing device 101 ). ) can be controlled.

The processor assembly 2000 may include a central processing unit (CPU) and/or a graphic processor unit (GPU). In addition, the processor assembly 2000 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and controllers. ), micro-controllers, microprocessors, and electrical units for performing other functions.

The communication module 130 may include one or more devices for communicating with other computing devices (eg, server system 500). This communication module 130 may communicate through a wireless network.

In detail, the communication module 130 may communicate with a computing device storing a virtual content source for implementing an augmented reality environment, and may communicate with various user input components such as a controller that receives a user input.

In an embodiment, the communication module 130 may transmit and receive communication data related to an augmented reality communication service between multiple users to and from the server system 500 and/or other computing devices 100 .

This communication module 130, technical standards or communication schemes for mobile communication (eg, LTE (Long Term Evolution), LTE-A (Long Term Evolution-Advanced), 5G NR (New Radio), WIFI) Alternatively, data may be transmitted and received wirelessly with at least one of a base station, an external terminal, and an arbitrary server on a mobile communication network constructed through a communication device capable of performing a short-range communication method.

The sensor system 160 may include various sensors such as an image sensor 161, a position sensor (IMU) 163, an audio sensor, a distance sensor, a proximity sensor, and a contact sensor.

Image sensor 161 may capture images and/or video of the physical space around computing device 101 .

In an embodiment, the image sensor 161 may capture and obtain an image related to an augmented reality communication service between multiple users.

In addition, the image sensor 161 may be disposed on the front or/or rear surface of the computing device 101 to acquire an image by photographing the disposed direction side, and may use a camera disposed toward the outside of the computing device 101. Through this, it is possible to film a physical space such as a work site.

The image sensor 161 may include the image sensor 161 and an image processing module. In detail, the image sensor 161 may process still images or moving images obtained by the image sensor 161 (eg, CMOS or CCD).

In addition, the image sensor 161 may extract necessary information by processing a still image or video obtained through the image sensor 161 using an image processing module, and transmit the extracted information to a processor.

The image sensor 161 may be a camera assembly including one or more cameras. The camera assembly may include a general camera that captures a visible light band, and may further include a special camera such as an infrared camera and a stereo camera.

The IMU 163 may sense at least one of motion and acceleration of the computing device 101 . For example, it may be made of a combination of various position sensors such as accelerometer, gyroscope, and magnetometer. In addition, spatial information on a physical space around the computing device 101 may be recognized by interworking with the location communication module 130 such as a GPS of the communication module 130 .

In addition, the IMU 163 may detect information for detecting and tracking the user's gaze direction and head movement based on the detected position and direction.

Also, in some implementations, the augmented reality application 111 may use the IMU 163 and the image sensor 161 to determine the user's location and orientation in physical space or recognize a feature or object in the physical space. .

The audio sensor 165 can recognize sounds around the computing device 101 .

Specifically, the audio sensor 165 may include a microphone capable of sensing voice input from a user of the computing device 101 .

In an embodiment, the audio sensor 165 may receive voice data of communication data to be transmitted through an augmented reality communication service from a user.

The interface module 140 can communicatively couple the computing device 101 with one or more other devices. Specifically, interface module 140 may include wired and/or wireless communication devices compatible with one or more different communication protocols.

The computing device 101 may be connected to various input/output devices through the interface module 140 .

For example, the interface module 140 may output audio by being connected to an audio output device such as a headset port or a speaker.

Although the audio output device has been described as connected through the interface module 140 as an example, an embodiment installed inside the computing device 101 may also be included.

The interface module 140 connects a device having a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, and an identification module. Ports, audio I/O (Input/Output) ports, video I/O (Input/Output) ports, earphone ports, power amplifiers, RF circuits, transceivers and other communication circuits It may be configured to include at least one of.

The input system 150 may detect a user's input related to an augmented reality communication service between multiple users (eg, gesture, voice command, button operation, or other type of input).

In detail, the input system 150 may include a button, a touch sensor, and an image sensor 161 that receives user motion input.

In addition, the input system 150 may be connected to an external controller through the interface module 140 to receive a user's input.

Display system 170 is transparent glass that transmits light from the physical space around computing device 101 to reach the user's eyes, while reflecting virtual content displayed by display system 170 toward the user's eyes. A display may be included.

The display system 170 may include a left display 171 corresponding to the left eye of a user wearing the computing device 101 and a right display 172 corresponding to the right eye, and the left display 171 and The right display 172 outputs different images offset by parallax as virtual content, so that the user can recognize the virtual content as a 3D image.

In an embodiment, the display system 170 may output various information related to an augmented reality communication service between multiple users as a graphic image.

Such displays include a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. , a 3D display, and an e-ink display.

Such a wearable type computing device 101 may be advantageous for use by a field worker located in a physical space such as a work site.

2. Mobile type computing device 200

3 is a conceptual diagram of experiencing an augmented reality environment through a mobile type computing device 200 according to an embodiment of the present invention, and FIG. 6 is an internal block of the mobile type computing device 200 according to an embodiment of the present invention. It is also

In an embodiment of the present invention, the mobile type computing device 200 may be used as the field user computing device 1000 .

In another example, the computing device 200 may be a mobile device such as a smart phone or a tablet PC having an augmented reality application installed thereon. The mobile type computing device 200 captures an image of a surrounding physical space with an image sensor, displays virtual content matched to the captured image and the physical space through a display system, and provides an augmented reality environment to the user. can

For example, the computing device 200 may include a smart phone, a mobile phone, a digital broadcast terminal, personal digital assistants (PDA), a portable multimedia player (PMP), a tablet PC, and the like.

Referring to FIG. 3 , the augmented reality application of the mobile type computing device 200 may capture images of real objects RO and markers MK in the surrounding physical space 20 and display them by controlling the display system. . In addition, the augmented reality application may control to display the virtual content VC1 at a location corresponding to the recognized marker MK. In addition, the augmented reality application may learn and recognize a specific real object, and control to display virtual content VC2 in the user's field of view corresponding to the position of the recognized specific real object.

Referring to FIG. 6 , a mobile type computing device 200 according to an exemplary implementation includes a memory 210, a processor assembly 220, a communication module 230, an interface module 240, an input system 250, and a sensor. system 260 and display system 270 . These components may be configured to be included within the housing of computing device 200 .

Redundant descriptions of the components of the mobile type computing device 200 will be replaced with descriptions of the components of the wearable type computing device 101. Hereinafter, differences from the wearable type computing device 101 will be described. mainly explained.

The components may be disposed within the housing of the mobile type computing device 200, and the user interface may include a touch sensor 273 on the display 271 configured to receive a user touch input.

In detail, the display system 270 may include a display 271 that outputs an image and a touch sensor 273 that detects a user's touch input.

For example, the display 271 may be implemented as a touch screen by forming a mutual layer structure or integrally with the touch sensor 273 . Such a touch screen may function as a user input unit providing an input interface between the computing device 200 and the user, and may provide an output interface between the computing device 200 and the user.

In addition, the sensor system 260 includes an image sensor 261, and the image sensor 261 may be disposed on one side and the other side of the housing of the computing device 200 as an example.

At this time, the image sensor on one side may be oriented toward the physical space to capture an image of the physical space, and the image sensor on the other side may be oriented toward the user and capture the user's field of view and gesture.

Such a mobile type computing device 200 may be suitable for a field worker located in a physical space such as a work site.

3. Desktop type computing device 300

7 is an internal block diagram of a desktop type computing device 300 according to an embodiment of the present invention.

Redundant descriptions of the components of the desktop type computing device 300 will be replaced with descriptions of the components of the wearable type computing device 101. Hereinafter, differences from the wearable type computing device 101 will be explained based on

Referring to FIG. 7 , in another example, the computing device 300 is a wired/wireless computer such as a fixed desktop PC, a laptop computer, and a personal computer such as an ultrabook in which an augmented reality application 311 is installed. It may further include a device on which a program for executing an augmented reality communication service between multiple users based on communication is installed.

The desktop type computing device 300 receives an image of a surrounding physical space captured by another user's computing device 300, augments and displays virtual content matched to the received image and the physical space, and augments the image. A real environment may be provided to the user.

In addition, the desktop type computing device 300 includes the user interface system 350 to receive user input (eg, touch input, mouse input, keyboard input, gesture input, motion input using a guide tool, etc.) can

Illustratively, the computing device 300 implements the user interface system 350 with a mouse 351, a keyboard 352, a gesture input controller, an image sensor 361 (e.g., a camera), and an audio sensor ( 365) may be connected to at least one device to obtain a user input.

In addition, the desktop type computing device 300 may be connected to an external output device through the user interface system 350, and may be connected to, for example, a display device 370 and an audio output device.

Also, in the embodiment of the present invention, the desktop type computing device 300 may be used as the manager computing device 2000 (main manager computing device and/or sub manager computing device).

As an exemplary implementation, the augmented reality application 311 of the desktop computing device 300 may obtain and output an image of another user's computing device 300 through the display device 370, and correspond to the image. A user input may be received, and virtual content corresponding to an image may be generated according to the received user input.

In addition, the augmented reality application 311 may receive data from the sensor system 260 in the physical space of the image or a previously matched virtual content source, and generate the received data as virtual content matched with the image.

In addition, the augmented reality application 311 may provide an augmented reality environment to the user by overlaying and outputting the virtual content generated in this way to an image output from a display device.

In addition, the augmented reality application 311 transmits the virtual content generated through the communication module 330 as communication data, and can utilize the virtual content corresponding to the physical space along with voice and image as a medium for communication. .

A desktop type computing device 300 according to an example implementation may include a memory 310 , a processor assembly 320 , a communication module 330 , a user interface system 350 and an input system 340 . These components may be configured to be included within the housing of computing device 300 .

Redundant descriptions of the components of the desktop type computing device 300 will be replaced with descriptions of the components of the wearable type computing device 300 .

Such a desktop type computing device 300 may be advantageous for use by a remote manager who transmits instructions or necessary information remotely in conjunction with the computing device 1000 of a field worker.

4. Table top type computing device 400

According to an exemplary implementation, the table top type computing device 400 may be implemented in a tabletop shape having a new shape and structure different from that of the existing desktop type computing device 300, and in this case, within the tabletop. An augmented reality environment can be provided through the included system.

In an embodiment of the present invention, the table top type computing device 400 may be used as a sub manager computing device.

Redundant descriptions of the components of the table-top type computing device 400 will be replaced with descriptions of the components of the desktop-type computing device 300. Hereinafter, the desktop-type computing device 300 It focuses on the differences between

4 is a conceptual diagram of experiencing augmented reality communication through a table top type computing device 400 according to an embodiment of the present invention, and FIG. 8 is a diagram of a table top type computing device 400 according to an embodiment of the present invention. This is the internal block diagram.

The tabletop type computing device 400 allows a remote manager to easily check an image received from a field worker's computing device 1000 and to provide work guidance for an object displayed in the image based on virtual content. It may be a device that provides an interface system for easy input.

That is, the tabletop type computing device 400 displays augmented/mixed reality that displays virtual content generated based on a real object of an image obtained from a current location on an image acquired from another user's computing device 100. It may be a system that generates and provides images.

Referring to FIGS. 4 and 8 , an exemplary table top type computing device 400 includes a memory 410, a processor assembly 420, a communication module 430, an interface module 440, an input system 450, It may include a sensor system 460 and a display system 470 .

These components may be configured to be included within the housing of computing device 400 . Redundant contents in the description of the components of the table top type computing device 400 will be replaced with descriptions of the above-described components, and hereinafter, differences will be mainly described.

Referring to FIG. 4 , the example table top type computing device 400 outputs a captured image captured by another computing device 100 through a display system 470 and virtual content related to the captured image to an augmented reality environment. can be provided to the user.

In addition, the table top type computing device 400 provides an input/output interface that receives a touch input such as a user's pointing or dragging on a captured image through a touch sensor 473 on the display 471 of the display system 470. can do.

In addition, the table top type computing device 400 receives a gesture input of the user's hands LH and RH through the sensor system 460 (eg, the image sensor 461) disposed on the display system 470. and receive motion input according to a guide tool (GT). The gesture input and the motion input of the guide tool may also be input to the display 471 corresponding to the captured image, and the computing device 300 may match the real object image displayed in the captured image to detect the user input.

That is, the augmented reality application 411 of the tabletop type computing device 400 according to the embodiment is a series of steps for realizing a service that visualizes and provides virtual content based on user input obtained through the sensor system 460. process can be executed. At this time, since the user input is performed corresponding to the captured image displayed on the display, the user can more intuitively input the user input to the captured image by providing an input/output interface together.

- Augmented reality communication method between multiple users

Hereinafter, an augmented reality communication method between multiple users according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

In the following embodiments, the main manager computing device (hereinafter referred to as “main device”) is described based on the desktop type computing device 300, and the sub manager computing device (hereinafter referred to as “sub device”) is described as a table top type computing device. It will be described based on the device 400.

In addition, the field user computing device 1000 (hereinafter referred to as “field device”) is described based on the mobile type computing device 200, but is not limited thereto.

9 is a flowchart illustrating an augmented reality communication method between multiple users according to an embodiment of the present invention.

Referring to FIG. 9 , in an embodiment of the present invention, the main device may execute an augmented reality application 311 for performing augmented reality communication based on augmented reality.

The augmented reality application 311 of the executed main device may perform a communication connection with a plurality of field devices 1000 . (S101)

In detail, as an embodiment, the augmented reality application 311 may provide a plurality of field devices 1000 and an interlocking interface for communicating with each other.

And the augmented reality application 311 may receive a request signal requesting augmented reality communication from the field device 1000 based on the provided interworking interface, and conversely, an acceptance signal for the request signal received from the field device 1000 can be created and sent.

At this time, the augmented reality application 311, when the acceptance signal for the request signal is generated and transmitted to the field device 1000, it may perform a communication connection with the field device (1000).

In another embodiment, the augmented reality application 311 may provide a field user search interface capable of searching for at least one field user based on the interworking interface, and perform a communication connection based on the provided field user search interface. can do.

10 is a diagram for explaining a method of performing a communication connection with a plurality of field user computing devices 1000 according to an embodiment of the present invention.

In detail, referring to FIG. 10 , the augmented reality application 311 may search field user information through a field user search interface.

Here, on-site user information refers to the name of the on-site user, area of expertise, work location, activation (information on whether or not communication connection is possible) and/or communication connection status (communication connection with several managers is provided to provide a guide). It may be information including information on whether or not the information is being received) and the like.

In an embodiment, the augmented reality application 311 may perform a search by inputting a user name or user identification code (eg, a management number or phone number assigned to the user) into a search window of the field user search interface.

In addition, the augmented reality application 311 may transmit a communication request signal to the field device 1000 of the field user by selecting at least one of the searched for at least one field user information.

And when the augmented reality application 311 receives an acceptance signal from the field device 1000 that transmitted the request signal, it may perform a communication connection with the field device 1000.

At this time, the augmented reality application 311 may perform a communication connection with at least one field user device, and may simultaneously perform communication with the connected field user devices in parallel.

As such, in an embodiment of the present invention, the augmented reality application 311 performs remote augmented reality communication (Remote AR communication) between a plurality of users in parallel and shares them in real time, so that at least one remote manager is at least one or more users. It is possible to perform data communication simultaneously with field users.

In another embodiment, the augmented reality application 311 may perform a communication connection based on a 3D site map, which is a map formed by 3D modeling of the physical space of the site.

11 is an example of a 3D site map according to an embodiment of the present invention.

In detail, referring to FIG. 11, the augmented reality application 311 may perform 3D modeling by scanning the physical space of the work site using a 3D scanner and camera, and through this, a 3D site map may be generated. can

That is, in the embodiment, the 3D site map may be a map created by 3D modeling the physical space of the work site.

At this time, the augmented reality application 311 may update the generated three-dimensional site map based on the captured images continuously obtained from the field device 1000 .

For example, the augmented reality application 311 may continuously match an image taken by the field device 1000 to a 3D site map using SLAM (Simultaneous localization and mapping) technology, through which the 3D site Maps can be updated.

In addition, the augmented reality application 311 may output the generated 3D site map through a display.

At this time, the augmented reality application 311, based on the captured image and / or location information obtained from the field device 1000, a three-dimensional site map matched to the actual location of the field user matched to the corresponding field device 1000 On-site user information of the on-site user may be displayed as a virtual object (eg, a virtual icon representing the on-site user) at a virtual location on the screen.

Here, the location information obtained from the field device 1000 may be obtained through a Global Positioning System (GPS), a beacon, and/or a location sensor.

In addition, the augmented reality application 311, when detecting a selection input for the displayed virtual object, confirms the field user information of the field user matched to the corresponding virtual object in more detail, or a button for requesting a communication connection to the field user. (button) can be provided.

As such, in an embodiment of the present invention, the augmented reality application 311 enables a communication connection with the field device 1000 based on a 3D site map, so that a manager can perform a communication connection in a specific situation (e.g., in a specific location). It is possible to quickly identify the most suitable on-site user for a situation (such as a situation requiring work) and request a communication connection.

In addition, the augmented reality application 311 performing a communication connection with a plurality of field devices 1000 may receive a basic photographed image, which is an image taken from a plurality of field devices 1000, and display it in real time. (S103)

In detail, the augmented reality application 311 may receive a plurality of basic captured images from a plurality of field devices 1000, and divide the received captured images into predetermined areas (first area, second area) of the display screen. , ... can be respectively displayed as the nth (n = 1, 2, 3, ...) area).

In more detail, the augmented reality application 311 may provide an augmented reality communication interface for performing augmented reality communication with at least one field device 1000 .

In detail, the augmented reality application 311 may divide the display screen into predetermined areas.

In addition, the augmented reality application 311 may output each of the basic captured images received from the plurality of field devices 1000 to each of the divided predetermined areas.

That is, the augmented reality application 311 divides the display screen into predetermined areas, allocates and displays a plurality of captured images to each area, and effectively collectively checks a plurality of captured images.

In addition, the augmented reality application 311 according to the embodiment may perform augmented reality communication with at least one field device 1000 . (S105)

12 is an example of an interface providing augmented reality communication according to an embodiment of the present invention.

In detail, referring to FIG. 12 , the augmented reality application 311 may provide an augmented reality communication interface capable of performing augmented reality communication with at least one field device 1000 . The user may select at least one field device 1000 from among a plurality of field devices 1000 through which a communication connection is performed through the provided interface.

For example, the augmented reality application 311 may detect a user input for selecting at least one or more of a plurality of basic photographed images respectively displayed on a predetermined area through an augmented reality communication interface. In addition, the augmented reality application 311 may execute augmented reality communication with the field device 1000 matching at least one captured image selected based on the detected user input.

At this time, the augmented reality application 311, when a plurality of captured images are selected, augmented reality communication (eg, voice data exchange, virtual content) in parallel with the field devices 1000 matching each of the selected plurality of captured images. sharing, etc.).

Here, the augmented reality application 311, in the case of performing augmented reality communication based on virtual content, may select one captured image, that is, a single site device 1000.

In detail, the augmented reality application 311 may select one basic photographed image through an augmented reality communication interface when exchanging communication data including virtual content.

In addition, the augmented reality application 311 may provide an interface for generating virtual content for a selected captured image.

In addition, the augmented reality application 311 may generate virtual content for the corresponding captured image according to the input of the main manager based on the provided interface.

In addition, the augmented reality application 311 may generate communication data including the generated virtual content, and transmits and displays the communication data including the generated virtual content to the field device 1000 that matches the corresponding captured image. can do.

As such, the augmented reality application 311 may perform augmented reality communication in an optimized manner according to the type of communication data exchanged, thereby improving the quality of augmented reality communication.

In addition, the augmented reality application 311 according to the embodiment generates virtual content matching the physical space of the site outside the area included in the captured images obtained from the plurality of field devices 1000 based on the three-dimensional site map can do.

In general, in a work site, unexpected abnormal situations may frequently occur anywhere on the site. When such an abnormal situation occurs, a manager may want to transmit guidance for at least one on-site user to move to a location where the abnormal situation occurs and perform a task according to circumstances.

13 is an example of providing an on-site navigation interface according to an embodiment of the present invention.

The augmented reality application 311 according to an exemplary embodiment may first output a 3D site map through an augmented reality communication interface.

In addition, the augmented reality application 311 may designate an abnormal location where a problem occurs on the output 3D site map, select at least one field user, and provide an on-site navigation interface capable of guiding the user to the abnormal location. .

In detail, the augmented reality application 311 may determine a pointing location indicating an abnormal location based on a user's (main manager's) input for a specific point on the 3D site map. For example, the augmented reality application 311 converts the virtual coordinates of a specific point selected by the user in the 3D site map into real coordinates corresponding to the physical space of the actual site, thereby converting the user input of the virtual 3D site map. It is possible to provide an interface that designates the actual location of the on-site physical space according to That is, the user can designate the location of the actual physical space to which the field user will move through a pointing input on the virtual coordinates of the 3D site map.

In addition, the augmented reality application 311 provides route guidance to the pointing location based on a user input for selecting at least one field user from among a plurality of field users displayed through virtual objects on a 3D site map. It is possible to determine at least one field device 1000 to be performed.

That is, the augmented reality application 311 may obtain a user input for selecting at least one or more of a plurality of on-site users displayed on the 3D site map, and a site matched to the selected on-site user based on the obtained user input. The device 1000 may be detected.

In addition, the augmented reality application 311 may search for an optimal path for each user to reach the determined pointing position.

In detail, the augmented reality application 311 may design a virtual path moving from the first virtual coordinates matched to the actual location of the on-site user in the virtual space of the 3D site map to the second virtual coordinates of the pointing location.

For example, the augmented reality application 311 uses a depth first search (DFS) algorithm, a Dijkstra algorithm, and/or an A* (A star) algorithm to reach a pointing position. An optimal virtual path for each user can be obtained.

Also, the augmented reality application 311 may generate virtual content that guides a movement path from each user's location to a pointing location, based on the searched virtual path for each user.

As an embodiment, the augmented reality application 311 converts a virtual path for each user found on an image captured through the field device 1000 of each user into a virtual object (eg, an arrow indicating a direction to a pointing position) Augmented reality navigation virtual content to be displayed based on can be generated.

For example, the augmented reality application 311 may generate a first virtual path, which is an optimal path to reach a pointing position from the virtual position of the first on-site user, based on the 3D site map.

Also, the augmented reality application 311 may generate first augmented reality navigation virtual content for guiding the field user to move along the first virtual path.

In this way, the augmented reality application 311, based on the 3D site map, the nth (n = 2, 3, ...) optimal path that can reach the pointing position from each virtual location of the field user nth An nth augmented reality navigation virtual content along the virtual route may be generated.

In addition, the augmented reality application 311 may transmit and display each generated virtual content to the field device 1000 matched for each user.

In this way, the augmented reality application 311, through a one-time workflow based on the three-dimensional site map and the field navigation interface, by collectively providing different navigation contents for each of a plurality of field users, promptly responds when an abnormal situation occurs. can be performed, and guidance can be provided to a plurality of field users at once without performing communication with each field user.

In addition, the augmented reality application 311 according to an embodiment of the present invention may provide an input/output interface for providing work guidance to field users based on a 3D site map.

Here, the work guidance according to the embodiment may be virtual content that designates a location and/or equipment where a field user needs to work, and provides information to be worked.

In detail, the augmented reality application 311 may acquire work guidance information based on a user input based on a guidance input/output interface of a 3D site map.

In addition, the augmented reality application 311 may generate and output virtual content representing work guidance on a 3D site map based on the obtained work guidance information.

For example, the augmented reality application 311 selects the first virtual location or first virtual object of the 3D site map, and the user describes a task to be performed for the selected first virtual location or first virtual object. input can be obtained. Also, the augmented reality application 311 may generate and display virtual content providing work guidance for the first virtual location or the first virtual object on the 3D site map based on the obtained user input.

At this time, the augmented reality application 311, based on the field sensor system, can match the virtual content generated for the virtual location or virtual object of the three-dimensional site map to the actual location or real object in the physical space of the field. .

Here, the on-site sensor system may include various sensors that sense and measure various information (eg, environment, mechanical facilities, and on-site user information) on the physical space of the site.

For example, the field sensor system may include at least one sensor of a temperature sensor, a humidity sensor, a pressure sensor, a flow sensor, a magnetic sensor, a light sensor, an acoustic sensor, an image sensor, a current sensor, and a gas sensor disposed in the field. can

In an embodiment, such a field sensor system may include a positioning sensor system, a camera system, a device sensor system, and a gateway.

In addition, the augmented reality application 311, based on the field sensor system, when a field user approaches a real location or a real object matched to virtual content that provides work guidance, to the field device 1000 of the field user The virtual content may be provided.

In detail, the augmented reality application 311, when the field device 1000 is detected in a predetermined area around the real location or real object (eg, within a 3 m area around the real location or real object), the corresponding field device In (1000), virtual content matched to the real location or real object can be transmitted.

At this time, the augmented reality application of the field device 1000 receiving the virtual content that provides work guidance for the actual location or real object may display work guidance through the virtual object based on the received virtual content.

For example, the augmented reality application of the field device 1000 may output a virtual object displaying a work location designated by the main manager as a dot, line, and/or arrow based on the received work guidance virtual content. .

In addition, the augmented reality application 311, when a plurality of field devices 1000 are detected in a predetermined area around the real location or real object, a plurality of field devices 1000, respectively, to the real location or real object Matched virtual content can be transmitted in batches.

That is, the augmented reality application 311 may allow a plurality of field users to perform a designation task for a real location or a real object matched to the work guidance virtual content.

At this time, the augmented reality application of the plurality of field devices 1000 receiving the virtual content that provides work guidance for the actual location or real object, in the case of collaborative work in which multiple users (field users) participate, virtual content The appearance of other users (other field users) in the displayed augmented image can be displayed in the form of a 3D human model (avatar).

For example, when the first and second field devices receive work guidance virtual content, the second field user in the augmented image of the first field device may be displayed as a 3D human model, and in the augmented image of the second field device The first on-site user of may also be displayed as a 3D human model.

In this way, the augmented reality application 311, based on the three-dimensional site map generates a work guidance that matches the actual location or real object of the work site, and the generated work guidance located close to the actual location or actual object By providing at least one field device 1000, it is possible to easily and clearly convey work instructions for a 3D physical space, and induce rapid response from nearby field users.

In addition, the augmented reality application 311, based on the field sensor system, when detecting the field user's work history for the actual location or real object (eg, mechanical equipment) of the field, the detected work history information (eg, field Work history information including text, etc.) of the user's work process can be stored by matching with the virtual location or virtual object of the 3D site map corresponding to the real location or real object.

For example, the augmented reality application 311 may receive work history information input by a first field user from a first field device. In addition, the augmented reality application 311 receiving the work history information may store the work history information by matching the virtual location or virtual object of the 3D site map corresponding to the location of the field user or the actual object selected by the field user. .

Thereafter, the augmented reality application 311 may execute a work history check and management function according to the input of the main manager, and work history information matched to a virtual location or virtual object of a 3D site map or work matched to a field user. It can be confirmed by displaying the history information.

As such, the augmented reality application 311 may provide an interface capable of storing and managing work history information for field users and real locations or real objects based on the 3D site map.

In addition, in an embodiment of the present invention, the augmented reality application 311 of the main device may provide a sub manager-site user matching interface. (S107)

Here, the sub-manager-site user matching interface means that a main manager designates any one of the sub-devices of at least one or more sub-administrators and the field device 1000 of at least one or more field users and matches them so that mutual communication connection is performed. It can be an interface.

In detail, the augmented reality application 311 of the main device may display sub-manager information for at least one sub-manager having a communication connection with the corresponding main device through an augmented reality communication interface.

Here, sub-administrator information includes the sub-administrator's name, area of expertise, activation status (information on whether or not communication connection is possible), and/or communication connection status (how many on-site users are being communicated with and the guide is being provided). It may be information including information about) and the like.

For example, the augmented reality application 311 may display sub-manager information as a list on one area of an augmented reality communication interface.

In addition, the augmented reality application 311 assigns at least one field user to the sub manager displayed through the augmented reality communication interface, so that the sub device of the sub manager and the field device 1000 of the assigned field user are connected in communication. A setting sub-administrator-on-site user matching interface can be provided.

For example, the augmented reality application 311 may detect a user input for selecting one or more of at least one basic photographed image displayed through an augmented reality communication interface.

In addition, the augmented reality application 311 may detect the field device 1000 matching the selected captured image based on the detected user input.

In addition, the augmented reality application 311 may detect a user input for selecting one of at least one sub-manager included in the sub-manager list output through the augmented reality communication interface.

Also, the augmented reality application 311 may detect a sub device that matches the selected sub manager based on the sensed user input.

In addition, the augmented reality application 311 may match the detected at least one field device 1000 and the sub-device so that a communication connection is performed.

That is, the augmented reality application 311 enables an augmented reality-based communication connection between at least one or more sub-administrators and at least one or more on-site users, so that 1: n (n = 1,2 ,3, …), 1:n:m (m = 1,2,3, …) or n:m:x (x = 1,2,3, …), etc. Through this, it is possible to efficiently distribute and systematically manage the management of multiple field operations.

In addition, the augmented reality application 311 may terminate the communication connection with the field user matched with a specific sub manager when the sub manager and the field user are matched based on the sub manager-field user matching interface.

Alternatively, according to the embodiment, the augmented reality application 311 may output communication data between the sub-manager and the field user when performing augmented reality communication by matching the sub-manager and the field user. (S109)

14 is an example of a conceptual diagram illustrating how augmented reality communication between multiple users is performed according to an embodiment of the present invention.

In detail, referring to FIG. 14, first, the augmented reality application 311 (hereinafter referred to as the sub augmented reality application 311) of the sub device 122 connected to the field device 1000 through communication is taken from the connected field device 1000. It is possible to generate virtual content for the basic photographed image.

At this time, the sub-device 122 that generates virtual content is implemented as a table-top type computing device 400 and may generate and display virtual content for a captured image obtained from the field device 1000 .

In addition, the sub-augmented reality application 311 may transmit virtual content generated with respect to the basic photographed image obtained from the field device 1000 to the corresponding field device 1000 .

At this time, in the embodiment of the present invention, when the virtual content generated from the sub-augmented reality application 311 is transmitted to the field device 1000, the corresponding field device as well as the image of the field device 1000 receiving the virtual content Communication data including virtual content generated from the sub-augmented reality application 311 is overlaid and displayed as an augmented image even in the basic captured image of the main device 121 receiving and displaying the basic captured image of 1000. It can be.

That is, in the embodiment, in the augmented reality application 311 of the main device 121, the sub device 122 generates virtual content for a basic photographed image through augmented reality communication with the field device 1000 to generate the field device 1000. ), the transmitted virtual content can be obtained from the field device 1000 or the sub device 122, and based on this, the basic captured image of the corresponding field device 1000 being output from the main device 121 Also, communication data including virtual content generated from the sub device 122 may be overlaid and displayed as an augmented image.

In addition, when the augmented reality application 311 of the main device 121 detects a user input for an augmented image displayed by overlaying virtual content generated from the sub device 122, other communication data for the corresponding augmented image ( For example, voice data transmitted from the sub device 122 to the field device 1000, etc.) may be provided.

As such, in the embodiment of the present invention, the augmented reality application 311 more conveniently and efficiently controls augmented reality communication between sub-administrators and field users performed multiple times by providing communication data between sub-administrators and field users. can do

As described above, the augmented reality communication method between multiple users according to an embodiment of the present invention performs remote augmented reality communication between multiple users in parallel and shares them in real time, so that at least one remote manager can use at least one remote AR communication. There is an effect of performing data communication with the above field users at the same time.

In addition, in the augmented reality communication method between multiple users according to an embodiment of the present invention, at least one or more remote managers perform real-time parallel data communication with at least one or more field users, thereby providing guidance for a plurality of field users collectively. There is an effect that can be processed and provided.

In addition, the augmented reality communication method between multiple users according to an embodiment of the present invention provides a table top system that effectively provides work guidance for the field remotely based on virtual content, thereby providing a realistic screen through virtual content. Easy and clear work guidance can be provided to remote field users in real time, and through this, the operator can perform the work more intuitively.

In addition, the embodiments according to the present invention described above may be implemented in the form of program instructions that can be executed through various computer components and recorded on a computer readable recording medium. The computer readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present invention, or may be known and usable to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. A hardware device may be modified with one or more software modules to perform processing according to the present invention and vice versa.

Specific implementations described in the present invention are examples and do not limit the scope of the present invention in any way. For brevity of the specification, description of conventional electronic components, control systems, software, and other functional aspects of the systems may be omitted. In addition, the connection of lines or connecting members between the components shown in the drawings are examples of functional connections and / or physical or circuit connections, which can be replaced in actual devices or additional various functional connections, physical connection, or circuit connections. In addition, if there is no specific reference such as “essential” or “important”, it may not be a component necessarily required for the application of the present invention.

In addition, the detailed description of the present invention described has been described with reference to preferred embodiments of the present invention, but those skilled in the art or those having ordinary knowledge in the art will find the spirit of the present invention described in the claims to be described later. And it will be understood that the present invention can be variously modified and changed without departing from the technical scope. Therefore, the technical scope of the present invention is not limited to the contents described in the detailed description of the specification, but should be defined by the claims.

Claims (7)

As an augmented reality communication method between multiple users performed in a main manager computing device,
establishing a communication connection with a plurality of on-site user computing devices;
obtaining and displaying captured images from the plurality of on-site user computing devices;
providing a sub-manager-site user matching interface matching a sub-manager computing device and the site user computing device; and
Performing augmented reality communication between a first sub-manager computing device determined according to the matching interface and a first on-site user computing device;
The step in which the augmented reality communication is performed,
generating virtual content for a basic photographed image captured by the first on-site user computing device in the first sub-manager computing device;
displaying an augmented image in which the virtual content is overlaid on the basic photographed image in the first on-site user computing device;
Displaying an augmented image in which the virtual content is overlaid on the basic captured image in the main manager computing device;
Sensing the main manager's input for the augmented image in the main manager's computing device; generating other communication data for the augmented image according to the main manager's input; Further comprising providing the first on-site user computing device and the first sub-manager computing device to at least one device.
Augmented reality communication method between multiple users. According to claim 1,
The step of performing the communication connection,
Performing the communication connection with a selected on-site user computing device based on a 3-dimensional site map, which is a map formed by 3-dimensional modeling of the physical space of the work site
Augmented reality communication method between multiple users. According to claim 2,
Further comprising providing an input/output interface for generating virtual content matching one virtual location of the 3D site map and transmitting it as communication data.
Augmented reality communication method between multiple users. According to claim 3,
Transmitting the virtual content to the detected field user computing device when the field user computing device is detected in a predetermined area of the actual location of the work site that matches one virtual location of the 3D site map. more inclusive
Augmented reality communication method between multiple users. According to claim 2,
displaying a virtual object representing the identification information of the field user on the virtual location of the 3D site map matched to the actual location of the field user located at the work site;
Further comprising providing an interface for selecting the virtual object on the 3D site map and selecting the on-site user computing device to which communication is to be made.
Augmented reality communication method between multiple users. delete delete

Images