WO2022037758A1

WO2022037758A1 - Remote collaboration using augmented and virtual reality

Info

Publication number: WO2022037758A1
Application number: PCT/EP2020/073073
Authority: WO
Inventors: Till Heinzerling; Daniel Labisch; Asa Macwilliams; Nischita SUDHARSAN
Original assignee: Siemens Aktiengesellschaft
Priority date: 2020-08-18
Filing date: 2020-08-18
Publication date: 2022-02-24

Abstract

What is proposed is a communication system (13) for transmitting first information data (7) from a first user (3) at a first location (6) within a technical plant (1) to a second user (10) at a second location (11), wherein the first information data (7) at least comprise first location data, and in particular additionally comprise process values, signal values, video images and/or voice signals, where-in the second user (10) is able to navigate to the first location (6) within a virtual reality environment of the technical plant (1), and for transmitting second information data (15) from the second user (10) at the second location (11) to the first user (1) at the first location (6) within the technical plant (1) wherein the second information data (15) comprise augmented reality information.

Description

Remote collaboration using augmented and virtual reality

The present disclosure is directed to a communication system and a method for transmitting information.

A user can wish to view, create or edit information in augmented reality (AR) , which is relevant to a large piece of physical technical infrastructure, to get support from an expert located in a separate place. Specific examples include :

- Viewing step-by-step workflow information for repairing process automation equipment, using a head-worn augmented reality display, in a chemical or a pharmaceutical plant.

- Performing an inspection walkthrough in a technical plant, creating annotations to document potential problems using an augmented reality app on a smartphone.

- Navigation to an asset. The user is guided to an asset using an augmented reality app on a smartphone or AR glasses.

The user usually needs both hands for his actual work but lacks detailed knowledge regarding the tasks. An expert with detailed knowledge about the plant, the component and/or the tasks to perform is available, but he's not available at the plant .

By phone call the user could describe the situation and the remote expert only could describe the steps to perform. As verbal descriptions might be ambiguous, mistakes are very likely in the case of complex tasks.

The user using a smartphone or tablet enables also visual support but using the device only one remaining hand is available for the actual work, which is not sufficient for every task. Sometimes both hands are used to hold the device in its place.

Using a camera mounted to the user' s head or body keeps his hands free and gives the expert the opportunity to directly see the local situation. This already reduces mistakes in the communication from worker to expert but shows no effect in the opposite direction.

The user using smart glasses creates the possibility to see information from the expert while keeping hands free.

Thereby, the information is visible but is still not directly linked with the local surrounding.

In WO 2016/206874 Al there is disclosed an interaction system for supporting an operator with operation by means of an expert wherein the system provides a remotely operated hint by the expert.

WO 00/52537 Al discloses a system and a method for situation- related interaction support with the aid of augmented reality technologies .

An objective of the present invention is to improve a remote guidance of a user of a technical plant by an expert.

The problem is solved by a communication system according to claim 1. Furthermore, the problem is solved by a method according to claim 9. Advantageous aspects of the invention are the subject of the dependent claims.

According to the invention, the problem is solved by a communication system for transmitting first information data from a first user at a first location within a technical plant to a second user at a second location, wherein the first information data at least comprise first location data, and in particular additionally comprise process values, signal values, video images and/or voice signals, wherein the second user is able to navigate to the first location within a virtual reality environment , and for transmitting second information data from the second user at the second location to the first user at the first location within the technical plant wherein the second information data comprise augmented reality information .

A "technical plant" can be a plant in process industries like a chemical or a pharmaceutical plant. This also includes any plant from the production industry, plants in which e.g. cars or goods of all kinds are produced. Technical plants that are suitable for carrying out the method according to the invention can also come from the field of energy generation. Wind turbines, solar plants or power plants for power generation are also covered by the term of technical plant. Also other applications might be possible like building sites, trains or the like.

"Augmented reality information" are designed for being shown by smartphones, tablets, glasses, projectors or the like. They are designed for providing additional virtual information to "classical" sensorial information like vision or audio.

A "virtual reality environment" is a digital twin of the technical plant. With other words, it is a three-dimensional model of the technical plant, which shows all or at least some (e.g. important) parts and components of the technical plant.

According to the invention, a first user of the techni- cal plant transmits information at least about the first location he is at one moment. Using this first information and a virtual reality environment of the technical plant the second user can be virtually next to the first user - without the need of being within the technical plant. Of course, the second user can also be within the technical plant, e.g. in a separate room.

With the help of the described communication system the second user, e.g. a technical expert, can support the first user, who can be a worker in the technical plant. The collaboration of an on-side augmented reality user and a remote virtual reality environment user can be useful to create a detailed three-dimensional map of the technical plant (the virtual reality environment user can guide the augmented reality user to locations that have not been scanned yet) .

With the help of the communication system according to the invention, local (first) users / workers within the technical plant are enabled to perform more complex tasks. More than this, the second user is enabled to support the first user more focused and efficiently since he can act like he was actually at the first location within the technical plant.

Preferably, the virtual reality environment the second user can navigate through is provided by means of a head-mounted display. The second user also can use hand controllers to get a more realistic haptic feeling during supporting the first user.

According to a preferred embodiment of the invention, the communication systems comprises detection means with a sensor system, in particular an image recording device, for generating the first information data, and visualization means for visualizing the second information data to the first user, in particular by means of an augmented reality glass. The first user can wear a transparent augmented reality glass allowing to show three- dimensional information in his view. Aside from an augmented reality glass, it is also possible to use a smartphone or a tablet for providing the augmented reality to the first user. For the first user, this is mainly useful to get an overview of the environment, and to send data to the second user. It is less useful while performing actual work, as the hands are not free. A tablet/smartphone may also be useful to other people within the technical plant, such as a trainee, or a third user, who is documenting the current task or learning about it, while the actual first user uses an augmented reality glass.

A live stream of a camera of the detection means can be shown to the second user. An exact location of the camera and where it is looking at could be shown to the second user in his virtual reality environment. This can eliminate errors caused by parts looking identically.

The first user can also take photos of interesting details. These can be transferred as a part of the first information data to the second user and be shown exactly in the first user' s position and view, so that the second user exactly sees his view.

The detection means can be designed for eye tracking, hand tracking and/or gesture recognition of the first user. This information can also be transmitted to the second user and be shown in the virtual reality environment. The first user could support his descriptions highlighting parts and details virtually. In addition, the first user can place virtual objects and/or details at relevant parts/components of the technical plant.

Preferably, the detection means is able to generate three-dimensional model of at least one part of the technical plant in an area around the first location, wherein the three-dimensional model is transmitted to the second user at the second location.

According to an advantageous embodiment of the invent- tion, the second information data comprise a live stream of a view the second user has within the virtual environment. Thereby, the second user can demonstrate some procedures or the like. The second information can also comprise instructions, documents, videos, plans or three-dimensional models of parts/components of the technical plant.

The second user can virtually highlight parts/components what supports his explanations. He can e.g. use a virtual laser pointer or a flashlight to point on plant details or highlight parts. The laser beam or at least the information about where it hits the plant is transferred from the virtual reality environment of the second user to the augmented reality of the first user.

Preferably, the first information data comprise a digital avatar of the first user, and the second information data comprise a digital avatar of the second user. This feature can improve the collaboration between the first user and the second user from a psychological perspective. Both users virtually see each other and feel like really working together in the same technical plant at the same location.

The invention is not limited to one first user and one second user. In contrast, there can be a plurality of first users and/or second users which are using the communication system as described above. Also, a plurality of first users without any second users and vice versa is possible. The problem as described above is also solved by a method for transmitting first information data from a first user at a first location within a technical plant to a second user at a second location, wherein the first information data at least comprise first location data, and in particular additionally comprise process values, signal values, video images and/or voice signals, wherein the second user is able to navigate to the first location within a virtual reality environment, and for transmitting second information data from the second user at the second location to the first user at the first location within the technical plant wherein the second information data comprise augmented reality information .

Features of examples of the present disclosure will become apparent by reference to the following descript- tion of an exemplary embodiment of the invention.

The figure shows a technical plant 1 and a remote service point 2. In the technical plant a first user 3 (a "worker") is initiating maintenance tasks with a component 4 of the technical plant 1 at a first location 5. In the course of the maintenance tasks the first user 3 wears an augmented reality glass 5 which is designed for showing the first user 3 virtual objects in addition to the normal surrounding at the first location 6. The augmented reality glass 5 is part of a communication system 13 which is described in the following.

First information data 7 comprising data about the first location ("first location data") as well as video images and voice signals are transmitted to a server 8 of the communication system 13 which is cloud-based, i.e. the server 8 is located outside of the technical plant 1 and can be reached via the internet. A local solution is also possible. In the server 8 there is stored a three- dimensional model of the technical plant 1. The server 8 combines the first information data 7 with the three- dimensional model of the technical plant 1 and transmits according data 9 to a second user 10 which is at a second location 11 at the remote service point 2. The second user 10 wears a head-mounted display 12 which is also part of the communication system 13. The headmounted display 12 can use the data get by the server 8 to generate a virtual reality environment the second user 10 can navigate through (virtually) . Information exchange is done in real time such that the collaboration feels natural for both users 3, 10.

Data 14 comprising an orientation / position of the second user 10 in the virtual reality environment are sent from the head-mounted device 12 to the server 8. Server 8 generates second information data 15 which are transmitted to the first user 3 at the first location 6 within the technical plant 1. The second information data 15 comprise augmented reality information generated by the server 8 based on the data 15 received by the second user 10. The second information data 15 can e.g. comprise a warning sign 16 which is shown to the first user 3 to prevent him from performing wrong tasks.

For every spatial information the relevant position and orientation of the first user 3 (respectively the augmented reality glass 5) is transferred to the server 8 in addition to the actual information. E.g. a photo is transferred together with the camera' s position and orientation. This spatial information needs very little storage and data rate capacity and are negligible compared to the other information.

The AR/VR (augmented reality / virtual reality) combination explained above can also be extended to include more AR or VR devices and would then have an environment which includes multiple devices interacting with each other and not limited to just one device in the AR or VR setting . An advantage of the described cloud solution is that even more than two users could collaborate directly and easily. So, every participant sees every information from each other in his plant, regardless of it being virtual or real. Thereby, also AR/AR and VR/VR collabo- ration are included.

While several examples have been described in detail, it is to be understood that the disclosed examples may be modified. Therefore, the foregoing description is to be considered non-limiting.

Claims

Patent claims

1. Communication system (13) for transmitting first information data (7) from a first user (3) at a first location (6) within a technical plant (1) to a second user (10) at a second location (11) , wherein the first information data (7) at least comprise first location data, and in particular additionally comprise process values, signal values, video images and/or voice signals, wherein the second user (10) is able to navigate to the first location (6) within a virtual reality environment of the technical plant (1) , and for transmitting second information data (15) from the second user (10) at the second location (11) to the first user (1) at the first location (6) within the technical plant (1) wherein the second information data (15) comprise augmented reality information.

2. Communication system (13) according to claim 1, wherein the virtual reality environment the second user (10) can navigate through is provided by means of a head-mounted display (12) .

3. Communication system (13) according to claim 1 or

2, which comprises detection means with a sensor system, in particular an image recording device, for generating the first information data (7) , and visualization means (5) for visualizing the second information data (15) to the first user (3) , in particular by means of an augmented reality glass (5) .

4. Communication system (13) according to claim 3, wherein the sensor system allows an eye tracking, hand tracking and/or a gesture recognition of the first user (3) .

5. Communication system (13) according to claim 3 or 4, wherein the detection means is able to generate a three-dimensional model of at least one part of the technical plant (1) in an area around the first location (6) , wherein the three-dimensional model is transmitted to the second user (10) at the second location (11) .

6. Communication system (13) according to one of the preceding claims, wherein the second information data (15) comprise a live stream of a view the second user (10) has within the virtual reality environment.

7. Communication system (13) according to one of the preceding claims, wherein the first information data (7) comprise a digital avatar of the first user (3) , and the second information data (15) comprise a digital avatar of the second user (10) .

8. Communication system (13) according to one of the preceding claims, wherein there is a plurality of first users (3) and/or second users (10) .

9. Method for transmitting first information data (7) from a first user (3) at a first location (6) within a technical plant (1) to a second user (10) at a second location (11) , wherein the first information data (7) at least comprise first location data, and in particular additionally comprise process values, signal values, video images and/or voice signals, wherein the second user (10) is able to navigate to the first location (6) within a virtual reality environment, and for transmitting second information data (15) from the second user (10) at the second location (11) to the first user (3) at the first location (6) within the technical plant (1) wherein the second information data (15) comprise augmented reality information.

10. Method according to claim 9, wherein the virtual reality environment the second user (10) can navigate through is provided by means of a head-mounted display (12) .

11. Method according to claim 9 or 10, wherein first information data (7) are generated by detection means with a sensor system, in particular an image recording device, for generating the first information data (7) , and wherein the second information data (15) are visualized to the first user (3) by visualization means (5) , in particular by means of an augmented reality glass ( 5 ) .

12. Method according to one of claims 9 through 11, wherein the sensor system allows an eye tracking, hand tracking and/or a gesture recognition of the first user (3) .

13. Method according to one of claims 9 through 12, wherein the sensor system is able to generate a three dimensional model of at least one part of the technical plant in an area around the first location (6) , wherein the three dimensional model is transmitted to the second user (10) at the second location (11) .

14. Method according to one of claims 9 through 13, wherein the second information data (15) comprise a live stream of a view the second user (10) has within the virtual environment.

15. Method according to one of claims 9 through 14, wherein the first information data (7) comprise a digital avatar of the first user (3) , and the second information data (15) comprise a digital avatar of the second user (10) .