DE102004061841B4 - Markerless tracking system for augmented reality applications - Google Patents

Abstract

Method for initializing a system, wherein objects of a real environment and virtual information can be superimposed with the system,
wherein a picture is taken with a camera which includes an initialization object located in the real environment:
characterized,
the captured image and a data model of the initialization object are shared with a user;
that the recorded initialization object and the data model of the initialization object are superimposed and displayed superimposed;
that a search area is set, which contains the recorded initialization object and the data model of the initialization object, as superimposed;
that the search area is searched for at least one feature of the initialization object; and
that the position, orientation and / or size of the at least one found feature is stored.

Description

State of the art

Out the prior art are so-called augmented reality systems known. These allow the overlay of computer generated, virtual information with visual impressions of the real environment. To this end, the visual impressions of the real world, preferably with semipermeable data glasses worn on the head, with virtual Information mixed. The insertion of the virtual information or objects can be context-dependent, d. H. adapted and derived be executed by the respective considered real environment. When Information can in principle Any type of data such as texts, illustrations etc. can be used. The real environment can be with one, for example, on the head of the User worn, camera captured.

Documented applications of technology envisage use in the production, service and development of complex products. Also, the use of technology from the production of aircraft is known. After the publication DE 198 32 974 A1 and DE 101 28 015 A1 is the use of augmented reality technologies in production environments known.

In addition are known from the prior art optical position detection systems, the position and or orientation of objects in a survey room determine. These so-called tracking systems allow, for example, the Capture of up to six degrees of freedom of an object. For use come systems with different physical principles of action. Are common so-called optical tracking systems by different methods the computer-aided Image processing the location of located in the survey room Objects and / or the position of the camera on the detection of the objects determine. Because the free search for objects using currently available computing power is not possible in sufficient speed become simplistic Assumptions made.

A possibility to simplify the complexity, is the equipment of the tracking room with high-contrast, easy Marks to be recognized, hereinafter referred to as markers. However, this approach has the disadvantage that the tracking room must be equipped and the tracking only in the equipped Range possible is. Optical tracking methods using this method are becoming common referred to as marker-based tracking method.

A different possibility to simplify the complexity is the use of models, for example statistical movement models and / or additional Sensors, for example, acceleration sensors, which the search space restrict the image processing algorithms. Although this approach the complexity simplified, without the need for the equipment of the room, he can not help with the initialization. initialization in this context means the determination of the internal state of the system, making the tracking process successful and in sufficient done a short time can be. The system must first be set to a valid initial state. This initial state includes the approximate position and orientation the reality to the camera and advantageously information about the current movement the camera.

• (Behringer, R.; Park, J.; Sundareswaran, V.: Model-Based Visual Tracking for Outdoor Augmented Reality Applications. In: International Symposium an Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, S. 277
• Genc, Y.; Riedel, S.; Souvannavong, F.; Akinlar, C.; Navab, N.: Marker-less Tracking for AR: A Learning-Based Approach, In: International Symposium an Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, S. 295
• Kar Wee Chia, Adrian David Cheok, Simon J. D. Prince: Online 6 DOF Augmented Reality Registration from Natural Features. In: International Symposium an Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, S. 277).

Systems which do not rely on special markers in space are known from the following documents and are referred to below as markerless tracking systems:

• (Behringer, R .; Park, J .; Sundareswaran, V .: Model-Based Visual Tracking for Outdoor Augmented Reality Applications): International Symposium on Mixed and Augmented Reality (ISMAR'02) Darmstadt 2002, p
• Genc, Y .; Riedel, S .; Souvannavong, F .; Akinlar, C .; Navab, N .: Marker-less Tracking for AR: A Learning-Based Approach, In: International Symposium on Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, p. 295
• KarDe Chia, Adrian David Cheok, Simon JD Prince: Online 6 DOF Augmented Reality Registration from Natural Features. In: International Symposium on Mixed and Augmented Reality (ISMAR'02). Darmstadt 2002, p. 277).

input information The system is a two-dimensional array of red, green, and blue intensities in the camera image. additionally can Acceleration data of additional Sensors are read. The output data of the system is the position and orientation of one or more objects relative to the camera. The elements of reality can then, for example, a coordinate system for augmented reality information span. Position and orientation can be exemplified by a transformation matrix will be described.

Various methods are known for determining the transformation matrix. Some methods recognize features, such as edges or corners, of real objects and compare them with previously stored digital models of reality (eg, CAD geometries, eg, surface models with texture information). Other methods start with a very simple initialization initialization model After the initialization step, add additional information extracted from the camera image to the model of reality.

• – Einfache oder mehrfache Annahme einer wahrscheinlichen Position und Orientierung vorteilhafterweise mittels eines Bewegungsmodells
• – Einfache oder mehrfache Projektion des internen Modells in das Videobild
• – Abgleich der Projektion mit dem Bild der Realität zum Beispiel über das Messen von Punktabständen
• – Zuordnung von Punkten des Modells zu deren Referenz im Kamerabild
• – Bestimmung der Transformationsmatrix zur Kamera und vorteilhafterweise Bestimmung einer statistischen Aussage zur Qualität der Transformationsmatrix

The determination of the transformation matrix takes place after the initialization in the following basic steps:

• Simple or multiple assumption of a probable position and orientation advantageously by means of a movement model
• - Single or multiple projection of the internal model into the video image
• - Matching the projection with the image of reality, for example, by measuring point distances
• - Assignment of points of the model to their reference in the camera image
• Determination of the transformation matrix to the camera and advantageously determination of a statistical statement on the quality of the transformation matrix

In EP 1 507 235 A1 An Augmented Reality System is described, which superimposes a data model of the real environment using a marker. In this case, the rendered virtual object is not a data model of the marker, but any virtual object that is simply placed in the image with the help of the marker. In particular, a reference value of the marker is determined and compared with a corresponding detected value of the recorded marker, which in turn is used to render the virtual object. No search scope is set, which is scanned for at least one characteristic of the marker.

In EP 1 369 769 A2 A method for determining a registration accuracy of an augmented reality system is described.

In US 2003/043270 A1 A method for extracting a depth map from known camera and tracking data is described.

In US 2003/137524 A1 An augmented reality system is described for mixing an image representing a real environment and an image representing a virtual environment.

In Simon Prince et al .: "3-D live: Real Time Interaction for Mixed Reality ", Proc. of the 2002 ACM conference to computer-supported cooperative work, New Orleans, 2002, Louisiana, USA, pages 364-371 describes a real-time 3-D augmented reality video conference system being the image of the distant interlocutor in the one user rendered real scene rendered using a marker becomes.

disadvantage

adversely in the known methods of markerless tracking systems a lack of solution for initialization at the beginning of the tracking process and after the Loss of tracking objects, so if the adjustment of the projection with the picture of reality fails or an advantageously generated statement about quality of these considered too low.

Existing Need procedure an extensive initialization procedure by an experienced User (eg by clicking on points with the mouse). Besides, they are no methods are known, which also acceleration and or motion states mitinitalisieren.

Object of the invention

Of the The invention is based on the object of the process for initialization markerless tracking system to simplify and thus more accurate and realize faster tracking system. Further tasks will be apparent from the following description of the invention.

Description of the invention

you distinguishes between an initialization process and a tracking process.

During the Initialization process will be a so-called marker or any Object of the real environment, in the following initialization object, related to the camera, for example by measurement the position and / or orientation of the marker or initialization object. As a result, a camera image can be assigned to the real environment. The real environment can be a complex device and the initialization object may be a prominent element of the device.

During the following tracking operation, that is, the actual operation, while, for example, a user of the system receives context-dependent information displayed at a desired position relative to the real environment in a used display device, the marker or the initialization object serves as a reference to calculate the position, where the virtual information is to be displayed or displayed on an image captured by the camera. Since the user can change his position and his orientation to the environment, the marker or the initialization object must be continuously tracked (tracking) to the virtual information even with a changed position and / or a changed orientation of the user to the kor to display the correct position in the display device. This ensures that the information is displayed regardless of the position and / or orientation of the user in relation to the reality in the display device.

waits for example, a user of the system is a complex device, could the system a device element (object) with a frame (virtual Information), regardless of the current position or orientation of the user, as long as the marker or the initialization object can be captured with the camera.

The The object of the invention is achieved by a method for initialization a system solved, being with the system objects of a real environment and virtual Information overlaid together can be wherein a picture is taken with a camera, which is an initialization object includes, which is in the real environment, the recorded Image and a data model of the initialization object to a user displayed together, the recorded initialization object and the data model of the initialization object are superimposed and superimposed a search range is set that captures the recorded Initialization object and the data model of the initialization object, like superimposed, contains the search area for at least one feature of the initialization object is searched and the position, Orientie tion and / or size of at least a found feature is stored.

The System may be an opto-electronic system, such as an image processing system or a so-called tracking system.

The Camera can be non-stationary, for example on the head of a user be arranged. It is also possible, with a fixed camera a moving environment or a moving one To capture object. The environment can be a complex device for example, be a complex machine or a control cabinet. The initialization object can be a particularly prominent element be the complex device, such as a highly visible Machine element or an emergency stop switch of the control cabinet.

The Presentation can be over a semipermeable Glasses are made, with the user looking into the environment and on the semipermeable Glasses the data model of the initialization object is projected. In this case, a calibration between the eyes of the user, the camera and the semi-transparent Data glasses over the so-called SPAAM (single point active alignment method) take place. Similarly, the data model of the initialization object can be applied to the Retina of the user are projected. In doing so, the user takes the initialization object without intervening image capture true. Furthermore, can the initialization object and a data model of the initialization object a head-worn screen system are shown.

The Data model of the initialization object can be geometric information, For example, lines, vertices, radii, etc. included. The data model of the initialization object can be indicated by lines or vertices become. Furthermore, the data model can be a representation of a complex geometry, for example, a rectangle, a circle, a cylinder, a cuboid, a cone, a prism, etc., included. Further can the data model contain a three-dimensional object data model, z. B. CAD data.

Overlaying Can be done by moving the camera so that the initialization object at a given position in the image, or by moving of the data model.

at the size of the initialization object, the size of the data model, an inaccuracy when overlaying the data model of the initialization object via the initialization object, a size tolerance and / or a deviation of the data model from the initialization object, the from the still unknown position and perspective of the camera result, considered become. Furthermore, could considered in the search area border areas which are an image processing method for proper screening of the search area required.

Of the Search range can be after the at least one characteristic with the help of Image processing procedures are searched. These can, for example, geometric Filter or edge filter can be used. Possible image processing methods are well known to those skilled in the art and need at this point not explained further to become.

The Position, orientation and / or size or scaling of at least a found feature is stored, since this information is at Tracking needed becomes.

The method may associate the at least one feature of the initialization object found in the scan with the data model of the initialization object. This can be done, for example, by heuristic methods, as known per se to the person skilled in the art. If the data model contains edges, edges can be searched for as features and, after finding them, the corresponding one Edges of the data model of the initialization object. The method thus compares the data model of the initialization object with the image of the initialization object.

The Method may be the illustrated data model according to size and / or Change orientation in such a way that it is with the image of the initialization object taken by the camera essentially coincides. This step can be done before or after overlaying the data model over the recorded one Initialization object done. Further, this step may be after Set the search range while of the search and while of the assignment.

at The procedure can change this the size and / or Orientation of the data model through manipulation of the data model by the user before setting the search range. This can for example during of overlaying respectively.

at The procedure can change this the size and / or Orientation of the data model by the system. Changing the Size and / or Orientation of the data model is preferably done after setting of the search area. It can also be during of searching or while of the assignment.

The Method may further the camera position after assigning the at least a found feature of the initialization object with the data model of the initialization object.

The Procedure can be terminated automatically if sufficient good assignment of the at least one found feature of the initialization object reached with the data model of the initialization object.

The previously derived and derived from the prior art task is also according to the invention a system and method solved in the solution space for the Finding features in the video image is reduced.

The method according to the invention provides for the use of a digital, three-dimensional data model ( 200 ) objects of the real environment for identifying features in the video image of the camera ( 400-2 ) in front ( 1 ). The three-dimensional data model ( 200 ) is the user ( 300 ), advantageously with a head-mounted display system ( 400-1 ). At the same time, the user's real environment, advantageously in the form of a video image of the real environment, is displayed. The user overlays the three-dimensional data model of the object with the real environment, so that, according to the current viewing direction of the user, the object of the real environment (FIG. 100 ) through its digital data model ( 200 ) is shown covered.

Features of the real object are identified via known features of the digital data model with the aid of algorithms. This is in 1 exemplarily for the vertices (P1, P2, P3) of a real cube ( 100 ). The dimensions of the real cube are determined by its digital data model ( 200 ) known. In addition, the positions of the vertices of the real cube with respect to a fixed coordinate system are known. With the method according to the invention, the search space in the video image of the camera ( 400-2 ) is searched for features, in the example by points, limited to the projected points (P'1, P'2, P'3) and / or their immediate environment. The identification of the points is thereby significantly reduced, since all other points found in the video image are neglected.

The system according to the invention is in 2 shown. Input variables of the system are a video image ( 400-3 ) the camera ( 400-2 ) as well as the digital data model of a real object ( 200 ) via the data management system ( 800 ) is made available. In the mixed system ( 600-1 ), the data model ( 200 ) as well as the video image ( 400-3 ) the user ( 300 ) in the display device ( 500 ), advantageously in a head-mounted screen system ( 400-1 ). The mixing system ( 600-1 ) passes the video image ( 400-3 ) and the data model ( 200 ) to a definition system ( 600-2 ).

By manipulating the data model ( 200 ) and / or interaction with the data model, the user generates ( 300 ) the input quantity ( 700 ) for the definition system ( 600-2 ). The definition system ( 600-2 ) places two-dimensional search spaces in the video image ( 400-3 ) and transfers their dimension and position to the search system ( 600-3 ). The search system ( 600-3 ) searches for features in the video image and passes them to the mapping system ( 600-4 ). This arranges the in the video picture ( 400-3 ) Features Features of the Data Model ( 200 ) and passes the coordinates of the features found in the video image to the positioning system ( 600-5 ). Input information of the position determination system are additionally the coordinates ( 900 ) of the real object in the stationary space coordinate system via the data storage system ( 800 ) to provide. The positioning system ( 600-5 ) calculated on the basis of the coordinates of the features in the camera image as well as the coordinates of the features in the fixed coordinate system ( 900 ) the position of the camera ( 400-2 ) in the form of a transformation matrix ( 1000 ).

Output of the positioning system ( 600-5 ) forms the transformation matrix ( 1000 ) of other systems, eg. As augmented reality systems, can be used.

Advantages and Applications of Systems and procedures

advantages consist of the described system and method for markerless position determination in the restriction of for the feature finding necessary search space. Features included in the video image be found for the position determination in the sense of tracking but are irrelevant are thus excluded from the computationally intensive image processing. As a result, the performance and stability of the overall system increases considerably.

Possible applications The system and method in particular include applications of Augmented reality technology in the areas of service and maintenance, Production and other applications in the mobile environment.

100

Reales object

200

Three-dimensional digital data model of the real object

300

user

400-1

head-mounted screen system

400.2

camera

400.3

video image the camera

500

display

600.1

mixing system

600.2

definition system

600.3

search system

600.4

mapping system

600.5

Positioning System

700

User Interaction

800

Data storage system

900

stationary object coordinates

1000

transformation matrix

Claims (16)

A method for initializing a system, wherein the system can superimpose objects of a real environment and virtual information with each other, with a camera capturing an image containing an initialization object located in the real environment: characterized in that the captured image and a data model of the initialization object are shared with a user; that the recorded initialization object and the data model of the initialization object are superimposed and displayed superimposed; that a search area is set, which contains the recorded initialization object and the data model of the initialization object, as superimposed; that the search area is searched for at least one feature of the initialization object; and that the position, orientation and / or size of the at least one found feature is stored. Method according to claim 1, characterized in that that the at least one feature of the initialization object found during the scan is assigned to the Data model of the initialization object is assigned. Method according to claim 1 or 2, characterized that the illustrated data model according to size and / or orientation so is changed, that it is with the image of the initialization object taken by the camera essentially coincides. Method according to claim 3, characterized that changing the size and / or orientation of the data model by manipulation of the data model by the user before setting the search area. Method according to claim 3, characterized that changing the size and / or orientation of the data model by the system. Method according to one of claims 1 to 5, characterized that the data model contains a three-dimensional object data model. Method according to one of claims 1 to 6, characterized the method further comprises the camera position after assigning the at least one found feature of the initialization object determined with the data model of the initialization object. Method according to one of claims 1 to 7, characterized that the procedure terminates automatically if a sufficient good assignment of the at least one found feature of the initialization object reached with the data model of the initialization object. Method according to one of claims 1 to 8, characterized that the search space for the search for features in the camera image through digital data models is reduced. Method according to one of claims 1 to 9, characterized in that a movement model by detecting the movement of Benut zers at the time of initialization by additional information (advantageously optical flow, or acceleration sensors) is also initialized. System for markerless tracking of objects, wherein the system can superimpose objects of a real environment and virtual information, consisting of a mixing system ( 600-1 ) for the simultaneous display of a video image and a three-dimensional data model, a definition system ( 600-2 ) for defining search spaces in the video image, a search system ( 600-3 ) for finding features in the camera image, an assignment system ( 600-4 ) for assigning features in the camera image to features of the data model and a position determination system ( 600-5 ) for determining the position of the camera, characterized in that the search space for features located in the camera image is restricted by the user by manipulating a three-dimensional data model. System according to claim 11, characterized in that that a digital data model for mapping features in the video image is used. System according to claim 11, characterized in that that the exchange of information between the subsystems mixed, Definition, search, assignment and positioning system via wireless transmission technology takes place. System according to one of claims 11 to 13, characterized that the user the position, scaling and orientation of the displayed digital Change data model interactively can adapt to its current line of vision and thus adjusts the state vector of the tracking system. System according to one of claims 11 to 14, characterized that the displayed digital data model is that of the algorithm used measuring ranges for emphasize the user to simplify initialization. System according to one of claims 11 to 15, characterized that the system is self-contained Detects when the initialization position is reached.