DE102020121408A1 - Computer-implemented method and system for predicting an orientation and spatial position of a user's head-mountable display device - Google Patents

Abstract

The invention relates to a computer-implemented method for predicting an orientation and spatial position of a display device (10) that can be mounted on a user's head and has an optical arrangement (12) through which at least part of an environment is visible and on which virtual content (14) can be displayed, with a display (S4) of the virtual content (14) on the display device (10) as a function of a spatial position (P3) of the display device (10) estimated by a Kalman filter. The invention also relates to a system for predicting an orientation and spatial position of a display device (10) that can be mounted on a user's head. The invention also relates to a computer program and a computer-readable data carrier.

Description

The invention relates to a computer-implemented method for predicting an orientation and spatial position of a display device that can be mounted on a user's head. The invention further relates to a system for predicting an orientation and spatial position of a display device that can be mounted on a user's head.

Head-up displays in vehicles have been known for a long time. This is a holographic display system comprising a projection unit and a holographic beam combiner, the projection unit comprising a light source for emitting a light signal and a controllable imaging unit illuminated by the light source.

DE 102 27 467 A1 discloses such a holographic display system for vehicles. This comprises an imaging unit with which image information can be modeled onto the light signal and the modeled light signal can be directed onto the holographic beam combiner, with at least one viewing mirror being provided in and/or on the vehicle, through which the light emitted by the holographic beam combiner can enter the optical perception area of a User is steerable.

Furthermore, so-called AR (augmented reality) glasses are known, which have an optical arrangement through which at least part of an environment can be seen and on which virtual content can be displayed.

The disadvantage of such devices is that when the user's head moves, the content displayed on the glasses has to be repositioned accordingly in order to be visible to the user at the same place.

Since the rendering of the virtual content to be displayed requires a specific calculation time, the positioning of the virtual content on the display device results in imprecise positioning and/or a delayed display of the content due to the required calculation time.

In order to ensure a smooth presentation of the content on the display device, the repositioning of the content on the display device must be synchronized with a user's head movement in such a way that a time offset between the user's head movement and the adapted presentation of the virtual content on the display device is significantly below is 20 milliseconds.

Known methods for this are Late Correction and Asynchronous Reprojection, which intervene in the rendering pipeline via hardware interfaces. In cases or with AR glasses where it is not possible to intervene in the rendering pipeline, the above methods cannot be implemented. Thus, there are inaccuracies or process noise when adapting the prediction or repositioning of the virtual content on the display device.

Therefore, there is a need to provide an improved method and system for predicting an orientation and spatial position of a user's head mountable display device.

The object is achieved with a computer-implemented method for predicting an orientation and spatial position of a display device that can be mounted on a user's head, having the features of patent claim 1 .

Furthermore, the object is achieved with a system for predicting an orientation and spatial position of a display device that can be mounted on a user's head, having the features of patent claim 9 .

In addition, the object is achieved with a computer program having the features of patent claim 10 and with a computer-readable data carrier having the features of patent claim 11 .

The present invention provides a computer-implemented method for predicting an orientation and spatial position of a head-mounted display device having an optical arrangement through which at least part of an environment can be viewed and on which virtual content can be displayed.

The method includes detecting the spatial position of the display device using at least one first sensor system, in particular an inertial measuring unit, at at least a first point in time and a second point in time with a predetermined sampling rate.

The method also includes receiving position data of spatial positions of the display device detected by the at least one first sensor system at at least the first point in time and the second point in time using a Kalman filter.

In addition, the method includes estimating the spatial position of the display device at a third point in time using the acquired position data through the Kalman filter and displaying the virtual content on the display device as a function of the spatial position of the estimated by the Kalman filter at the third point in time display device.

The present invention also provides a system for predicting an orientation and spatial position of a display device that can be mounted on a user's head and has an optical arrangement through which at least part of an environment can be seen and on which virtual content can be displayed.

The system comprises at least one first sensor system, in particular an inertial measuring unit, which is set up to detect the spatial position of the display device at least at a first point in time and a second point in time with a predefined sampling rate.

The system also includes a computing device which is set up to receive position data of spatial positions of the display device detected by the at least one first sensor system at least at the first point in time and second point in time through a Kalman filter, and wherein the computing device is set up to calculate the spatial Estimate position of the display device at a third point in time using the acquired position data through the Kalman filter.

The system also has means for displaying the virtual content on the display device as a function of the spatial position of the display device estimated by the Kalman filter of the computing device at the third point in time.

The present invention also creates a computer program with program code to carry out the method according to the invention when the computer program is run on a computer.

The present invention also creates a computer-readable data carrier with program code of a computer program in order to carry out the method according to the invention when the computer program is run on a computer.

An idea of the present invention is to use the Kalman filter to estimate the future spatial position of the display device that can be mounted on the user's head to obtain an exact position at a defined point in time, based on which an adapted representation of the virtual content is displayed on the display device the correct coordinates is possible. In this way, a display result can advantageously be achieved which is at the level of an intervention in the rendering pipeline, without the need for a hardware interface in order to carry out the method.

According to a preferred development, it is provided that a comparison of the estimated spatial position of the display device at the third point in time is carried out with a position of the display device detected by the at least one first sensor system at the third point in time, with a difference between the estimated spatial position of the display device at the third point in time and the detected spatial position of the display device at the third point in time is applied as feedback to a control input of the Kalman filter.

As a result, process noise of the Kalman filter can be reduced, since applying the feedback or feedback to the control input of the Kalman filter allows the state space to be remodeled, on the basis of which a more accurate prediction can be achieved. The more iterations the Kalman filter goes through, the more precise the prediction becomes.

According to a further preferred development, it is provided that the difference between the estimated spatial position of the display device at the third point in time and the detected spatial position of the display device at the third point in time is used to adapt a covariance matrix of the Kalman filter.

The covariance matrix advantageously describes the relationship between the detected spatial position of the display device at the third point in time and the spatial position of the display device estimated by the Kalman filter at the third point in time, with the adaptation of the covariance matrix enabling the prediction accuracy of the Kalman filter to be improved.

According to a further preferred development, it is provided that the spatial position of the display device is detected using a second, optical sensor system at least at a first point in time and a second point in time with a predetermined sampling rate.

Due to the additional use of the second, optical sensor system can advantageously improved detection of spatial union position of the display device are made possible.

According to a further preferred development, it is provided that the second, optical sensor system detects markings arranged on the display device using a plurality of camera sensors, with the second, optical sensor system determining a spatial Position of markers calculated. In this way, an exact spatial positioning of the display device can be calculated in an advantageous manner.

According to a further preferred development, it is provided that the second, optical sensor system, using a plurality of camera sensors arranged on the display device, detects markings arranged in a space surrounding the display device, the second, optical sensor system from a position of the markings on the camera images and a relative positioning of the camera sensors to each other calculates a spatial position of the display device.

This alternative embodiment also enables an exact calculation of the position of the sensor device in space, with this variant advantageously eliminating the spatially arranged camera sensors, since these are arranged on the display device itself.

According to a further preferred development, it is provided that the Kalman filter estimates an angular velocity, in particular a rotational movement, and a lateral velocity, in particular a translational movement, of the display device. A precise estimation of the spatial position of the display device can thus be made possible, which includes both a head rotation of the user and a lateral movement as well as a movement which partially includes both a rotational movement and a lateral movement.

According to a further preferred development it is provided that an estimated position of the display device at the third time is formed by the product of the detected position of the display device at the third time and the derivation of the detected position of the display device at the third time. The Kalman filter is thus able to undertake an exact estimation of the position of the display device in space on the basis of the above-mentioned equation of motion.

The configurations and developments described can be combined with one another as desired.

Further possible refinements, developments and implementations of the invention also include combinations of features of the invention described above or below with regard to the exemplary embodiments that are not explicitly mentioned.

character list

The accompanying drawings are provided to provide a further understanding of embodiments of the invention. They illustrate embodiments and, together with the description, serve to explain principles and concepts of the invention.

Other embodiments and many of the foregoing advantages will become apparent by reference to the drawings. The illustrated elements of the drawings are not necessarily shown to scale with respect to one another.

• 1 ein Ablaufdiagramm eines computerimplementierten Verfahrens zur Vorhersage einer Orientierung und räumlichen Position einer am Kopf eines Nutzers montierbaren Anzeigevorrichtung gemäß einer bevorzugten Ausführungsform der Erfindung;
• 2 eine schematische Darstellung eines Systems zur Vorhersage der Orientierung und räumlichen Position der am Kopf des Nutzers montierbaren Anzeigevorrichtung gemäß der bevorzugten Ausführungsform der Erfindung;
• 3 eine schematische Darstellung des Systems zur Vorhersage der Orientierung und räumlichen Position der am Kopf des Nutzers montierbaren Anzeigevorrichtung gemäß einer weiteren bevorzugten Ausführungsform der Erfindung; und
• 4 eine schematische Darstellung einer räumlichen Positionsänderung der Anzeigevorrichtung gemäß der bevorzugten Ausführungsform der Erfindung.

Show it:

• 1 a flow chart of a computer-implemented method for predicting an orientation and spatial position of a head-mountable display device according to a preferred embodiment of the invention;
• 2 12 is a schematic representation of a system for predicting the orientation and spatial position of the user's head-mountable display device according to the preferred embodiment of the invention;
• 3 a schematic representation of the system for predicting the orientation and spatial position of the user's head mountable display device according to a further preferred embodiment of the invention; and
• 4 a schematic representation of a spatial change in position of the display device according to the preferred embodiment of the invention.

In the figures of the drawings, the same reference symbols designate the same or functionally identical elements, parts or components, unless otherwise stated.

1 Figure 1 shows a flow diagram of a computer-implemented method for predicting an orientation and spatial position of a user's head mountable display direction according to a preferred embodiment of the invention.

The method includes detecting S1 the spatial position P1, P2 of the display device 10 using at least one first sensor system M1, in particular an inertial measuring unit, at at least a first point in time T1 and a second point in time T2 with a predetermined sampling rate.

Alternatively or additionally, the detection S1′ of the spatial position P1, P2 of the display device 10 can be carried out using a second, optical sensor system M2 at at least a first point in time T1 and a second point in time T2 with a predetermined sampling rate.

The method also includes receiving S2 position data of spatial positions P1, P2 of the display device 10 detected by the at least one first sensor system M1 at the at least first point in time T1 and second point in time T2 by a Kalman filter.

The method also includes estimating S3 the spatial position P3 of the display device 10 at a third point in time T3 using the acquired position data through the Kalman filter, and displaying S4 the virtual content 14 on the display device 10 as a function of the through the Kalman filter spatial position P3 of the display device 10 estimated at the third point in time T3.

A comparison S5 of the estimated spatial position P3 of the display device 10 at the third point in time T3 is then carried out with a position P3′ of the display device 10 detected by the at least one first sensor system M1 at the third point in time T3.

A difference ΔP between the estimated spatial position P3 of the display device 10 at the third point in time T3 and the detected spatial position P3′ of the display device 10 at the third point in time T3 is applied as feedback to a control input of the Kalman filter.

2 Figure 12 shows a schematic representation of a system for predicting the orientation and spatial position of the user's head mountable display according to the preferred embodiment of the invention.

The display device 10 has an optical arrangement 12 through which at least part of an environment can be seen and on which virtual content 14 can be displayed.

The virtual content 14 appears three-dimensionally projected into space for the user. The virtual content 14 can be, for example, navigation data from a vehicle navigation system, driving information such as a vehicle speed, traffic information and/or information from an infotainment system.

The system comprises at least one first sensor system M1, in particular an inertial measuring unit, which sensor system M1 is set up to detect the spatial position P1, P2 of the display device 10 at least at a first point in time T1 and a second point in time T2 with a predetermined sampling rate.

The system also includes a computing device 26, which is set up to receive position data of spatial positions P1, P2 of the display device 10 detected by the at least one first sensor system M1 at the at least first point in time T1 and second point in time T2 through a Kalman filter.

Furthermore, the arithmetic unit 26 is set up to estimate a spatial position of the display device 10 at a third point in time T3 using the acquired position data through the Kalman filter.

In addition, the system comprises means for displaying the virtual content 14 on the display device 10 as a function of the spatial position of the display device 10 estimated by the Kalman filter of the computing device 26 at the third point in time T3.

When the position of display device 10 changes or is repositioned in space 22, preferably a passenger compartment of a motor vehicle, the virtual content on optical arrangement 12 of display device 10 is correspondingly repositioned in such a way that the user always sees it at the same position in space 22 .

Thus, for example, navigation information can be displayed spatially at a predetermined position in the driver's field of vision above a hood. For example, other virtual content such as infotainment information, incoming calls and/or messages or emails may be presented in another designated area.

in the in 2 The representation shown shows the display device 10 in the first spatial position P1 at the time T1. In addition to the first sensor system M1, the present Embodiment also uses a second sensor system M2. The second sensor system M2 is an optical sensor system that is also set up to detect the spatial position P1, P2 of the display device 10 at least at the first point in time T1 and at the second point in time T2 with a predefined sampling rate.

The position of the display device 10 in space is continuously recorded with the specified sampling rate. The spatial position of the display device 10 is also continuously recorded with the first sensor system M1 at the specified sampling rate, so that the Kalman filter always receives the current last position of the display device 10 in space as input data.

The second, optical sensor system M2 detects markings 24a, 24b arranged on the display device 10 using a plurality of camera sensors 20, 21, in the present embodiment two.

The second, optical sensor system M2 calculates a spatial position PM1 of the markings 24a, 24b from the position of the markings 24a, 24b on the camera images and a relative positioning of the camera sensors 20, 21 to one another. Thus, the spatial arrangement of the display device 10 is known at any given point in time.

The Kalman filter estimates an angular velocity, in particular a rotational movement, and a lateral velocity, in particular a translational movement, of the display device 10.

3 FIG. 12 shows a schematic representation of the system for predicting the orientation and spatial position of the user's head-mountable display device according to a further preferred embodiment of the invention.

In the present illustration, the display device 10 is shown in the second position P2 at the second point in time T2 and in the third position P3 at the third point in time T3.

The representation of the display device 10 in the second position P2 and the third position P3 represents all embodiments of the invention.

The specific design of 3 differs therefrom only in that the second, optical sensor system M2′ is provided on the display device 10 using a plurality of camera sensors 20′, 21′ arranged on the display device 10, two in the present embodiment. These detect markings 24a′, 24b′ arranged in the surrounding space 22 . The second, optical sensor system M2' then calculates a spatial position PM2 of the display device 10 from a position of the markings 24a', 24b' on the camera images and a relative positioning of the camera sensors 20', 21' to one another.

4 shows a schematic representation of a spatial change in position of the display device according to the preferred embodiment of the invention.

The points marked along the curve relate to the detected and estimated spatial positions of the display device 10. As can be seen from the figure, there is a difference between the estimated spatial position P3 of the display device 10 at the third point in time T3 and that determined by the first sensor system M1 and/or the second sensor system M2 at the third point in time T3 detected position P3 'of the display device a difference ΔP.

This difference ΔP is applied as feedback to a control input of the Kalman filter. In particular, the difference ΔP is used to adapt a covariance matrix of the Kalman filter.

The term vehicle includes cars, trucks, buses, mobile homes, motorcycles, etc., which are used to transport people, goods, etc.

In particular, the term includes motor vehicles for passenger transport. Additionally or alternatively, a hybrid or electric vehicle according to embodiments may be a pure electric vehicle (BEV) or a plug-in hybrid vehicle (PHEV). However, other forms of propulsion can also be used, for example in the form of a diesel or petrol-powered vehicle. The vehicle can also be in the form of a rail vehicle.

Although the invention has been illustrated and explained in more detail by means of preferred exemplary embodiments, the invention is not restricted by the disclosed examples and other variations can be derived therefrom by a person skilled in the art without departing from the protective scope of the invention.

It is therefore clear that a large number of possible variations exist. Embodiments given by way of example only represent examples and are not to be construed in any way as limiting the scope, application or use abilities or the configuration of the invention are to be understood.

Rather, the preceding description and the description of the figures enable the person skilled in the art to concretely implement the exemplary embodiments, whereby the person skilled in the art, knowing the disclosed inventive concept, can make a variety of changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without having to Leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

The data relating to the spatial position P1, P2 of the display device 10 is not only recorded, but preferably also stored in a data memory and/or temporarily stored in a temporary data memory for data processing purposes.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 10227467 A1 [0003]

Claims (11)

Computer-implemented method for predicting an orientation and spatial position of a display device (10) that can be mounted on a user's head and has an optical arrangement (12) through which at least part of an environment is visible and on which virtual content (14) can be displayed the steps: Detection (S1) of the spatial position (P1, P2) of the display device (10) using at least a first sensor system (M1), in particular an inertial measuring unit, at least at a first point in time (T1) and a second point in time (T2) with a predetermined Sampling rate; Receiving (S2) by the at least one first sensor system (M1) detected position data of spatial positions (P1, P2) of the display device (10) at least at the first time (T1) and second time (T2) by a Kalman filter; Estimating (S3) the spatial position (P3) of the display device (10) at a third point in time (T3) using the detected position data through the Kalman filter; and Displaying (S4) the virtual content (14) on the display device (10) as a function of the spatial position (P3) of the display device (10) estimated by the Kalman filter at the third point in time (T3). Computer-implemented method claim 1 , wherein a comparison (S5) of the estimated spatial position (P3) of the display device (10) at the third point in time (T3) with a position (P3') of the display device detected by the at least one first sensor system (M1) at the third point in time (T3). (10) is performed, wherein a difference (ΔP) between the estimated spatial position (P3) of the display device (10) at the third point in time (T3) and the detected spatial position (P3 ') of the display device (10) at the third point in time (T3 ) is applied as feedback to a control input of the Kalman filter. Computer-implemented method claim 2 , wherein the difference between the estimated spatial position (P3) of the display device (10) at the third point in time (T3) and the detected spatial position (P3') of the display device (10) at the third point in time (T3) is used to create a covariance matrix of the Adjust Kalman filter. Computer-implemented method according to one of the preceding claims, wherein a detection (S1') of the spatial position (P1, P2) of the display device (10) using a second, optical sensor system (M2) at least at a first point in time (T1) and a second point in time ( T2) is performed at a predetermined sampling rate. Computer-implemented method claim 3 , wherein the second, optical sensor system (M2) using a plurality of camera sensors (20, 21) on the display device (10) arranged markings (24a, 24b) detects, wherein the second, optical sensor system (M2) from a position of the markings (24a, 24b) calculates a spatial position (PM1) of the markings (24a, 24b) on the camera images and a relative positioning of the camera sensors (20, 21) to one another. Computer-implemented method claim 3 , wherein the second, optical sensor system (M2') using a plurality of camera sensors (20', 21') arranged on the display device (10) arranged markings (24a', 24b) in a space (22) surrounding the display device (10). '), the second optical sensor system (M2') determining a spatial position (PM2) of the Display device (10) calculated. Computer-implemented method according to one of the preceding claims, wherein the Kalman filter estimates an angular velocity, in particular a rotational movement, and a lateral velocity, in particular a translational movement, of the display device (10). Computer-implemented method according to one of the preceding claims, wherein an estimated position (P3) of the display device (10) at the third time (T3) by the product of the detected position (P3') of the display device (10) at the third time (T3) and the derivative the detected position (P3') of the display device (10) at the third point in time (T3). System for predicting an orientation and spatial position of a display device (10) that can be mounted on a user's head and has an optical arrangement (12) through which at least part of an environment can be seen and on which virtual content (14) can be displayed, comprising: at least one first sensor system (M1), in particular an inertial measuring unit, which is set up to determine the spatial position (P1, P2) of the display device (10) at least at a first point in time (T1) and a second point in time (T2) with a predetermined capture sampling rate; a computing device (26) which is set up for this purpose by the at least one first sensor system (M1) to receive detected position data of spatial positions (P1, P2) of the display device (10) at least at the first point in time (T1) and second point in time (T2) through a Kalman filter, and wherein the computing device (26) is set up to to estimate a spatial position of the display device (10) at a third point in time (T3) using the detected position data through the Kalman filter; and means for displaying the virtual content (14) on the display device (10) as a function of the spatial position of the display device (10) estimated by the Kalman filter of the computing device (26) at the third point in time (T3). Computer program with program code to implement the method according to one of Claims 1 until 8th to be performed when the computer program is run on a computer. Computer-readable data carrier with program code of a computer program for the method according to one of Claims 1 until 8th to be performed when the computer program is run on a computer.

Images