EP3685213A1

EP3685213A1 - Method, device and computer-readable storage medium with instructions for controlling a display of an augmented-reality head-up display device for a motor vehicle

Info

Publication number: EP3685213A1
Application number: EP18759927.9A
Authority: EP
Inventors: Vitalij SADOVITCH
Original assignee: Volkswagen AG
Current assignee: Volkswagen AG
Priority date: 2017-09-21
Filing date: 2018-08-23
Publication date: 2020-07-29
Also published as: WO2019057439A1; US11250816B2; DE102017216774A1; CN111095078A; US20210110791A1; DE102017216774B4

Abstract

The invention relates to a method, a device and a computer-readable storage medium with instructions for controlling a display of an augmented-reality head-up display device for a motor vehicle. An appropriate methodology is used to detect (10) interfering movements of the motor vehicle. In addition, a graphics generator generates (11) a marker for display by means of the augmented-reality head-up display device, the marker being dynamically embodied. Finally, the generated marker is outputted (12) for display by the augmented-reality head-up display device.

Description

description

A method, apparatus and computer readable storage medium having instructions for controlling a display of an augmented reality head-up display apparatus for a computer

motor vehicle

The present invention relates to a method, a device and a computer-readable storage medium with instructions for controlling a display of an augmented reality head-up display device for a motor vehicle. The invention further relates to a motor vehicle in which a method according to the invention or a device according to the invention is used.

With the continuous development of virtual and augmented reality technologies and applications, these are finding their way into the automobile. Augmented reality (AR), augmented reality, is the enrichment of the real world by virtue of virtual elements that are registered in three-dimensional space in the correct location and allow real-time interaction. Augmented reality "has prevailed over the term" augmented reality "is used in the following the former.

A possible technical realization to enrich the driver's workplace with perspectively correct virtual extensions is provided by the Head-Up-Display (HUD). The light rays of a built-in dashboard display are folded over several mirrors and lenses and mirrored through a projection screen in the eye of the driver so that he perceives a virtual image outside the vehicle. In the automotive sector, the front screen, whose curved shape must be taken into account in the illustration, often serves as the projection surface. As an alternative, an additional disc made of glass or plastic is used in part, which is arranged between the driver and the windshield on the dashboard. Due to the optical overlay of the display and the driving scene, fewer head and eye movements are required to read the information. In addition, the adaptation effort for the eyes is reduced since, depending on the virtual distance of the display, less, if at all, does not have to be accommodated. Current serial HUDs, however, are limited to displays just below the driver's primary field of view and contain redundant information, such as the speed gauge, which can also be found elsewhere in the cockpit. Although this display technology reduces eye gaze away from the street, it still has the disadvantage that the presented information is interpreted and based on the real situation must be transmitted because they are not registered in the real scene. In complex situations, this can be a mentally demanding cognitive process. By marking objects and displaying information at their real reference location, ie by a contact-analog representation, environmental information can be displayed directly in the driver's field of vision. This direct visual enrichment of the environment in the form of augmented reality can significantly reduce cognitive transfer requirements.

In this context, DE 10 2014 214 516 A1 discloses a device, in particular a data glasses or a head-up display, for reproducing data in an augmented reality for an occupant of a vehicle. The device is set up to display data, in particular visual media contents, in an image section of the field of view of the occupant, which currently and probably also in the future does not contain any environmental information essential for traffic around the vehicle.

Augmented Reality offers a wide range of possible applications to support the driver by contact-analogous marking of lanes and objects. Relative

obvious examples mostly refer to the field of navigation. While traditional navigation displays in conventional head-up displays typically display schematic representations, e.g. a right arrow pointing to the right as a sign that you want to turn right at the next opportunity, offers AR ads much more effective options. Because the ads can be viewed as "part of the environment," the user is provided with extremely fast and intuitive interpretations of what is happening in modern automobiles

Environment sensors detected. The aim of current developments is to make this data meaningful

To integrate augmented reality representations.

Registration of the virtual information in the driver's real field of vision in the sense of augmented reality places very high demands on the technical implementation. In order to be able to represent virtual contents in the right place and perspective in the real world, very detailed knowledge about the environment and proper movement of the vehicle is necessary. In order for the driver to be correctly positioned in the world, the virtual overlay must be positioned and aligned in three-dimensional space, depending on the position of the vehicle, since the head-up display is installed in the vehicle. The vehicle attitude changes by pitch, roll and yaw movements permanently and highly dynamically. A detection by rotation rate and acceleration sensors is sufficiently precise possible. However, the data processing chain is from the sensory Data acquisition over the data processing and the 3D rendering up to the representation of the display so long that a latency within the range of 100ms results. In this case, latencies in the range of 15 ms are perceived as disturbing in augmented reality applications.

Pitching movements of the vehicle have the greatest effect on the positioning of the virtual display. The display will appear to move up and down. Since the relative height represents a strong monocular depth criterion, objects that are higher up in the

Visual field of view, perceived by humans as farther away. The

Nick movements therefore have a direct effect on the perceived imaging distance. As a result, the influences of driving dynamics lead to the virtual image being perceived as restless and unstable.

Against this background, DE 10 2014 221 608 A1 describes a method for

Deactivating a contact analog display on data glasses in a motor vehicle. For this purpose, data on the movement of the motor vehicle, in particular for

Lateral acceleration of the motor vehicle; provided. As long as the vehicle movement of the motor vehicle exceeds a predetermined threshold, the display of a contact-analogue representation on the data goggles is suspended.

Although the above approach ensures that the virtual image is perceived as relatively stable, but with the mandatory restriction that no contact-analogous representation occurs during larger vehicle movements. Another approach to reducing the problem described is to reduce the latencies occurring by optimizing the hardware or software. For example, with regard to the hardware, the tracking method can be optimized by increasing the temporal and spatial sampling rate or by synchronization methods [1]. In terms of software, Predictive Tracking [2], Late Latching [3] and Image Deflection [4] should be mentioned. However, the approaches mentioned are associated with a significant cost.

It is an object of the invention to provide alternative solutions for the control of a display of an augmented reality head-up display device for a motor vehicle.

This object is achieved by a method with the features of claim 1, by a computer-readable storage medium with instructions according to claim 10 and by a Device with the features of claim 1 1 solved. Preferred embodiments of the invention are the subject of the dependent claims.

According to a first aspect of the invention, a method for controlling a display of an augmented reality head-up display device for a motor vehicle comprises the steps of:

Generating a marker for display by the augmented reality head-up display device, the marker having a dynamic design; and

Outputting the mark for display by the augmented reality head-up display device.

In accordance with another aspect of the invention, a computer-readable storage medium includes instructions that, when executed by a computer, cause the computer to perform the following steps to control a display of an augmented reality head-up display device for a motor vehicle:

The term computer is to be understood broadly. In particular, it also includes

Controllers and other processor-based data processing devices.

According to another aspect of the invention, an apparatus for controlling a display of an augmented reality head-up display device for a motor vehicle comprises:

- A graphic generator for generating a marker for display by the

An augmented reality head-up display device, the graphics generator configured to generate the tag with a dynamic design; and

An output for outputting the mark for display by the augmented reality head-up display device.

Inappropriate movements of the ad are perceived as disturbing by users because the visual system is very sensitive to movement in a quiet or static environment. If a calm display is expected, movements will be lifted off the visual scene, thus disturbing the overall impression. According to the invention, this effect is reduced by a dynamic or movable design of the display, because then additional disturbance-related movements by the driving dynamics not so salient stand out from the correct display, which already contains a basic movement. By means of the dynamic design of the marking thus the subjective perception of the disturbing motion is reduced.

In one aspect of the invention, the label is a contact analog label. Inappropriate movements of the marking are particularly noticeable in contact-analogous markings, since the user has a reference object available, for example a vehicle or traffic sign highlighted by the marking. The application of the solution according to the invention is therefore particularly advantageous for contact-analogous labels.

According to one aspect of the invention, the dynamic design of the marking comprises a design as a running light or a pulsation of the marking. In this case, a running direction of the running light preferably corresponds to a direction of disturbing movements of

Motor vehicle. Both design options can be easily realized without bringing excessive movement and thus restlessness in the display.

According to one aspect of the invention, the marker only has a dynamic design when jamming movements of the motor vehicle are detected. Because a dynamic display can have a higher deflection potential, it is

It is advisable not to use the solution on a permanent basis, but only in times when strong movements of the advertisement occur or are to be expected. In this case, it is preferably provided that the marking has a dynamic design only if the disturbing movements of the motor vehicle exceed a threshold value. In this way, too frequent a change between a static configuration of the marking and a dynamic embodiment of the marking is prevented, which could otherwise be perceived by the user as disturbing.

In one aspect of the invention, the jamming motions are around

Nick movements of the motor vehicle. As described above, the result

Disturbing movements of the display mainly from pitching movements of the vehicle in combination with a relatively high system latency. The higher the pitch angle and pitch rate of the vehicle at constant latency, the greater the impact on the display movement. In practice, it is therefore sufficient to detect only the pitching movements and to evaluate them for the purpose of intelligent application of the dynamic design. According to one aspect of the invention, the disturbing movements of the motor vehicle are detected with an acceleration sensor or derived from a road condition. Since the latency is almost constant, the strength of the display movement can be derived directly from the vehicle movements. These can be detected relatively easily by means of a gyroscope. Movements of the vehicle body are not only excited by dynamic processes such as acceleration and braking, but also by the nature of the road surface. Therefore, preferably also an analytical evaluation of the road condition for the purpose of an intelligent application of dynamic display elements is provided.

According to one aspect of the invention, the roadway condition by means of a

Laser scanner or a stereo camera detected or taken from maps. Some modern map material already contains information about the type of roadway so that the road surface can be taken directly from the map material without any additional sensors. Furthermore, it is also possible to scan the road using a stereo camera and laser scanner. Since such sensors are already partially installed for other applications in the vehicle, can at no additional cost to this

Components are used.

Particularly advantageous are a method according to the invention or a

Device according to the invention in a vehicle, in particular a motor vehicle used.

Further features of the present invention will become apparent from the following description and the appended claims, taken in conjunction with the figures.

Fig. 1 shows schematically a method for controlling a display of an augmented reality head-up display device for a motor vehicle;

FIG. 2 shows a first embodiment of a device for controlling a display of an augmented reality head-up display device for a motor vehicle; FIG.

Fig. 3 shows a second embodiment of a device for controlling a

Display of an augmented reality head-up display device for a motor vehicle; Fig. 4 schematically illustrates a motor vehicle in which a solution according to the invention is realized;

Fig. 5 shows schematically the general structure of a head-up display device for a motor vehicle;

Fig. 6 shows an example of an augmented reality representation of a head-up display device;

Fig. 7 shows a system design of the inventive solution for controlling a

Display of an augmented reality head-up display device for a motor vehicle;

Fig. 8 shows a first example of a dynamic embodiment of a marker; and

Fig. 9 shows a second example of a dynamic embodiment of a marker.

For a better understanding of the principles of the present invention, embodiments of the invention will be explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments, and that the features described can also be combined or modified without departing from the scope of the invention as defined in the appended claims.

1 schematically shows a method for controlling a display of an augmented reality head-up display device for a motor vehicle. By means of a suitable method, interference movements of the motor vehicle are detected 10, in particular pitching movements of the motor vehicle. For example, the disturbing movements of the motor vehicle can be detected with an acceleration sensor or derived from a road condition. In this case, the road condition can be detected by means of a laser scanner or a stereo camera or removed from map material. In addition, a graphics generator generates a marker for display by the augmented reality head-up display device, wherein the marker has a dynamic design. The dynamic embodiment may include an embodiment as a running light or a pulsation of the marking. The marking may be, for example, a contact-analogous marking. The generated tag is finally displayed by the augmented reality head-up display device to a controller of the Augmented reality head-up display device output 12. Preferably, the mark has only a dynamic design, when jamming movements of the motor vehicle are detected. It can be provided that the interference movements must also exceed a threshold. Alternatively, instead of outputting the generated tag to the controller, only one instruction may be issued to the controller to generate a tag with a dynamic design. It is also possible that the described method is carried out completely by the control unit of the augmented reality head-up display device.

2 shows a simplified schematic representation of a first embodiment of a device 20 for controlling a display of an augmented reality head-up display device for a motor vehicle. The device 20 has an input 21, via which information on movements of the motor vehicle or on a road surface can be received by a sensor system 44 or by a navigation system 45. The sensor system 44 may include an acceleration sensor. The

Movements of the motor vehicle can alternatively be derived from the road condition. In this case, the sensor system 44 may comprise a laser scanner or a stereo camera for detecting the road condition. The

Road condition can also be taken from map material.

The device 20 also has an analysis unit 22 for detecting jamming movements of the motor vehicle, in particular pitching movements of the motor vehicle. On

Graphics generator 23 generates a marker for display by the augmented reality head-up display device, the marker having a dynamic design. Via an output 26 of the device 20, the generated marking is output to a control unit 42 of the augmented reality head-up display device. Alternatively, only one instruction may be output to the controller 42 to generate a tag with a dynamic design. The controller 42 may then insert the generated tag into a display of the augmented reality head-up display device. Preferably, the marker has a dynamic design only if jamming movements of the motor vehicle are detected. It can be provided that the interference movements must also exceed a threshold.

The analysis unit 22 and the graphics generator 23 can be controlled by a control unit 24. If necessary, settings of the analysis unit 22, the graphics generator 23 or the control unit 24 can be changed via a user interface 27. If required, the data accumulating in the device 20 can be stored in a memory 25 of the device 20, for example for later evaluation or for use by the components of the device 20. The analysis unit 22, which

Graphics generator 23 and the control unit 24 can be implemented as dedicated hardware, for example as integrated circuits. Of course, they may also be partially or fully combined or implemented as software running on a suitable processor, such as a GPU. The input 21 and the output 26 may be implemented as separate interfaces or as a combined bidirectional interface. In the example described, the device 20 is a separate component. However, it can also be integrated in the control unit 42 of the augmented reality head-up display device.

3 shows a simplified schematic representation of a second embodiment of a device 30 for controlling a display of an augmented reality head-up display device for a motor vehicle. The device 30 has a processor 32 and a memory 31. For example, device 30 is a computer or controller. In the memory 31 instructions are stored, which cause the device 30 when executed by the processor 32, the steps according to one of

to carry out the described method. The instructions stored in the memory 31 thus embody a program which can be executed by the processor 32 and implements the method according to the invention. The device 30 has an input 33 for receiving information, such as graphics data or jamming information. Data generated by the processor 32 is provided via an output 34. In addition, they can be stored in the memory 31. The input 33 and the output 34 can be combined to form a bidirectional interface.

The processor 32 may include one or more processing units, such as microprocessors, digital signal processors, or combinations thereof.

The memories 25, 31 of the described embodiments can have both volatile and non-volatile memory areas and a wide variety of memory devices and

Storage media include, for example, hard disks, optical storage media or semiconductor memory.

Fig. 4 schematically illustrates a motor vehicle 40 in which a solution according to the invention is realized. The motor vehicle 40 has an augmented reality head-up display device 41 with an associated control unit 42. Furthermore, the motor vehicle 40, a device 20 for controlling a display of the augmented reality head-up display device 41 on. Of course, the device 20 may also be integrated in the augmented reality head-up display device 41 or in the control device 42 of the augmented reality head-up display device 41. Other components of the motor vehicle 40 include a camera 43, a sensor system 44 for detecting vehicle movements, a navigation system 45, a data transmission unit 46 and a number of

Assistance systems 47, one of which is shown as an example. By means of

Data transmission unit 46 can be set up a connection to service providers, for example for retrieving map data. For storage of data, a memory 48 is present. The data exchange between the various components of the motor vehicle 40 takes place via a network 49.

Hereinafter, preferred embodiments of the invention will be described with reference to FIGS 5 to 9.

5 schematically shows an augmented reality head-up display device 41 for a motor vehicle 40, with the aid of which contents can be displayed on a projection surface 52 of the motor vehicle 40, for example on the windshield or on an additional pane of glass or Plastic, which is arranged between the driver and the windshield on the dashboard. The displayed contents are generated by an imaging unit 50 and projected on the projection surface 52 by means of an optical module 51. Typically, the projection takes place in an area of the windshield above the steering wheel. The imaging unit 50 may be, for example, an LCD TFT display. The augmented reality head-up display device 41 is usually installed in a dashboard of the motor vehicle 40.

6 shows an example of an augmented reality representation of an augmented reality head-up display device. The actual environment is superimposed, by way of example, on a contact-analogue representation 60, which is a preceding vehicle 61 for visualizing an ACC system status (ACC: Adaptive Cruise Control;

Speed control). In addition, navigation information,

Warnings or other items are displayed. A disadvantage of the marking 60 is that it moves in the case of pitching movements of the motor vehicle for the viewer up and down relative to the preceding vehicle 61.

7 shows a system design of the solution according to the invention for controlling a display of an augmented reality head-up display device for a motor vehicle. On Sensor system 44 transmits sensor data to the device 20 for controlling the display of the augmented reality head-up display device 41. Alternatively or additionally, a navigation system 45 provides map data or information about

Road condition transmitted to the device 20. An analysis unit 22 detects interference movements by means of this information. Information on the occurrence of interference movements are passed on by the analysis unit 22 to a graphics generator 23, which then generates a dynamic marking. This is output to a controller 42 of the augmented reality head-up display device 41, which in turn generates the head-up display for the augmented reality head-up display device 41. Alternatively, an instruction can also be output to the control unit 42, in turn generating a dynamic marking. In addition, the device 20 may also be part of the

Control unit 42 be.

Fig. 8 shows a first example of a dynamic embodiment of a marker 60. In this example, the contact-analogous marker 60 of the preceding vehicle 61 is configured as a running light. Shown is the temporal evolution of the marker 60 on the basis of three consecutive times. The density of the filling of the marker 60 indicates approximately the perceptibility of the marker 60 at the corresponding position. The visibility can be controlled for example via a luminous intensity. The movement of the running light is from top to bottom, i. along the z-axis of the motor vehicle. The running light thus generates a basic dynamic in the same direction in which the virtual image also moves in front of the stable background. The image movement becomes less visible.

9 shows a second example of a dynamic embodiment of a marker 60. In this example, the contact-analogous marker 60 of the preceding vehicle 61 is designed to be pulsating. In turn, the temporal development of the mark 60 is shown on the basis of three successive times. The density of the filling of the marking 60 again indicates the perceptibility of the marking 60 at the corresponding position. While dynamic is not associated with a direction of motion in this form of dynamic design, the fade-and-fade-out of the marker 60 distracts the viewer to some degree from the relative motion of the virtual image against the stable background.

For the concrete implementation of the dynamic design, the person skilled in various design options are available. In addition to the described running light or the pulsation dynamics can, for example, by a color change of Marking, by flashing (as an extreme form of pulsation) of the marking or by a spatial displacement or jumping of the mark can be realized.

LIST OF REFERENCE NUMBERS

Detecting disturbing movements of the motor vehicle

Generate a dynamic marker for display

Output the generated dynamic marker

device

entrance

analysis unit

graphic generator

control unit

Storage

exit

User interface

device

Storage

processor

entrance

exit

motor vehicle

Augmented reality head-up display device

Control unit of the augmented reality head-up display device

camera

sensor system

navigation system

Data transfer unit

assistance system

Storage

network

Imaging unit

Optical module

projection

mark

Preceding vehicle references

[1] http://www.hizook.com/blog/2015/05/17/valves-lighthouse-tracking-system-may-b big-news-robotics

[2] https://developer3.oculus.com/blog/the-latent-power-of-prediction/

[3] https://developer3.oculus.com/blog/optimizing-vr-graphics-with-late-latching/

[4] Pelot: "Dynamic Blanking for Virtual Reality Image Displacement",

http://web.stanford.edu/class/ee267/Spring2016/report_pelot.pdf

Claims

claims

1 . A method of controlling a display of an augmented reality head-up display device (41) for a motor vehicle (40), comprising the steps of:

- generating (1 1) a marker (60) for display by the augmented reality head-up display device (41); and

- outputting (12) the marker (60) for display by the augmented reality head-up display device (41);

characterized in that the mark (60) has a dynamic design.

2. The method according to claim 1, wherein the marker (60) is a contact analog

Mark is.

3. The method according to claim 1 or 2, wherein the dynamic embodiment of

Marking (60) comprises a design as a running light or a pulsation of the marker (60).

4. The method according to claim 3, wherein a running direction of the running light corresponds to a direction of a jamming movement of the motor vehicle (40).

5. The method according to any one of the preceding claims, wherein the mark (60) only has a dynamic design, when jamming movements of the

Motor vehicle (40) are detected (10).

6. The method of claim 5, wherein the marker (60) only has a dynamic configuration when the jamming movements of the motor vehicle (40) exceed a threshold value.

7. The method according to any one of claims 4 to 6, wherein the jamming movements of the

Motor vehicle (40) pitching movements are.

8. The method according to any one of claims 4 to 7, wherein the jamming movements of the

Motor vehicle (40) are detected with an acceleration sensor or derived from a road condition.

9. The method according to claim 8, wherein the roadway condition by means of a

Laser scanner or a stereo camera is detected or removed from maps.

A computer readable storage medium having instructions, when executed by a computer, for executing the steps of a method according to any one of claims 1 to 9 for controlling a display of an augmented reality head-up display device (41) for a motor vehicle (40).

1 1. A device (20) for controlling a display of an augmented reality head-up display device (41) for a motor vehicle (40), with

- a graphics generator (23) for generating (1 1) a marker (60) for display by the augmented reality head-up display device (41); and

- an output (26) for outputting (12) the marker (60) for display by the augmented reality head-up display device (41);

characterized in that the graphics generator (23) is adapted to generate the marker (60) with a dynamic design (1 1).

12. Device (20) according to claim 1 1, wherein the graphics generator (23) is adapted to generate the marker (60) only with a dynamic design (1 1), if jamming movements of the motor vehicle (40) are detected (10) ,

13. motor vehicle (40) with an augmented reality head-up display device (41), characterized in that the motor vehicle (40) comprises a device (20) according to claim 1 1 or 12 or is arranged, a method according to one of claims 1 to 9 for controlling a display of the augmented reality head-up display device (41).