DE102019133948A1 - Method for driver assistance for a combination of a motor vehicle and a trailer - Google Patents

Abstract

A method for driver assistance comprises the following: providing a model of a surface (20) as an estimate of a surface that is exposed by an environment surrounding a vehicle (10), providing a model for the vehicle on the surface (20) and providing a model for a trailer on the surface (20) at an angle to the model for the vehicle, the models for the vehicle and trailer including the location and posture of multiple cameras. For each particular angular orientation of the vehicle (10) and the trailer (16, 16 '), one or more cameras are defined to potentially provide image information for display for each of several areas of the surface (20) which represent the models of the vehicle and trailer surround. A point of view (POV) for a virtual camera defines a display window (22); and each pixel of the display window (22) is displayed as a function of image information from the camera (s) defined for an area of the surface including a projection of the pixel location of the display window onto the surface (20).

Description

Field of invention

The present invention relates to a method for driver assistance for a combination of a motor vehicle and a trailer.

Background of the invention

Now referring to 1 , it is known that vehicles, such as a car 10 , have a number of cameras, which are arranged on the front (FV), on the rear (RV) and on the left and right exterior mirrors (ML, MR) of the vehicle in order to record images of the surroundings surrounding the vehicle. (It should be noted that the side cameras do not necessarily have to be located on the exterior mirrors, and these can be located in any location suitable for capturing an image of the surroundings on the sides of a vehicle). Images from these cameras are typically transmitted to an electronic control unit (ECU) that includes a processor that processes the images before sending one or more processed images to a display 12th which is located in a vehicle interior and is visible to a driver in order to provide assistance to the driver of the vehicle. Further uses of such camera information include autonomous or semi-autonomous driving, parking or braking of the vehicle and, for example, storage of image sequences that were recorded by one or more of the cameras FV, MR, ML, RV as dashcam or safety recordings for later retrieval.

The German patent application no. 102019131971.4 (Ref .: 2018PF02667) under the Title "An image processing module", submitted November 26, 2019 , and the German patent application no. DE102019126814.1 (Ref .: 2019PF00721) under the title "An electronic control unit", filed on October 7, 2019 , provide examples of electronic control units that provide an all-round view of a vehicle in order to assist a driver in particular when parking a vehicle.

Such vehicles 10 can also be used with towbars 14th be provided that allow them to place objects such as a pendant 16 to pull. In the present application, any object pulled by a vehicle is referred to as a trailer. It is understood that trailers lead to large blind spots for a driver as he is unable to see most of the areas around the trailer and it can be difficult, especially for inexperienced drivers, to attempt to drive a vehicle with one without assistance Reset trailer.

It is therefore known that such trailers also have a rear view camera (TV) from the trailer 16 from is directed to the rear, and in some cases side cameras of the trailer (TL, TR), which point outwards from the respective sides of the trailer, can be integrated. Images from these cameras can be captured by an ECU-T in the trailer and either wired or wirelessly connected to the vehicle 10 transmitted or otherwise provided where they can be used, for example, by the vehicle ECU to generate an enhanced display for assisting a driver.

US9908558 discloses, for example, a method for steering a car / trailer combination in which a mobile display unit is used to display at least a portion of the car / trailer combination and wherein the mobile display unit is in contact with the vehicle via a wireless connection and is used to monitor the driving functions of the vehicle and wherein at least one orientation aid is shown on the mobile display unit.

EP3342645 (Ref .: SIE0455) discloses a method for providing information from the surrounding area of a motor vehicle for a driver, at least one first image being recorded by a vehicle-side camera and at least one second image being recorded by a non-vehicle camera that is in the field of view of the vehicle-side Camera lies. A picture-in-picture representation is generated from the first image and the second image in order to be displayed on a vehicle-mounted display device. A first image area for displaying the first image and a second image area for displaying the second image, which is separated from the first image area, are determined for the picture-in-picture display.

WO2016 / 026870 and DE102019122086 (Ref .: 2019P00307) under the title “Driver assistance for a combination”, filed on August 16, 2019, disclose generating a first camera image from a camera at the rear of a vehicle and a second camera image from a camera at the rear of a trailer connected to the vehicle is coupled. A combined image is generated by fading in the camera images in such a way that the second camera image covers a section of the first camera image depending on a coupling angle of the combination.

US 2019/0176698 (Ref .: SIE 411) discloses a method for assisting a driver of a motor vehicle when maneuvering the motor vehicle with a trailer, with image data from an area surrounding the motor vehicle with the aid of at least one vehicle-mounted camera and with With the help of at least one trailer-side camera and an image of the surrounding area is generated to be displayed on a vehicle-side display device depending on the recorded image data, a viewpoint from which the surrounding area is displayed in the image depending on a Pivot angle between the trailer and the motor vehicle is determined.

The above approaches may still tend to neglect blind spots and it is thus an object of the present invention to provide a driver of the vehicle with an improved view of an environment surrounding a vehicle and trailer combination.

Summary

According to the present invention, a method for driver assistance for a combination of a motor vehicle and a trailer according to claim 1 is provided.

Further aspects include a driver assistance system, a motor vehicle with driver assistance and a computer program product that is configured to carry out the method according to the invention.

Embodiments use multiple cameras from the vehicle and trailer to create a three-dimensional view of an environment surrounding a vehicle for display to a driver.

The driver can select a location and an angular orientation for a virtual camera and thereby define the point of view for the three-dimensional view of the surroundings by allowing them to be seen around the trailer from several different views.

At least two of the cameras have overlapping fields of view and embodiments combine image information from combinations of the cameras on the exterior mirrors of the vehicle with the cameras on the rear of the vehicle and on the rear of the trailer and, if available, cameras on the sides of the trailer in merged areas.

Embodiments are responsive to a change in the angle between the trailer and the vehicle to change the configuration of merged areas to continue to provide a three-dimensional view of the environment surrounding the vehicle as the angle changes.

In an image in a display window provided to the driver, at least the trailer can be displayed in a transparent or semi-transparent manner to allow the driver, after selecting a viewpoint on one side of the combination, to the opposite side of the trailer to see through. In some embodiments, the level of transparency of the trailer can be adjusted by the driver.

In some embodiments, the trailer can be equipped with both a rear view camera and side cameras to limit the occurrence of blind spots within the three-dimensional view.

Figure list

• 1 eine schematische Ansicht eines Fahrzeugs, das einen Anhänger zieht, gemäß einer Ausführungsform der vorliegenden Erfindung ist;
• 2 eine virtuelle Umgebung darstellt, die Polygonnetzmodelle eines Fahrzeugs und eines Anhängers beinhaltet;
• 3 schematisch Bereiche um ein Fahrzeug herum darstellt, die gemäß den Kameras festgelegt sind, von denen Bildinformationen abgeleitet werden, um eine Rundumansicht des Fahrzeugs zu erzeugen;
• 4 schematisch Bereiche um ein Fahrzeug und einen Anhänger herum in kollinearer Ausrichtung darstellt, die gemäß den Kameras festgelegt sind, von denen Bildinformationen abgeleitet werden, um eine Rundumansicht der Fahrzeug/Anhänger-Kombination gemäß einer ersten Ausführungsform der vorliegenden Erfindung zu erzeugen;
• 5 schematisch Bereiche um das Fahrzeug und den Anhänger herum in Winkelausrichtung darstellt, die gemäß den Kameras festgelegt sind, von denen Bildinformationen abgeleitet werden, um eine Rundumansicht der Fahrzeug/Anhänger-Kombination gemäß der ersten Ausführungsform der vorliegenden Erfindung zu erzeugen;
• 6 schematisch Bereiche um das Fahrzeug und den Anhänger herum darstellt, sowohl in kollinearer als auch in Winkelausrichtung, die gemäß den Kameras festgelegt sind, von denen Bildinformationen abgeleitet werden, um eine Rundumansicht der Fahrzeug/Anhänger-Kombination gemäß einer zweiten Ausführungsform der vorliegenden Erfindung zu erzeugen;
• 7(a) die Oberflächen einer Umgebung, die mit Kamerabildinformationen gemäß einer Ausführungsform der vorliegenden Erfindung wiederzugeben ist, darstellt; und
• 7(b) ein Anzeigefenster der wiedergegebenen Umgebung von 7(a) darstellt.

Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

• 1 Figure 13 is a schematic view of a vehicle towing a trailer in accordance with an embodiment of the present invention;
• 2 represents a virtual environment that includes mesh models of a vehicle and trailer;
• 3 schematically depicts areas around a vehicle defined in accordance with the cameras from which image information is derived to generate an all-round view of the vehicle;
• 4th schematically depicts areas around a vehicle and a trailer in collinear orientation defined in accordance with the cameras from which image information is derived to generate an all-round view of the vehicle / trailer combination according to a first embodiment of the present invention;
• 5 schematically depicts areas around the vehicle and trailer in angular orientation determined in accordance with the cameras from which image information is derived to generate a 360 degree view of the vehicle / trailer combination according to the first embodiment of the present invention;
• 6th schematically depicts areas around the vehicle and trailer, in both collinear and angular orientations, determined according to the cameras from which image information is derived to generate a 360 degree view of the vehicle / trailer combination in accordance with a second embodiment of the present invention ;
• 7 (a) illustrates the surfaces of an environment to be rendered with camera image information according to an embodiment of the present invention; and
• 7 (b) a display window of the reproduced environment of 7 (a) represents.

Detailed description of the embodiments

Now referring to 2 , Embodiments of the present invention provide a display image of a three-dimensional environment to a driver of a vehicle 10 when pulling a trailer 16 to assist.

In the illustrated embodiment, the environment is defined by three main objects: a bowl 20th , a model of the vehicle 10 and a model of the dragged object 16 , which is referred to here as a trailer.

In the embodiment, a bowl shape is chosen as an approximation of the shape of the surface on which the vehicle is located 10 surrounding environment is exposed, wherein the bowl has a flat bottom and a generally cylindrical side wall. This means that images of the assumed generally flat surface over which the vehicle / trailer combination is traveling can be projected onto the flat ground with minimal distortion, whereas images of other three-dimensional objects surrounding the vehicle / trailer combination can be tend to be less distorted when mapped onto the surface of the side wall of the bowl. It is understood, however, that variants of the bowl shape of 2 are possible and that, for example, in a simple implementation or where appropriate, a completely flat surface surrounding the vehicle / trailer combination could be assumed; or that the bowl shape could take on a continuous shape that curves upward around the vehicle / trailer combination; or that, if more precise information is available about the surface exposed by the environment surrounding the vehicle / trailer combination, for example from LIDAR sensors or a depth map constructed from optical image information, and that this surface information can then be used to determine a more complex shape for the bowl.

In each case, a three-dimensional bowl-shaped mesh is constructed, which is respective three-dimensional mesh models of the vehicle 10 and the trailer 16 contains.

The scale of the bowl 20th is determined by the resolution and the field of view of the vehicle 10 and the trailer 16 attached cameras - it certainly doesn't make sense the size of the bowl 20th set so large that image information from the on the vehicle 10 and the trailer 16 attached cameras, if imaged on the surface of the bowl, would either be unduly distorted or provide poor resolution for the display image.

The mesh model of the vehicle 10 represents the vehicle in the vicinity of the bowl and is by default placed in the direction of the center of the bowl. Of course, all information about the size and shape of the vehicle is available from the manufacturer, including the location, relative orientation and projection model of the various cameras located around the vehicle.

It will be understood that the distance and the orientation of the cameras relative to the surface on which the vehicle is located may change according to the load on the vehicle or in response to a body roll of the vehicle or in response to active control of the vehicle suspension, and the effect on changes in vehicle attitude, although not discussed in further detail here, can be taken into account when adjusting the mesh model of the vehicle, as described in German Patent Application No. 102019114404.3 (Ref .: 2018PF02113 (SIE0883)) under the Title "Image acquisition system", submitted on May 29, 2019 , is revealed.

That being said, the mesh model of the vehicle 10 Changed when in use so that certain parts of the vehicle, such as wheels, doors or a sunroof, can be animated, or the color and texture of certain parts, such as direction indicators or lights, can be updated in real time in response to vehicle sensor / actuator information, the fidelity of the vehicle's polygon mesh model 10 and maintain its correspondence with the actual condition of the vehicle.

Vehicle software, either an interactive application running directly on a vehicle console or on a device that communicates with the vehicle, may allow the user to change the color of the vehicle, or in some embodiments, the user may be allowed to change the transparency of the Vehicle so that the display image may be able to show camera information from both sides of the vehicle with certain views of the surroundings.

The mesh model of the trailer 16 put that behind the vehicle 10 again, the mesh model of the trailer 16 be available from the manufacturer of the trailer and this can be added to the vehicle 10 can be provided through any number of mechanisms. So the information for the mesh model of the trailer 16 for example stored in the trailer and attached to the vehicle 10 be transferred when the vehicle 10 and the trailer 16 either coupled together or simply in close proximity to each other. Alternatively, the trailer could 16 the vehicle 10 simply reveal his identity and type, so that the vehicle 10 can call up the required data for the trailer via a network connection to a data source. Alternatively, the user may have pre-stored the mesh model information for the trailer in the vehicle to avoid having to rely on a network connection and then access it when the trailer 16 to the vehicle 10 is coupled.

If predefined mesh model information is not available for the trailer, for example, if a driver has his own camera installed on a trailer, vehicle software can allow the driver to enter his own dimensions for the trailer along with the details of the camera.

Additional techniques for generating the mesh model for the trailer 16 could employ machine vision using either separate and / or vehicle cameras or scanning techniques.

1. 1) Bereitstellen einer interaktiven Anwendung, die entweder unmittelbar in einer Fahrzeugkonsole oder auf einer Vorrichtung läuft, die mit dem Fahrzeug kommuniziert, um dem Benutzer zu gestatten, Polygonnetzmodell-Informationen des Anhängers einzugeben;
2. 2) Scannen des gezogenen Objekts und Hochladen der gescannten Informationen entweder unmittelbar in das Fahrzeug oder in eine Vorrichtung, die mit dem Fahrzeug kommuniziert;
3. 3) Erlangen von Daten über eine Verbindung mit dem gezogenen Fahrzeug;
4. 4) Herunterladen von Polygonnetzmodell-Informationen des Anhängers über eine Netzverbindung; oder
5. 5) Verwenden von Bildinformationen, die von jeglichen zu dem Anhänger weisenden Fahrzeugkameras erfasst wurden.

Thus, any combination of the following techniques can be used to create the mesh model of the trailer 16 for the vehicle 10 to make available:

1. 1) providing an interactive application running either directly on a vehicle console or on a device that communicates with the vehicle to allow the user to enter mesh model information of the trailer;
2. 2) scanning the dragged object and uploading the scanned information either directly to the vehicle or to a device that communicates with the vehicle;
3. 3) obtaining data via a connection with the towed vehicle;
4. 4) downloading mesh model information of the trailer over a network connection; or
5. 5) Using image information captured by any vehicle cameras facing the trailer.

In either case, vehicle software can allow the shape of the model, color, and details of the mesh model of the trailer to be changed by the driver as needed, again either as an interactive application running directly on a vehicle console or on a device that works with communicated to the vehicle.

Finally, the driver can also specify the transparency of the trailer, so that the display image in certain views of the surroundings can be able to show camera information from both sides of the vehicle / trailer combination.

In any case, it can detect that a trailer 16 to the vehicle 10 hitched can be done in a number of ways including via: manual user input; Signaling via a wired or wireless connection between the vehicle 10 and the trailer 16 was produced; automatic direct or indirect detection of a connection on the trailer coupling of the vehicle; and / or via an image analysis, in particular using the camera located at the rear of the vehicle.

Once a follower 16 is detected as being connected to the vehicle, the trailer polygon network can be behind the vehicle polygon network in the three-dimensional environment 10 be placed as in 2 shown. The exact location of the vehicle / trailer combination can be set to be centered on the environment by default, or this can be adjusted either manually or algorithmically as needed.

Similarly, it is possible for the shape and size of the bowl 20th Can be customized to fit both the vehicle 10 and the trailer 16 inside the bowl 20th to fit in. For example, for longer trailers, the bowl could be stretched to have an elliptical rather than circular cutout with a major axis parallel to the longitudinal axis of the vehicle. Alternatively, the side wall of the bowl can be more open so that it has a conical rather than cylindrical shape.

How out 2 visible are the vehicle 10 and the dragged object 16 around the trailer hitch 14th articulated so that there may be an angle between the longitudinal axes of the vehicle and the trailer. In the embodiment described below, it is assumed that this angle is a simple one and indicates an angle of rotation about a vertical axis, but it should be understood that embodiments represent a compound angle and possibly a changing compound angle between the vehicle 10 and the trailer 16 may take into account.

This angle can be detected in any number of ways, such as using image information from a rear-view camera (RV) that images the forward facing surface of the trailer, about the rotation of the trailer about the vertical axis passing through the hitch , to be detected as the pictured features of the trailer moving sideways in the field of view of the camera. Likewise, image information from the cameras on the vehicle's exterior mirrors (ML, MR) can detect features from the surface of the trailer moving laterally within their respective fields of view to estimate the relative angle of the vehicle and trailer. Other techniques for determining the relative angle of the vehicle and trailer include using information from rear-facing ultrasonic or radar sensors (not shown) located at the rear of the vehicle ( 10 ) are attached, with changing differences measured by the sensors signaling changes in the relative angle of the vehicle and trailer. Still other techniques may involve more direct or dedicated gauges that are between the vehicle 10 and the trailer 16 are arranged.

In any case, as soon as the relative angle of the vehicle and trailer is known, this can be done in the arrangement of the respective objects in the three-dimensional environment, as in 2 shown.

In embodiments of the invention, vehicle software can allow the driver to interact with a three-dimensional view of the environment (as detailed in FIG 7 (b) shown) to freely define a point of view (POV) for a virtual camera in the environment. In some cases, the user may be able to set the camera to any of the six degrees of freedom X, Y, Z, Rx, Ry, Rz in the environment as well as in the field of view (FOV) of the camera - defining them a display window 22nd , which is finally played back to provide the driver of the vehicle with their view of the three-dimensional environment surrounding the vehicle.

In other implementations, the location and orientation of the POV as well as the FOV of the virtual camera can be automatically selected and adjusted according to the state of the vehicle / trailer combination or the environment around the combination. So the POV can, for example, when the vehicle 10 relative to the trailer turns left, be shifted to the left of the combination, as in 2 shown. On the other hand, if the driver of such a combination were sitting on the left side of the vehicle and reversing the combination and an object was detected near the right side of the combination, the POV could be rotated around the combination to the right side of the vehicle for better visibility of the object. Indeed, in some variations of the described embodiment, a mesh model of any such detected object could be incorporated into the bowl environment and used to improve the quality of the rendered display window.

In response to sensing that the relative angle of the vehicle and trailer has become too great and the trailer is beginning to stand sideways, the POV could also be changed to a higher position in the area, possibly even immediately above the combination, to provide this information more clearly to the driver.

Now that the environment 20th , their objects 10 , 16 , whose relative orientation and POV and FOV of a virtual camera in the area have been determined, proceeding to describe how the bowl surface can be rendered before the display window 22nd is generated for the driver.

Now referring to 3 , instructs the vehicle 10 in one embodiment four cameras FV, ML, MR, RV, as in FIG 1 described. Each of these cameras has a wide field of view and this can for example be up to 180 ° in the diagonal and beyond. This means that the fields of view of neighboring cameras overlap around the body of the vehicle, as indicated by the areas FV + MR, MR + RV, FV + ML and ML + RV in 3 specified. (It should be noted that the boundaries for these areas are only shown schematically and not necessarily the actual geometry of the overlapping areas on the surface of the bowl 20th reflect.)

As in the above-mentioned German patent application no. 102019131971.4 (Ref .: 2018PF02667), it is known that the camera information from neighboring cameras that cover this area is merged in these overlapping areas. So can based on the assumption that the surface of the bowl 20th approximately corresponds to the shape of the surface of the vehicle 10 surrounding environment is exposed, and knowing the relative positions and orientations of the neighboring cameras relative to the bowl surface, image information from the vehicle cameras on the surface of the bowl 20th and image information from neighboring cameras can be merged in the overlapping areas. Thus, areas in which image information from more than one camera is combined become the surface of the bowl 20th are referred to as "merged areas". A relatively simple merge function involves blending image information from each adjacent camera across the merged areas based on a simple function such as an alpha function, where the function image information from the closest of the adjacent cameras includes the image information for the merged area for any given location of the merged area is weighted more heavily. Further merging functions can use techniques for stitching together images, for example based on a feature analysis, in order to take into account image semantics when generating merged image information. As in the German patent application no. 102019131971.4 (Ref .: 2018PF02667), even more sophisticated approaches to mixing or stitching based on generative neural networks can be used to remove artifacts resulting from mismatches between the actual surface of the environment and that of the surface of the bowl 20th provided estimate to weaken.

Embodiments of the present invention deal with a trailer 16 with one or more cameras that monitor the area around the vehicle 10 to be added.

Now referring to 4th , includes the trailer 16 in this example a rear-facing camera TV that is on a rear surface of the trailer 16 is arranged.

As explained above, image information generated by the rear view camera of the trailer TV becomes the vehicle 10 for use in generating the view of the environment for display to the driver of the vehicle.

In a first configuration in which the vehicle 10 and the trailer 16 are collinear, a number of areas are defined around the periphery of the vehicle / trailer combination. In this case, the merged areas that are in 3 generated from a combination of vehicle exterior mirrors (ML, MR) and rear (RV), cropped to cover the camera at the rear of the vehicle RV by the trailer 16 to consider.

Additional areas are added to this modification of the merged areas MR + RV and ML + RV. These areas include an area that is completely within the field of view of the camera on the rear of the trailer TV. The merged areas ML + TV and MR + TV are provided on both sides of this area, where the fields of view of the camera on the trailer rear TV and the cameras on the vehicle exterior mirrors ML, MR overlap. Now it goes without saying that these areas are usually further away from the cameras on the vehicle exterior mirrors ML, MR than the camera on the rear of the trailer TV and thus each merging of the image information from the respective cameras of these areas is done by corresponding weighting of image information from the Camera at the rear of the trailer TV can take into account. Indeed, in some cases, as will be seen below, it may be preferred not to merge image information from the vehicle cameras and the trailer cameras, especially when their relative positions (particularly about the horizontal axis) are not constant or determined.

However, between the merged areas ML + TV and MR + TV and the trimmed merged areas MR + RV and ML + RV are defined areas in which image information is derived solely from the cameras on the vehicle exterior mirrors ML, MR in order to avoid obscuring problems the camera on the rear of the vehicle RV through the trailer 16 despite the distance between these areas and the camera locations.

It should be noted that the delimitation of the various areas in 4th is shown schematically and with straight lines for the sake of simplicity. It is therefore understood that the area directly at the rear of the vehicle in each of the areas MR + RV and ML + RV is not visible to the cameras on the vehicle exterior mirrors ML, MR and these therefore cannot contribute any image information in this area. Therefore, this zone, which is immediately between the vehicle 10 and the trailer 16 is either ignored, taking into account that the road surface in this zone is very likely completely covered by the trailer coupling in any case, or another area that uses image information obtained from the camera at the rear of the vehicle RV can only be used for this section of the surface of the bowl 20th should be defined if this would be considered appropriate.

Now referring to 5 , of course, when the articulation angle of vehicle 10 and trailers 16 changes - in this case because the vehicle 10 has turned right - therefore not only does the position of the camera on the rear of the trailer TV change relative to the remaining cameras of the vehicle and thus the projection of the image information from the camera on the rear of the trailer TV onto the surface of the bowl 20th changes accordingly, but also because some vehicle cameras can now see areas of the environment that were previously not visible, while other vehicle cameras see less the configuration of the areas surrounding the vehicle / trailer combination in terms of number, size and shape changes.

Thus shows 5 that if the articulation angle is large enough, image information from the camera on the trailer rear TV can no longer be merged with information from other cameras, and that this is solely for reproducing the surface of the bowl 20th over the entire field of view of the camera at the rear of the trailer TV, based solely on this camera image information.

While part of this area can be visible from the camera on the vehicle exterior mirror MR, image information from the camera on the trailer rear TV in this area can be of such a higher resolution than is available from the camera on the vehicle exterior mirror MR, so that it is preferred to only display this image information use in place of information mixed by two cameras. Not using the merge for this area also reduces the total amount of processing required to extract display window information from such a portion of the surface of the bowl 20th to generate, but also due to possible angular differences around a horizontal axis between the vehicle and the trailer, failure to merge image information from the vehicle cameras with the trailer cameras prevents artifacts from merging image information from two cameras, with their relative positions not being determined or constant for example, when the vehicle / trailer combination is traveling over a curved surface.

In any case, the merged area of MR + RV is now even more cropped than when the vehicle was 10 and the trailer 16 would be collinear, whereas the merged area ML + RV on the other hand can remain the same size - with the note as before that the zone immediately behind the vehicle is not visible to the cameras on the vehicle exterior mirrors ML, MR.

A new area RV based solely on image information from the camera at the rear of the vehicle RV that is not obstructed by the trailer is provided towards the left side of the combination. This range increases in the angular range while the vehicle continues to be relative to the trailer 16 turns right.

It will also be understood that if only one camera is provided on the trailer rear TV, areas around the sides of the trailer will also be obscured by both the camera on the vehicle rear RV and by one or both of the cameras on the vehicle exterior mirrors ML, MR. These zones are defined as blind spots and there is no reproduction of the surface of the bowl in these areas 20th performed with camera image information.

In particular, it can be seen that, while the area that is based solely on image information from the camera at the rear of the vehicle RV, increases in angle while the vehicle is moving 10 relative to the trailer 16 turns right to cover an area that is obscured by both the camera on the rear trailer TV and the camera on the vehicle wing mirror ML, and the size of the dead area on the left side of the trailer increases.

In the example shown, an area to the right of the trailer and within what the merged area MR + RV is when the vehicle and trailer were collinear is determined to be dead. Even if this could actually be visible to the camera on the vehicle exterior mirror MR - if the image resolution, which is provided for this area by the camera on the vehicle exterior mirror MR alone, were so low or distorted that it is unsatisfactory, such an area can be considered dead be determined.

In this regard, it is understood that until the vehicle 10 relative to the trailer 16 turns right to the extent that the trailer extends over the merged area MR + RV when the vehicle and trailer were collinear, the in 4th The area designated by MR between the trailer and the merged areas MR + RV, although it is increasingly cropped, can be defined as an area from which image information is obtained only from the vehicle camera MR.

With regard 6th it will now be seen that if a follower 16 ' each equipped with left and right side cameras TL, TR, then at least for areas of the surface of the bowl 20th who otherwise would have been determined to be dead can now be determined to be theirs Derive image information solely from the cameras on the sides of the trailer TL, TR.

In the example it can be seen that due to the increased resolution available from the cameras on the sides of the trailer TL, TR relative to the cameras on the vehicle exterior mirrors ML, MR, for the areas on the side of the trailer 16 , for which in the example of 4th it was determined that they derive their information from the cameras on the vehicle exterior mirrors ML, MR, it can now be determined that they derive their image information only from the cameras on the sides of the trailer TL, TR.

It should be noted that this means that when the vehicle 10 and the trailer 16 ' are collinear, image information from the cameras on the sides of the trailer TL, TR to the vehicle 10 must be fed, which increases the data traffic between the two. When the vehicle 10 and the trailer 16 ' are collinear, image information for these areas can thus continue to be derived from the cameras on the vehicle exterior mirrors ML, MR in some embodiments.

It will also be seen that it depends on the length of the trailer 16 ' It may be preferred not to provide merged areas towards the edge of the field of view of the camera on the trailer rear TV in order to mix their image information with respective cameras on the sides of the trailer TL, TR to both reduce the total amount of processing that is required to the surface of the bowl 20th and since, depending on the positioning of the cameras on the sides of the trailer TL, TR, the quality and resolution in these areas of the cameras on the sides of the trailer TL, TR may not be satisfactory.

In the example of 6th , at least if the vehicle 10 and the trailer 16 ' are collinear, the merged areas TL + TV and TR + TV at the edge of the field of view of the camera on the trailer rear TV set in order to mix their image information with that of the cameras on the sides of the trailer TL, TR. This can continue until the angle between the vehicle 10 and the trailer 16 ' exceeds a threshold after which these merged regions will not be applied.

It should be noted that when image information from one of the trailer cameras TL, TR, TV is mixed with that of another camera, any change in the position of the trailer on the surface of the bowl must be taken into account, for example in response to a change in the load , and that techniques such as those in the above-mentioned German patent application no. 102019114404.3 (Ref .: 2018PF02113 (SIE0883)) can be used to adjust the mesh model of the trailer in relation to the surface of the bowl, in particular to ensure that merged areas based on image information from both neighboring trailer cameras and from trailer and vehicle -Cameras based, are less prone to artifacts.

In any case, when defining the one or more cameras from which image information is to be derived, it is possible to reproduce different areas around the trailer / vehicle combination while the vehicle is at its angle in relation to the trailer 16 , 16 ' changes, taking into account the pixel quality / resolution of any given camera, the distance from the camera to the area, and the processing required to both acquire image information and generate image information.

While the description of the examples of the 5 and 6th in the event that the vehicle was provided 10 based on the trailer 16 , 16 ' turns right, it is understood that the areas in the event that the vehicle 10 based on the trailer 16 , 16 ' turns left, can be mirrored.

Nevertheless, in some embodiments it must be taken into account whether the vehicle 10 is designed for left-hand drive or right-hand drive and / or whether the vehicle is being driven in a country with left-hand traffic or in a country with right-hand traffic, so that the areas that have been defined around the vehicle / trailer combination are in response to whether the Vehicle turns left or right do not have to be symmetrical.

From the foregoing description it will be understood how image information can be derived from the various cameras located around the vehicle / trailer combination, around any given portion of the surface of the bowl 20th according to the relative angle of the vehicle 10 and trailers 16 , 16 ' and that this is independent of the POV at any given point in time.

It will now be explained how for any given POV and display window 22nd selected either through user interaction or algorithmically in response to sensor inputs or otherwise, image information from the visible surface of the bowl 20th at the respective positions in the display window 22nd can be mapped.

Referring to 7 (a) , can do that for the vehicle 10 and / or the trailer 16 , 16 ' selected transparency, as mentioned above, should be taken into account. In this example, it has been selected that the vehicle is opaque, and so it is only image information from the mesh model of the vehicle that is used to generate the displayed image in this section of the display window.

On the other hand, it was chosen that the tag should be substantially transparent and so image information in this portion of the display window comprises a blended combination of image information from the surface of the bowl 20th and the mesh model of the trailer.

It is therefore understood that a final displayed display window 22nd , as in 7 (b) shown can be generated with this available information. It is noted that in this display window, while the POV is toward the left side of the vehicle, the surface of the surroundings toward the right side of the vehicle through the trailer 16 is visible.

It should be noted that the upper right-hand side of the display is black as it is beyond the field of view of one of the cameras positioned towards the left-hand side of the vehicle / trailer combination.

It is also noted that the quality of the image varies from area to area around the vehicle / trailer combination in terms of texture, since image information is obtained from different cameras and combinations of cameras. It can be seen that in this case the portion of the display window immediately to the right of the trailer is derived from image information from a camera on a vehicle exterior mirror MR and tends to be of lower resolution / quality than other zones of the display window.

While the above description has been provided in terms of how camera information is on the surface of the bowl 20th mapped and like information from the surface of the bowl 20th as well as the mesh models of the vehicle and the trailer in the display window 22nd It will be understood that in practice it is more likely that the display window information will be generated in reverse by knowing how any given pixel of the display window will map onto the surface of the bowl and thus the combination and location of camera image information that need to be captured, interpolated, and possibly blended and possibly combined with vehicle and / or mesh model information to produce a pixel value.

However, it is understood how using various inputs, such as mechanical information about the vehicle and trailer as well as sensor information, the location and relative position of the vehicle and the trailer in the key environment can be continuously updated, and with knowledge of the relative angle of the vehicle and pendant both shape the bowl 20th and the set of areas and merged areas around the vehicle / trailer combination can be retrieved and updated to a display window 22nd for every POV in the bowl 20th easy to provide.

It will be understood that embodiments of the invention rely on obtaining image information from multiple cameras located around a vehicle / trailer combination and processing it before ultimately presenting a display window to a driver 22nd provided. In some embodiments, vehicle software running either on a vehicle console or on a device that communicates with the vehicle may allow a driver to specify features of the vehicle, the trailer, and the environment to be used for the display, but in either case can Processing required to input information can be performed anywhere from the ECU, which initially acquires image information from the various cameras, to other image processing modules that communicate with the ECU via a vehicle bus such as a CAN or FlexRay bus can be connected, up to a stand-alone display module that is closely related to the display 12th is coupled.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 102019131971 [0003, 0046]
• DE 102019126814 [0003]
• US 9908558 B [0006]
• EP 3342645 A [0007]
• WO 2016/026870 [0008]
• DE 102019122086 [0008]
• US 20190176698 A [0009]
• DE 102019114404 [0026, 0068]

Non-patent literature cited

• Title "An image processing module", filed on November 26, 2019 [0003]
• "An electronic control unit", filed on October 7, 2019 [0003]
• Title "Image acquisition system", submitted on May 29, 2019 [0026]

Claims (15)

A method for driver assistance for a combination of a motor vehicle (10) and a trailer (16, 16 '), comprising the following steps: Providing a model of a surface (20) as an estimate of a surface exposed from an environment surrounding the vehicle (10); Providing a model for the vehicle on the surface (20), the vehicle model including at least the location and position of a plurality of vehicle cameras (FV, ML, MR, RV); Determining an angular orientation about a vertical axis of the vehicle (10) and the trailer (16, 16 '); Providing a model for the trailer on the surface (20) at the specific angle to the model for the vehicle, the model of the trailer including the location and position of at least one trailer camera (TV, TL, TR) relative to the vehicle cameras; for each particular angular orientation of the vehicle (10) and the trailer (16, 16 '), setting, for each of several areas of the surface (20) which surrounds the models of the vehicle and the trailer, of one or more of the vehicle or Trailer cameras that are potentially intended to provide image information for display; Determining a point of view (POV) for a virtual camera in the vicinity to define a display window (22); Rendering each pixel of the display window (22) as a function of image information from the one or more cameras set for an area of the surface, including a projection of the pixel location of the display window onto the surface (20) and the projection of the models of the Vehicle and trailer in the display window (22); and Providing the display window information to a vehicle display (12) to assist the driver. Procedure according to Claim 1 wherein the trailer includes a rearward facing camera disposed on a rear surface of the trailer. Procedure according to Claim 2 wherein when the vehicle and trailer are in collinear alignment, the plurality of areas include: a first area to the right of the vehicle from which image information is provided only from a camera located on the right side of the vehicle; a second area to the right of the trailer, of which image information is only provided by the camera located on the right side of the vehicle; a third area between the first and second areas from which image information is provided from the camera located on the right side of the vehicle and a camera located on the rear of the vehicle; a fourth area on the left of the vehicle, of which image information is provided only by a camera located on the left side of the vehicle; a fifth area to the left of the trailer, of which image information is provided only by the camera located on the left side of the vehicle; a sixth area between the fourth and fifth areas, of which image information is provided from the camera located on the left side of the vehicle and the camera located at the rear of the vehicle; and a seventh area from which image information is provided only by a camera at the rear of the trailer. Procedure according to Claim 3 wherein, when the vehicle and the trailer are in non-linear alignment, the multiple areas include: an eighth area from which image information is provided from a field of view of a camera at the rear of the vehicle that is not obscured by the trailer at the determined angle; wherein an angular range of one of the third or sixth ranges is clipped when the determined angle of the vehicle and trailer exceeds a given angle. Procedure according to Claim 2 wherein the trailer further includes left and right facing cameras disposed on opposite side surfaces of the trailer. Procedure according to Claim 5 wherein when the vehicle and trailer are in collinear alignment, the plurality of areas include: a first area to the right of the vehicle from which image information is provided only from a camera located on the right side of the vehicle; a second area to the right of the trailer, of which image information is only provided by the camera located on the right side of the vehicle; a third area between the first and second areas from which image information is provided from the camera located on the right side of the vehicle and a camera located on the rear of the vehicle; a fourth area on the left of the vehicle, of which image information is provided only by a camera located on the left side of the vehicle; a fifth area to the left of the trailer, of which image information is provided only by the camera located on the left side of the vehicle; a sixth area between the fourth and fifth areas, of which image information is provided from the camera located on the left side of the vehicle and the camera located at the rear of the vehicle; and a seventh area from which image information is provided only by a camera at the rear of the trailer. Procedure according to Claim 6 wherein, when the vehicle and the trailer are in non-linear alignment, the multiple areas include: an eighth area from which image information is provided from a field of view of a camera at the rear of the vehicle that is not obscured by the trailer at the determined angle; wherein an angular range of one of the third or sixth ranges is clipped when the determined angle of the vehicle and trailer exceeds a given angle. Procedure according to Claim 7 wherein the plurality of areas surrounding the vehicle are contiguous with one another. Procedure according to Claim 1 , wherein no area from the plurality of areas is defined as that it can potentially provide image information from both a vehicle camera and a trailer camera. Procedure according to Claim 1 , where the point of view (POV) for the virtual camera is user-defined via an interaction with application software. Procedure according to Claim 1 wherein rendering for a projection of a pixel location of the display window onto a location on the surface (20) in an area provided with image information from more than one camera, blending interpolated image information from the more than one camera according to the location of the projected one Includes pixels in the area. Driver assistance system for a combination comprising a motor vehicle and a trailer, wherein the driver assistance system is configured, the steps according to one of Claims 1 to 11 perform. Motor vehicle with a driver assistance system Claim 12 . Combination with a motor vehicle according to Claim 13 and a trailer. Computer program with commands that follow when the computer program is executed by a driver assistance system Claim 12 cause the driver assistance system to implement a method according to one of the Claims 1 to 11 perform.

Images