DE102018132649A1 - Method for calibrating a detection area of a camera system using an artificial neural network; Control unit; Driver assistance system and computer program product - Google Patents

Abstract

The invention relates to a method for enabling an improved calibration of a detection area of a camera system (12) of a motor vehicle (1) during intended operation of the motor vehicle (1), the camera system (12) comprising at least a first camera (FV) and a second camera ( ML), which have a common overlap area (21), with the steps: - providing a first and a second image (5), the first image (5) using the first camera (FV) and the second image (5) is recorded by means of the second camera (ML), - determining a height value for a surrounding area of the motor vehicle (1) shown in the overlap area (21) by evaluating the first and the second image (5) in their overlap area (21) taking into account a model (17) of the motor vehicle (1), the model having at least one target value for the relative position of the first and the second camera (FV, ML) relative to one another b - determining at least one calibration value by means of an artificial neural network (19) using the height value as an input variable, and - calibrating a detection range (20) of the camera system (12) based on the calibration value.

Description

The invention relates to a method for calibrating a camera system of a motor vehicle during intended operation of the motor vehicle. The invention also relates to a control unit for performing such a method and a driver assistance system for a motor vehicle. The invention also includes a computer program product for implementing the above-mentioned method and a computer-readable storage medium on which such a computer program product is stored.

The motor vehicle can be equipped with a camera system to inform a driver about the surroundings of a motor vehicle or to provide driver assistance functions by the motor vehicle. Such a camera system usually comprises at least two cameras, in many cases there are four cameras. In the case of four cameras, these can be arranged, for example, on the left and right of the motor vehicle, in particular on the mirrors, on the front and on the rear. Ideally, the cameras of the camera system provide a detection area around a large part of the motor vehicle, in particular around the entire motor vehicle. With the help of images from the camera system, objects in the surroundings can be recognized, a course of the carriageway of a carriageway can be determined or any other information in the surrounding area can be acquired. To enable detection, however, the camera system, in particular the individual cameras of the camera system, must be calibrated.

When calibrating the camera system, it can be determined which detection area the camera system or its individual cameras have. This detection range can largely depend on the installation position or pose of the respective cameras. In addition, a detection characteristic of the respective camera can have a great influence on the detection area. The detection characteristic can be influenced in particular by the lens used, for example a so-called fish eye lens or "fish eye" in English.

The calibration of a camera with a fisheye lens is from, for example CN 106 960 456 known.

The detection range of the camera system can be calibrated at the factory, for example by a manufacturer of the motor vehicle or the camera system. With such a calibration, the detection range of a respective camera can be related to a relative position in the surroundings of the motor vehicle in relation to the motor vehicle. For example, individual image areas and / or even individual pixels of the camera system are assigned a coordinate in a coordinate system aligned with the motor vehicle. In this way, the position of an object, a roadway or any other information relative to the motor vehicle, in particular based on the coordinate system of the motor vehicle, can be determined on the basis of the images of the camera system or the respective cameras.

It is obvious that the correct and precise detection of objects or lanes in the surroundings of the motor vehicle depends heavily on a correct calibration of the camera system or its detection area. By shifting the detection area, incorrect relative positions for objects or lanes in the surroundings of the motor vehicle can be determined. When using the camera system to inform the driver or to provide the driver assistance function, this can lead to an increased risk of collision for the motor vehicle. This applies in particular to driver assistance functions that enable the motor vehicle to drive autonomously or partially autonomously, for example parking systems. In order to curb collisions due to incorrect detection of objects or lanes, it is possible to increase the safety distance that the motor vehicle maintains during such an autonomous or partially autonomous journey. This in turn makes maneuvering more difficult.

The object of the present invention is to enable improved calibration of a detection area of a camera system.

This object is achieved according to the invention by the subject matter of the independent claims. Advantageous embodiments with appropriate developments are the subject of the dependent claims.

• - Bereitstellen eines ersten und eines zweiten Bildes, wobei das erste Bild mittels der ersten Kamera und das zweite Bild mittels der zweiten Kamera aufgenommen wird,
• - Bestimmen eines Höhenwertes für einen in dem Überlappungsbereich dargestellten Umgebungsbereich des Kraftfahrzeugs durch Auswerten des ersten und des zweiten Bildes in deren Überlappungsbereich unter Berücksichtigung eines Modells des Kraftfahrzeugs, wobei das Modell zumindest einen Soll-Wert für die Relativposition der ersten und der zweiten Kamera relativ zueinander bereitstellt,
• - Bestimmen zumindest eines Kalibrationswerts mittels eines künstlichen neuronalen Netzes unter Nutzung des Höhenwertes als Eingangsgröße, und
• - Kalibrieren eines Erfassungsbereichs des Kamerasystems basierend auf dem Kalibrationswert.

A first aspect of the invention relates to a method for calibrating a camera system of a motor vehicle during an intended operation of the motor vehicle, the camera system comprising at least a first camera and a second camera different from the first camera, and wherein the first and second cameras have a common overlap area, which is part of a respective detection area of both the first and the second camera. The process involves the following steps:

• Provision of a first and a second image, the first image being recorded by means of the first camera and the second image being recorded by means of the second camera,
• - Determining a height value for a surrounding area of the motor vehicle shown in the overlap area by evaluating the first and the second image in their overlap area, taking into account a model of the motor vehicle, the model providing at least one target value for the relative position of the first and second cameras relative to one another ,
• - Determining at least one calibration value by means of an artificial neural network using the height value as an input variable, and
• - Calibrate a detection area of the camera system based on the calibration value.

The calibration value can be, for example, an indication of the extent to which the detection range of the camera system has been shifted, expanded or restricted compared to a normal state. Of course, the present method is not limited to a single calibration value. Of course, several calibration values can also be determined and subsequently used for calibrating the detection area. In the context of the present application, a calibration value in the singular is usually mentioned. However, this wording should explicitly include the use of multiple calibration values.

A calibration of the camera system during the intended operation of the motor vehicle can also be referred to as a so-called online calibration. The intended operation of the motor vehicle can, for example, be an everyday use of the motor vehicle by an end user. For example, the intended operation of the motor vehicle includes trips of the motor vehicle, parking maneuvers and / or any other driving maneuvers of the motor vehicle, in particular at the end customer. In particular, calibration of the camera system during the intended operation of the motor vehicle does not include calibrations that are carried out in a workshop or on the production side, for example during manufacture of the motor vehicle. A calibration during the intended operation of the motor vehicle can be characterized by regular repetition during any use of the motor vehicle.

The first camera and the second camera each have a detection area. The respective detection areas of the first and the second camera overlap when the first camera and the second camera are arranged as intended on the motor vehicle. The overlap area is captured by both the first camera and the second camera. The overlap area thus represents the intersection of the respective detection areas of the first and the second camera.

When determining the height value, the first and the second image are evaluated in each case in particular in their overlap area. The height value can be determined using stereoscopy. The model of the motor vehicle is used for the use of stereoscopy. The target value for the relative position of the first and the second camera relative to one another is provided by the model of the motor vehicle. The target value can, for example, be specific for a predetermined state of the motor vehicle. In this predetermined state, the first and the second camera can be in a relative position to one another which corresponds to the target value. The predetermined state can be defined, for example, as follows: the motor vehicle stands on a flat base surface, with no additional acceleration acting on the motor vehicle apart from gravitational acceleration. In particular, the gravitational acceleration in this predetermined state is parallel to a vertical axis of the vehicle or perpendicular to the flat base.

As an alternative or in addition, the model can comprise an absolute value for the position of the first and / or the second camera in relation to a coordinate system aligned with the motor vehicle. In this case, the target value for the relative position of the first and the second camera can be provided implicitly on the basis of the corresponding absolute positions. The height value can represent a height of the surrounding area, which is represented in the overlap area by the first and the second camera. For example, the height value represents a height of a flat or curved surface on which the motor vehicle is located. The height value can be related, for example, to a reference system of the first camera, a reference system of the second camera or a reference system of the motor vehicle, in particular the coordinate system aligned with the motor vehicle. In particular, the height value is aligned with the vertical axis of the motor vehicle. In other words, the height value can indicate the height of the surrounding area shown in relation to the vertical axis of the motor vehicle.

The artificial neural network can be taught to determine the at least one calibration value using the height value as an input variable. In particular, a correspondingly trained artificial neural network is used to determine the at least one calibration value used. The artificial neural network can, for example, be a folding neural network, also known as the “convolutional neuronal network” (CNN), or a recurrent neural network, also known as “recurrent neuronal network” (short) : RNN), act. The detection value of the first and / or the second camera can be determined, for example, explicitly or implicitly by the calibration value. The camera system is then calibrated based on the calibration value. For example, a reference system of the camera, in particular based on the detection range of the camera, is calibrated on the basis of the calibration value.

The invention is based on the knowledge that, depending on the height value in the overlap area, the detection area of the camera system can be calibrated. For example, the height value can be used to recognize that the motor vehicle is on an uneven roadway, in particular a dome, and / or that some of its wheels are on an elevated surface, for example a curb. Such conditions can shift the detection range of the camera system. Alternatively or additionally, the detection range of the camera system can be shifted by loading. A shift in the detection range, for example on the basis of one of the conditions mentioned, can be detected on the basis of the height value. The artificial neural network is trained to use such height values as an input variable for determining the at least one corresponding calibration value. Using this calibration value, the detection area can then be calibrated or it can be determined where the detection area extends.

Of course, the camera system can also include more than two cameras. For example, the camera system comprises four cameras. In this case, several of the cameras can have a respective common overlap area with one another. For example, the four cameras are located on four different sides of the motor vehicle. In this case, each of the cameras can have a respective overlap area with those of the cameras which are arranged on an adjacent side. Thus, in the example with four cameras, each of the four cameras can have two overlap areas, a total of four overlap areas being present. In particular, a respective height value is then determined for each of the overlap areas. The at least one calibration value can then be determined using the plurality of height values as an input variable.

Within the scope of the present application, there is often talk of a height value for the surrounding area shown in the overlap area. Of course, several height values can be determined for the overlap area or for the surrounding area shown in the overlap area. In this case, several height values are used per overlap area for determining the at least one calibration value.

According to a further development, it is provided that a slope and / or an unevenness of a subsurface on which the motor vehicle is located are taken into account for the calibration of the detection range on the basis of the height value. As already described above, the detection area can shift due to such a slope or such unevenness. For example, the higher the height value, the larger the detection area. This is due to the fact that the first and second cameras can then “see further” due to the geometry.

According to a further development, it is provided that an angular deviation between a vertical vehicle axis and a surface normal of the surface on which the motor vehicle is located is taken into account for the calibration of the detection range on the basis of the height value. Such an angular deviation between the vertical vehicle axis and the surface normal can result, for example, from a load on the motor vehicle. If, for example, the rear storage space of the motor vehicle is heavily loaded, the motor vehicle can accordingly tilt backwards. This can be recognized, for example, by respective height values both in an area in front of the motor vehicle and in an area behind the motor vehicle. The detection range of individual cameras can be reduced and / or expanded by a corresponding inclination or angular deviation of the motor vehicle. Overall, the detection range of the camera system can change as a result. In the above-mentioned example with the heavily loaded rear storage space, the detection area of a forward-facing camera can expand and / or the detection area of a rear-facing camera of the motor vehicle can be reduced.

According to a development, it is provided that an influence of a loading of the motor vehicle and / or an influence of a trailer attached to the motor vehicle on the detection range is taken into account on the basis of the height value. For example, such a trailer attached to the motor vehicle can have an angular deviation, as described above, between the vertical axis of the vehicle and the surface normal of the subsurface cause. In other words, the detection area of individual cameras and / or the entire camera system can be shifted by loading the motor vehicle and / or a trailer attached to the motor vehicle. This is taken into account in this way.

A mechanical deformability of the motor vehicle, in particular a body of the motor vehicle, can be taken into account by the model of the motor vehicle. Such mechanical deformability can be caused, for example, by loading the motor vehicle and / or a trailer attached to the motor vehicle. The mechanical deformability of the motor vehicle can be based on a finite rigidity of a frame of the motor vehicle. Alternatively or additionally, such a mechanical deformation of the motor vehicle can result from an acceleration of the motor vehicle. This will be discussed in more detail later. The detection range of individual cameras, in particular the first or the second camera, of the camera system can shift due to the mechanical deformability. This can be based in particular on a change in position of the respective camera (s) based on the mechanical deformation.

According to a further development, an actual value for the position of the first camera and / or the second camera relative to a reference system of the motor vehicle and / or relative to one another is determined taking into account the mechanical deformability as the calibration value. For example, the actual value or a respective actual value for the position of the first camera and / or the second camera relative to the reference system of the motor vehicle, in particular the coordinate system aligned with the motor vehicle, is determined. Alternatively or additionally, the actual value for the position of the first camera and the second camera can be determined relative to one another. In this way, the mechanical deformability can be taken into account in a particularly advantageous manner.

According to a development, in addition to the position of the respective camera, that is to say in particular the first and / or the second camera, a corresponding actual value for the orientation can be determined as the calibration value. In such exemplary embodiments, it is provided in particular that the model of the motor vehicle additionally provides at least one target value for the orientation of the first camera and / or the second camera. In this way, a shift in the detection range of the camera system or the individual cameras in the camera system can be taken into account by a shift in the orientation, in particular due to the mechanical deformability.

The above-mentioned actual value for the position and / or the above-mentioned actual value for the orientation can be determined in particular by the artificial neural network, taking into account the height value and the model of the motor vehicle. In particular, the actual value for the position and / or the orientation of the respective camera is determined by the artificial neural network on the basis of the height value, taking into account the mechanical deformability of the motor vehicle. Accordingly, the artificial neural network can be trained on it, or a change in the pose (position and orientation) of the first camera and / or the second camera relative to the overlapping area in the respective images of the first and second cameras, taking into account the model of the motor vehicle to determine the respective target value.

According to a further development, it is provided that the calibration value, in particular the actual value for the position, is initially determined in arbitrary length units and is only converted into length units related to the motor vehicle in a later method step. In other words, the calibration value, in particular the actual value for the position, can be determined by the artificial neural network in relative or arbitrary length units. Only in the later process step, in particular after the calibration value has been output by the artificial neural network, can the conversion into the length units relating to the motor vehicle be carried out. The length units related to the motor vehicle can be absolute length units. In this way, the artificial neural network can be taught in particularly universally on a large number of different motor vehicles, since the absolute length units or those relating to the motor vehicle are only calculated later, in particular in a method step following the artificial neural network.

According to a further development, it is provided that motion data of the motor vehicle are used as a further input variable for the artificial neural network for determining the calibration value, the influence of an acceleration of the motor vehicle on the detection area along any spatial direction being taken into account by means of the motion data. For example, the acceleration can be an acceleration along a longitudinal axis of the motor vehicle. Such an acceleration along the longitudinal axis of the vehicle can be accelerated or braked. Alternatively or additionally, the acceleration can be an acceleration along a vehicle transverse axis of the motor vehicle. Such acceleration along the The vehicle's transverse axis can be caused, for example, by steering the motor vehicle. Alternatively or additionally, the acceleration can be an acceleration along the vertical axis of the vehicle. Such acceleration along the vertical axis of the vehicle can be caused, for example, by the motor vehicle traveling over an uneven surface. The influence of the acceleration of the motor vehicle along all three axes mentioned is particularly advantageously taken into account for the calibration of the detection range. The movement data of the motor vehicle can be recorded, for example, by odometry or corresponding sensors, in particular acceleration sensors of the motor vehicle. In this way, the detection area can be calibrated even more precisely.

According to a further development, a further input variable for the artificial neural network is formed based on a first pair of images and a second pair of images, the first pair of images being two images taken at different times by the first camera of the camera system and the second pair of images being two using the second camera of the camera system includes images taken at different times. The first image can be part of the first image pair, but this is purely optional. The second image can be part of the second image pair, but this is purely optional. The images of the first pair of images can be taken at different times. Accordingly, the images of the first pair of images can represent the surrounding area of the motor vehicle at different times. The images of the second pair of images can be taken at different times. Accordingly, the images of the second pair of images can represent the surrounding area at different times.

The images of the first and the second pair of images can be taken in pairs across the pairs at the same time. However, this is purely optional. In this example it can be provided that a respective image of the first pair of images and a respective image of the second pair of images are each taken at the same time. In summary, a further input variable for the artificial neural network is formed from the first pair of images and the second pair of images. The further input variable can relate, for example, to an optical flow or a change between the images of a respective one of the image pairs. Alternatively, the further input variable can relate to a movement of an object represented in the images and / or a displacement of such an object in the respective images of one of the image pairs. In particular, the images of the first image pair and / or the second image pair are individual images of a video sequence. Such single images are also called frames. The further input variable can be formed on the basis of the optical flow, the change, the movement of an object or another change in successive frames. In this way, the above-mentioned movement data in particular can be provided. In other words, the movement data of the motor vehicle can be extracted from the first pair of images and / or the second pair of images. In this way, the detection area can be calibrated even more precisely.

According to a development, it is provided that the first and the second pair of images are each evaluated by a different input part of the artificial neural network, with each input part processing both images of the respective image pair in parallel, in particular by a different encoder of the corresponding input part. In other words, the artificial neural network has two different input parts, each of which evaluates a different one of the image pairs. In particular, each of the image pairs is evaluated exclusively by one of the two input parts of the artificial neural network. Each of the input parts can comprise a respective different encoder, in particular a CNN encoder. Advantageously, the first and second image pairs are then evaluated at least partially by the respective encoder of the corresponding input part. Alternatively or additionally, it can be provided that the two images of the first pair of images and the two images of the second pair of images are each evaluated by different input parts of the artificial neural network, in particular by different encoders of the artificial neural network. For example, it is provided that the images of the first pair of images are evaluated by different input parts or different encoders. For example, it is provided that the images of the second pair of images are each evaluated by different input parts or different encoders. In total, up to four different input parts or different encoders, in particular CNN encoders, can thus be provided for the first and the second pair of images. In this way, the first and second image pairs can be evaluated particularly efficiently.

According to a further development, it is provided that respective movement data are extracted from the first image pair and the second image pair and the artificial neural network is taught thereon, the determination of the calibration value based at least in part on the movement data perform. In other words, as already described above, the movement data are extracted from the first and the second pair of images as a further input variable for the artificial neural network. In this way, the artificial neural network can learn to recognize the influence of the movement data, in particular the acceleration, on the detection area of the camera system. Based on this, the calibration value is subsequently determined based at least in part on the movement data.

A second aspect of the invention relates to a control unit for a motor vehicle, which is set up to carry out a method for calibrating a camera system of a motor vehicle, as described in the context of this application. Accordingly, further developments of the corresponding method apply analogously to the control unit and vice versa.

• - einem Kamerasystem, wobei das Kamerasystem zumindest eine erste Kamera und eine von der ersten Kamera verschiedene zweite Kamera umfasst und wobei die erste und die zweite Kamera einen gemeinsamen Überlappungsbereich, der Teil eines jeweiligen Erfassungsbereichs sowohl der ersten als auch der zweiten Kamera ist, aufweisen, und
• - der oben genannten Steuereinheit.

Another aspect of the invention relates to a driver assistance system for a motor vehicle, with

• a camera system, the camera system comprising at least a first camera and a second camera different from the first camera, and the first and second cameras having a common overlap area which is part of a respective detection area of both the first and the second camera, and
• - the control unit mentioned above.

In particular, the driver assistance system, advantageously the control unit, is set up to carry out the method described here. Accordingly, features and advantages that are disclosed in relation to the method also apply to the driver assistance system.

In addition, part of the invention is a motor vehicle which comprises the above-mentioned driver assistance system. The motor vehicle is in particular a motor vehicle, for example a truck or a passenger car. For example, the motor vehicle can have an electric drive and / or an internal combustion engine. The camera system, in particular the first and the second camera, of the driver assistance system are arranged on the motor vehicle. Features and advantages that are disclosed in relation to the method also apply to the driver assistance system.

The invention also includes a computer program product with program code means which are stored in a computer-readable medium in order to carry out the method for calibrating the detection range of the camera system when the computer program product is processed on a processor of an electronic control unit.

The computer program product can be stored on a computer-readable medium. Thus, this invention also claims such a computer-readable medium, in particular in the form of a computer-readable floppy disk, CD, DVD, memory card, USB memory unit, or the like, in which program code means are stored in order to carry out the method for calibrating the detection range of the camera system when the program code means loaded into a memory of an electronic control unit and processed on a processor of the electronic control unit. The readable medium can be volatile (non-volatile) or non-volatile. Such a volatile memory can in particular be formed by a working memory of a microprocessor.

Further features of the invention result from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the combination indicated in each case but also in other combinations without departing from the scope of the invention . Embodiments of the invention are thus also to be regarded as encompassed and disclosed, which are not explicitly shown and explained in the figures, but can be derived from the explanations explained and can be generated by separate combinations of features. Embodiments and combinations of features are also to be regarded as disclosed, which therefore do not have all the features of an originally formulated independent claim. Furthermore, versions and combinations of features, in particular those explained above, are to be regarded as disclosed, which go beyond or differ from the combinations of features set out in the references of the claims.

• 1 ein Kraftfahrzeug mit einem Kamerasystems, wobei ein Erfassungsbereich des Kamerasystems aufgrund einer Anhängelast gegenüber einem Normalzustand verschoben ist;
• 2 äußerst schematisch die Verschiebung von Kamerapositionen eines Kamerasystems im Falle unterschiedlicher Fahrmanöver aus einer Vogelperspektive;
• 3 weitere mögliche Verschiebungen von Kamerapositionen aus einer perspektivischen Seitenansicht;
• 4 eine beispielhafte Architektur für ein Fahrerassistenzsystem zum Kalibrieren eines Erfassungsbereichs eines Kamerasystems;
• 5 einen Auszug aus der Architektur; und
• 6 die Überlappungsbereiche unterschiedlicher Kameras eines Kamerasystems in einer schematischen Vogelperspektive.

Show:

• 1 a motor vehicle with a camera system, a detection range of the camera system being shifted from a normal state due to a trailer load;
• 2nd extremely schematically the shift of camera positions of a camera system in the case of different driving maneuvers from a bird's eye view;
• 3rd further possible shifts of camera positions from a perspective side view;
• 4th an exemplary architecture for a driver assistance system for calibrating a detection range of a camera system;
• 5 an excerpt from the architecture; and
• 6 the overlapping areas of different cameras of a camera system in a schematic bird's eye view.

1 shows a motor vehicle 1 with a camera system, the camera system comprising four cameras in the present exemplary embodiment. Specifically, the motor vehicle or the camera system has a front camera FV , a rear-facing camera RV as well as two side-aligned cameras ML and MR on, being in the 1 only one on the left side of the motor vehicle 1 arranged side camera ML is shown. One on the right side of the vehicle 1 arranged side camera MR is in 1 not shown. The front camera FV has a detection area 92 on. The rear-facing camera RV has a detection area 94 on.

On the motor vehicle 1 is according to 1 a follower 98 attached. By one from the trailer 98 on the motor vehicle 1 transmitted vehicle load decreases the motor vehicle 1 in the area of its trailer coupling according to an arrow 97 downwards. Opposite the descent according to the direction of the arrow 97 operates a rear axle of the motor vehicle 1 as the axis of rotation. The motor vehicle is lifted by this axis of rotation 1 in the front area according to an arrow 96 . Thus the detection area 92 the front camera FV deflected upwards, in particular according to an arrow 93 . In addition, the detection area 94 the rear-facing camera RV deflected downwards, in particular according to an arrow 95 . By deflecting the respective detection areas 92 , 94 the respective detection range of the corresponding camera shifts FV , RV . By an upward deflection, as here in the case of the front camera FV , the corresponding detection area increases 92 . By a downward deflection, as in the present case with the rear-facing camera RV , the corresponding detection area is reduced 94 accordingly.

2nd shows the cameras very schematically ML , FV , MR , RV of a camera system 12 from an extremely schematic bird's eye view. For a better overview, three axes are shown, with a z-axis of a vehicle vertical axis of the motor vehicle 1 corresponds to a y-axis of a vehicle longitudinal axis of the motor vehicle 1 corresponds and an x-axis of a vehicle transverse axis of the motor vehicle 1 corresponds. In an exemplary situation C1 the motor vehicle is, for example, in a uniform movement or when standing on a flat surface. The positions of all four cameras ML , FV , MR , RV correspond to a normal state or a target value. In an exemplary situation C2 becomes a front axle of the motor vehicle 1 slowed down. Due to an incomplete stiffness of a body of the motor vehicle 1 and due to the acceleration resulting from braking, the motor vehicle is compressed in its front part. This is evident from the fact that the position of the front camera FV shifted from their normal state or their target value. In another exemplary situation C3 becomes the motor vehicle 1 accelerates. In the present example, the acceleration is via a corresponding power output via the rear axle of the motor vehicle 1 and / or by pushing the trailer 98 (e.g. downhill). This results in a compression of the body of the motor vehicle 1 in the rear area. For this reason, the position of the rear-facing camera shifts RV compared to their normal state or their target value.

Other exemplary situations are in 3rd shown. In 3rd are the cameras FV , MR , RV , ML shown extremely schematically in a side perspective view. A view of the cameras in 3rd thus corresponds at least partially to the vehicle transverse axis or the x-axis. In the situation C4 is the motor vehicle 1 on a flat surface. This is through representation 80 of the motor vehicle 1 shown. The four cameras RV , MR , FV , ML are in their normal state or their position corresponds to the target value. In the exemplary situation C5 is the motor vehicle 1 with its front axle on a bump, especially on a hill or an elevation. This is through representation 81 shown. In the situation C5 for example, there is essentially a rotation of the motor vehicle about its transverse vehicle axis or about the x axis. The positions of the individual cameras shift relative to their normal state or their respective target value. The same applies analogously to the situation C6 , in which the motor vehicle is on an inclined base. The inclination of the base area runs partly along the longitudinal axis of the vehicle (see representation 82 which the motor vehicle 1 from behind). In particular, the motor vehicle rotates with respect to its y-axis.

Several exemplary states have now been shown, in which there is a detection range of a camera system 12 a motor vehicle 1 can move. In the case of such a shift, it is possible that for foreign objects, which wrong positions and / or wrong dimensions are to be determined by means of the camera system. If a driver assistance function, for example an emergency brake assistant, a parking assistant or a lane departure warning system, is controlled by the camera system based on an environment detection, this can lead to incorrect or incorrect behavior of the driver assistance function. For this reason, the detection area of the camera system is provided in the present case 12 to calibrate.

4th shows an exemplary architecture of a driver assistance system 2nd , which is set up to cover the detection area of a camera system 12 as part of a so-called "online calibration", that is, while the motor vehicle is in operation 1 to calibrate. An artificial neural network 19th become respective image pairs 4th from each of the cameras VF , ML , RV , MR provided. Each of the pairs of images 4th includes two pictures 5 which at different times with the respective camera VF , ML , RV , MR were recorded. For example, the pictures 5 of a respective image pair 4th by two different frames of a video signal. The respective image pairs 4th of the different cameras VF , ML , RV , MR become a respective input part 18th of the artificial neural network 19th fed. In other words, the artificial neural network 19th to process the images 5 so many input parts 18th on how the camera system 12 different cameras VF , ML , RV , MR having. Any input part 18th again has two encoders 3rd on. It is a respective encoder 3rd for each picture 5 of a respective pair of images 4th intended. Such an architecture is also referred to with the English technical term "multi stream convolutional neural network" or "two stream convolutional neural network". This way every camera is VF , ML , RV , MR a respective input part 18th of the artificial neural network 19th assigned. Any input part 18th each has two encoders, in particular so-called CNN encoders, that is, encoders of an artificial neural network. The two pictures 5 of a respective image pair 4th are each a different one of the two encoders 3rd of the corresponding input section 18th evaluated. In other words, both are single frames 5 through the respective encoder 3rd evaluated separately. For example, the encoders 3rd each have one or more folded layers, also called convolutional layers, and / or summary layers, also called pooling layers. This way the pictures 5 by the respective encoder 3rd preprocessed.

Then the preprocessed images from the two encoders 3rd through a link position 6 of the artificial neural network. Any input part 18th thus shows a respective link position 6 on. In the link position 6 one of the entrance parts 18th will the pictures 5 through the corresponding input section 18th processed image pair 4th linked together. In particular, the link position includes 6 one or more fully linked layers, also called fully-connected layers. In some embodiments, the linkage exists 6 exclusively from one or more fully-connected layers. The input parts 18th are in particular essentially designed to generate movement data from the respective image pair 4th to extract. For example, the input parts 18th learned the movement data from the pictures 5 that were recorded at different times. For this purpose, for example, an optical flow, a shift in features, or any other change between the images 5 of a pair of images 4th be evaluated. By appropriate linkage 6 can have respective values for roll angle, pitch angle and / or yaw angle of the motor vehicle 1 be issued. Alternatively or additionally, respective translatory coordinates for the motor vehicle can be used 1 determined and issued. In particular, the translational coordinates are output in arbitrary units. In this way, the artificial neural network 19th are taught in a particularly general manner, the conversion of the translational coordinates into vehicle-specific length units only in a later processing step, in particular after output by the artificial neural network 19th , he follows. In this case, a scale that is necessary for this can be used for the subsequent conversion using the link position 6 be issued.

In one height unit 7 become height values for respective overlap areas 21st of the cameras ML , VR , MR , LV certainly. In 6 are the overlap areas 21st shown. Each of the cameras ML , LV , MR , RV has a respective detection area 20th on. In the overlap areas 21st the detection areas overlap 20th two of the cameras. In this way, the overlap areas 21st stereoscopically by two different cameras ML , RV , MR , LV be recorded. With the help of stereoscopic evaluation, respective height values for the overlap areas can be obtained 21st be determined. This is basically known from the prior art. In particular, a sub-surface on which the motor vehicle is located 1 , respective height values for each of the overlap areas 21st be assigned. To illustrate, let's go back to 1 referred. Because of the shift according to the arrow 96 would be in the front of the vehicle 1 result in a higher altitude value than in a normal state and due to the shift along the arrow 97 would result in a lower height value compared to a normal state in the rear area of the motor vehicle. This will be used to calibrate the camera system 12 of the motor vehicle 1 utilized.

The determination of the height values by the height unit 7 can be based on a model 17th of the motor vehicle. This is done by the model 17th a respective relative position of the individual cameras FV , ML , RV , MR provided relative to each other. In other words, the model 17th indicated in which position on the motor vehicle 1 a respective one of the cameras FV , ML , RV , MR is arranged. This is necessary in order to be able to correctly determine the height values using stereoscopy. The evaluation of the stereoscopic images of the different cameras FV , ML , RV , MR can always be done with the target value for the respective camera pose / camera position. Alternatively, when the present method is carried out repeatedly, the corresponding height value can be determined with a predetermined actual value for the corresponding position. For each of the overlap areas 21st a respective height value is advantageously determined. It is even more advantageous if for each of the overlap areas 21st a plurality of respective height values are determined, which in particular a plurality of values over the corresponding overlap areas 21st affect distributed areas.

The size and location of the overlap areas 21st can depend on the surrounding area recorded therein, in particular the base area on which the motor vehicle is located 1 varies. For each of the detection areas 20th represents the overlap area 21st contributing part only a small area at the edge of the corresponding detection area 20th To determine the height values, individual image areas can be used, which are in the images of two different cameras FV , ML RV MR are represented by means of the model 17th be evaluated using triangulation. For this purpose, beam paths of individual light beams striking the camera can be evaluated. In particular, such light beams can first be converted into Plücker coordinates. Alternatively or additionally, the cameras FV , ML , RV , MR in a fixed, on the motor vehicle 1 aligned coordinate system are projected. The triangulation then determines the altitude values. The described procedure can cause problems with large deviations in the display due to fisheye lenses of the individual cameras FV , ML , RV , MR be counteracted.

The output of the link layers 6 as well as the altitude values from the altitude unit 7 become an output part 8th of the artificial neural network. The output part 8th can for example be designed as a recurrent neural network (RNN). Such a recurrent neural network is also referred to with the English technical term recurrent neural network. The output part 8th is trained on this, based on the height values on the one hand and the output of the link positions 6 on the other hand a pose 9 of the motor vehicle 1 and / or respective information for the pose of the individual cameras FV , ML , RV , MR to determine. The output of the link layers 6 can in particular include or be formed from the movement data. In addition, the model 17th of the motor vehicle 1 through the output part 8th to determine the pose 9 or the poses 10th be used. Through the model 17th can setpoints for position and / or pose of the individual cameras FV , ML , RV , MR to be provided. Through the model 17th can continue the mechanical deformability of the body of the motor vehicle 1 be taken into account. The poses 10th of the individual cameras FV , ML , RV , MR include an actual value for the respective pose of the corresponding camera. These actual values for the pose can thus be determined based on the respective target value taking into account the height values and / or the movement data. The output part is advantageous 8th thus learned to do so based on the model 17th or based on the target values for the pose of a particular camera FV , ML , RV , MR and taking into account the movement data and the height values, the actual value for the position / pose 10th the corresponding camera FV , ML , RV , MR to determine. The positions / poses 10th of the cameras FV , ML , RV , MR can be called "camera extrinsics".

The artificial neural network 19th is taught as a whole. In particular, teaching takes place on the basis of a corresponding learning algorithm, in particular a so-called “deep learning” algorithm. The artificial neural network learns 19th exactly those features from the image pairs 4th extract which for the movement information or odometry as well as for determining the pose 9 of the motor vehicle 1 are important. The output unit 8th or the recurrent neural network automatically learns structural information from the model 17th of the motor vehicle 1 to merge the movement information as well as the altitude values and / or the positions / poses 10th of the individual cameras FV , ML , RV , MR as well as the pose 9 of the motor vehicle 1 to determine.

Based on the positions / poses 10th of the cameras FV , ML , RV , MR as well as the pose 9 of the motor vehicle 1 can thus cover the detection range of the camera system 12 be calibrated. For example, the detection area of the Camera system 12 using geometric calculations or an assignment table from the positions / poses 10th of the cameras FV , ML , RV , MR as well as the pose 9 of the motor vehicle 1 be derived. The positions / poses 10th of the cameras FV , ML , RV , MR as well as the pose 9 of the motor vehicle 1 in the present example from the height values in the overlap areas 21st as well as the movement data derived.

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• CN 106960456 [0004]

Claims (15)

Method for calibrating a camera system (12) of a motor vehicle (1) during intended operation of the motor vehicle (1), the camera system (12) having at least a first camera (FV) and a second camera (ML ) and wherein the first and the second camera (ML) have a common overlap area (21), which is part of a respective detection area (20) of both the first and the second camera (FV, ML), with the steps: Provision of a first and a second image (5), the first image (5) being recorded by means of the first camera (FV) and the second image (5) being recorded by means of the second camera (ML), - Determining a height value for a surrounding area of the motor vehicle (1) shown in the overlap area (21) by evaluating the first and the second image (5) in their overlap area (21) taking into account a model (17) of the motor vehicle (1), wherein the model provides at least one target value for the relative position of the first and the second camera (FV, ML) relative to one another, - Determining at least one calibration value by means of an artificial neural network (19) using the height value as an input variable, and - Calibrating a detection area (20) of the camera system (12) based on the calibration value. Procedure according to Claim 1 , characterized in that a slope and / or an unevenness of a subsurface on which the motor vehicle (1) is located is taken into account for the calibration of the detection area (20) on the basis of the height value. Procedure according to Claim 1 or 2nd , characterized in that an angular deviation between a vertical vehicle axis and a surface normal of a surface on which the motor vehicle (1) is located is taken into account for the calibration of the detection area (20) on the basis of the height value. Method according to one of the preceding claims, characterized in that an influence of a loading of the motor vehicle (1) and / or an influence of a trailer attached to the motor vehicle (1) on the detection area (20) is taken into account on the basis of the height value. Method according to one of the preceding claims, characterized in that the model (17) of the motor vehicle (1) takes into account a mechanical deformability of the motor vehicle (1), in particular a body of the motor vehicle (1). Procedure according to Claim 5 , characterized in that an actual value for the position of the first camera (FV) and / or the second camera (ML) relative to a reference system of the motor vehicle (1) and / or relative to one another taking into account the mechanical deformability as the calibration value becomes. Procedure according to Claim 6 , characterized in that in addition to the position of the respective camera (FV, RV, ML, MR), a corresponding actual value for the orientation is determined as the calibration value. Method according to one of the preceding claims, characterized in that the calibration value is first determined in arbitrary length units and is only converted into length units related to the motor vehicle (1) in a later method step. Method according to one of the preceding claims, characterized in that movement data of the motor vehicle (1) are used as a further input variable for the artificial neural network (19) for determining the calibration value, the influence of an acceleration of the motor vehicle (1) along using the movement data any spatial direction on the detection area (20) is taken into account. Method according to one of the preceding claims, characterized in that a further input variable for the artificial neural network (19) is formed on the basis of a first pair of images (4) and a second pair of images (4), the first pair of images (4) being two by means of the first camera (FV) of the camera system (12) comprises images (5) taken at different times and the second pair (4) comprises two images (5) taken by means of the second camera (ML) of the camera system (12) at different times. Procedure according to Claim 10 , characterized in that the first and the second pair of images (4) are each evaluated by a different input part (18) of the artificial neural network (19), with each input part (18) processing both images (5) of the respective image pair in parallel (4), in particular a respective different encoder (3) of the corresponding input part (18). Procedure according to Claim 10 or 11 , characterized in that the first pair of images (4) and the second pair of images (4) are used to extract movement data and the artificial one neural network (19) is learned to carry out the determination of the calibration value based at least in part on the movement data. Control unit for a motor vehicle (1), which is set up to carry out a method according to one of the preceding claims. Driver assistance system for a motor vehicle (1), with - a camera system (12), the camera system (12) comprising at least a first camera (FV) and a second camera (ML) different from the first camera (FV), and wherein the first and the second camera (FV, ML) has a common overlap area (21) which is part of a respective detection area (20) of both the first and the second camera (FV, ML), and - a control unit Claim 13 . Computer program product with program code means, which are stored in a computer-readable medium, for the method for calibrating a camera system of a motor vehicle (1) according to one of the preceding Claims 1 to 12 to be carried out when the computer program product is processed on a processor of an electronic control unit.

Images