DE102009033854A1 - Method for observation of stereo camera arrangement in vehicle or robot, involves recording image data pixels by two cameras, where image data pixels are processed stereoscopically with corresponding pixels of image pair by processing unit - Google Patents

Abstract

The method involves recording image data pixels (BP1,BP2) by two cameras (1.1,1.2). The image data pixels are processed stereoscopically with corresponding pixels of an image pair by a processing unit (1.3). An actual movement of the stereo camera arrangement is compared with a movement determined from the image data according to disparity values or optical data flow.

Description

The The invention relates to a method for monitoring a Stereo camera arrangement, using two cameras as image data Pixels are detected, which by means of a processing unit stereoscopic processed to mutually corresponding pixels of a pair of images being involved in detection and / or processing with each other Corresponding pixels occurring errors are determined.

From the DE 103 02 671 A1 For example, a method and a device for adjusting at least one parameter of an image sensor of a stereo camera in a motor vehicle are known. The stereo camera consists of two image sensors that record essentially the same scene. When an error occurs in one of the image sensors, at least one parameter of this image sensor is set as a function of at least one measured value of the error-free image sensor of the image sensor system. In one embodiment, the parameter is at least one exposure parameter of the image sensor. If an image sensor of the stereo camera fails, the at least one exposure parameter is set as a function of at least one measured value of the second image sensor, wherein the measured value is a measure of the exposure of at least part of the image of the second image sensor.

Furthermore, the disclosure DE 101 49 115 A1 an object detection device for driver assistance systems in motor vehicles having at least two sensor systems which measure data about the location and / or movement state of objects in the surroundings of the vehicle and whose detection areas overlap one another. The object detection device comprises an error detection device, which checks the data measured by the sensor systems for their consistency and outputs an error signal upon detection of a contradiction.

Of the Invention is based on the object, a relation to the Prior art improved method of monitoring indicate a stereo camera arrangement. The invention is still The task is based on a use of a method of monitoring indicate a stereo camera arrangement.

Regarding of the method according to the invention by the features specified in claim 1 and in terms of use by solved specified in claim 6 features.

advantageous Embodiments of the invention are the subject of the dependent claims.

at the method for monitoring a stereo camera arrangement be captured by two cameras as image data pixels, which by means of a processing unit stereoscopically to each other corresponding pixels of a picture pair are processed, wherein in the detection and / or the processing with each other corresponding Pixels occurring errors are determined.

According to the invention in a plausibility check a real movement of the stereo camera assembly with one of the image data based on disparity values and / or optical flow data compared to determined movement, with deviations between the real movement and the movement detected and monitored become.

The Stereo camera is in particular part of a vehicle. At a deviation between the real movement, which in particular from vehicle-specific measured values, such as vehicle speed, a yaw rate, a yaw angle and / or acceleration values is determined and can be determined from the image data movement advantageously to a fault of the stereo camera arrangement getting closed. Another special advantage is the method according to the invention is possible besides errors like a failure of one or both cameras, one Impairment of an image signal of the cameras by partially or complete occlusion also sync error, in which single pictures of the cameras "freeze" or in which there is an asynchronous detection of the cameras, in particular easier way to capture. It is also advantageous that no additional test units and / or test patterns to determine the errors are required.

embodiments The invention will be described below with reference to a drawing explained.

there shows:

1 schematically a process flow for monitoring a stereo camera assembly.

The only 1 shows a possible procedure for monitoring a stereo camera arrangement 1 , The stereo camera arrangement 1 is arranged for example in a vehicle, not shown. Furthermore, there is also a use of the stereo camera arrangement 1 and the method for monitoring the stereo camera arrangement 1 possible in other conceivable mobile and immobile applications. The stereo camera arrangement 1 and the method for its control are particularly suitable for a control a robot, for example in the manufacture of a vehicle.

The stereo camera arrangement 1 comprises two, in particular horizontally arranged side by side cameras 1.1 . 1.2 for capturing image data, ie images B1, B2 of an environment of the vehicle. The images B1, B2 then become a processing unit 1.3 fed to stereoscopic image processing.

In this stereoscopic image processing, correspondences of pixels BP1, BP2 (also called pixels) in the stereoscopically recorded images B1, B2 are determined. In this case, the images B1, B2 are processed in a stereo algorithm in such a way that coordinates of a pixel BP1 of the one image B1 are compared with coordinates of a pixel BP2 of the other image B2 regarded as potentially corresponding. From a distance of the pixels BP1 and BP2 from each other, the so-called disparity, and a known distance of horizontally adjacent cameras 1.1 . 1.2 , the so-called base width, the distance of an object having the pixels BP1 and BP2 to the cameras 1.1 . 1.2 certainly. According to this algorithm, disparities are generated for all the pixels BP1, BP2 of the images B1, B2, and a disparity image D or a disparity map is generated which represents a three-dimensional representation of the object in its context. In this way, the distance and spatial position of the object relative to the cameras 1.1 . 1.2 determined.

The determination of the disparity image D takes place for example on the basis of a so-called "off" H. Hirschmüller: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information; 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, pp. 807-814 "Known semi-global matching algorithm, wherein the images B1, B2 sampled and costs corresponding to the similarity of the pixels BP1, BP2 are accumulated horizontally, vertically and diagonally. The cost calculation is based on a calculation of the transinformation related to the individual pixels BP1, BP2 of the images B1, B2.

In addition, jumps in disparities of adjacent pixels BP1, BP2, as occur, for example, on gray value edges, are preferably assigned a small penalty P1 or a large penalty P2, depending on their size. This so-called smoothness constraint, also known as smoothness constraint, in turn, for example, by the " H. Hirschmüller: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information; 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, pp. 807-814 Realized "known semi-global matching algorithm.

Furthermore, errors of the stereo camera arrangement are easily eliminated from the disparity image D 1 like a failure of one or both cameras 1.1 . 1.2 , an impairment of the image signals of the cameras 1.1 . 1.2 by partial or complete occlusion, e.g. B. caused by pollution, weather or a windshield wiper, detectable. In the event of a fault, a quality of the three-dimensional representation of the surroundings of the vehicle decreases sharply, especially in a close range.

However sync errors, in which individual images B1, B2 of the cameras 1.1 . 1.2 "Freeze" or in which an asynchronous detection of the cameras 1.1 . 1.2 To detect hardware faults, calibration errors and optical channel faults, including soiling and rain, such as rain or snowfall, are two consecutive image pairs by the processing unit 1.3 analyzed. The pairs of images do not have to be pairs of images recorded one after the other.

During this analysis of the image pairs by means of the processing unit 1.3 a movement of the captured pixels BP1, BP2 between a first time and a second time is determined. This movement is preferably carried out by calculating flow data, ie a so-called SceneFlow, which is a change in the distance of the pixels BP1, BP2 to the cameras 1.1 . 1.2 represents determined. The calculation of the SceneFlow is determined in particular by A. Wedel et al .: Efficient Dense Scene Flow from Sparse or Dense Stereo Data; 10th European Conference on Computer Vision; Marseilles Executed "known method.

At the same time, the distance change to the cameras is calculated for each pixel BP1, BP2 by calculating the SceneFlow 1.1 . 1.2 determined. For example, in the error-free case, the plurality of pixels BP1, BP2 are static while only a few pixels BP1, BP2 are moving. In this case, a self-motion of the vehicle is compensated in the evaluation, wherein the proper motion in particular from the image data of the cameras 1.1 . 1.2 is appreciated.

In an error case, in which, for example due to a synchronization error, the image B1 of the camera 1.1 Is "frozen", ie that a time recorded later resembles a previously acquired image, is in the calculation of the SceneFlows a large change in distance of a large Part of the pixels BP1, BP2 detected, wherein the change in distance takes place in particular with a high rate of change. This indicates a sensor error, since such a speed jump between the two evaluation steps, ie between the images B1, B2 the two successive captured image pairs, is virtually impossible. For example, the so-called median evaluation of the speed of the world points in the direction of travel of the vehicle, ie the median evaluation of the rate of change of the pixels BP1, BP2 in the direction of travel of the vehicle, is suitable for analyzing and evaluating the SceneFlow results and consequentially for reliable detection of the error.

to Determine the error based on the calculation of the SceneFlow Therefore, a distance change limit A and a speed limit G predetermined for the change in distance, the Error especially when the distance change limit value is exceeded A and a speed limit G is detected. With others Words, when a pixel BP1, BP2 between the first time and the second time point such that the distance change limit value A and the speed limit G exceeded during the movement be, there is an error and is detected.

As already explained, the self-motion of the vehicle can be determined from the stereo sequences by means of a multiplicity of known methods. Particularly preferred for determining the intrinsic motion of the vehicle is a " H. Badino: A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform; International Workshop on Complex Motion; Günzburg 2004, Germany "Known method used.

However, in the illustrated error in which the plurality of pixels BP1, BP2 appear to be moved, an error-free estimation of the proper motion is not possible. For this reason, the proper motion determined from the image data, ie from the disparities and / or the scene flow, is additionally associated with a sensor-determined proper motion of the sensor arrangement 1 , ie the vehicle compared and optionally corrected, wherein the sensed self-motion is determined by means of an inertial sensor detected characteristics.

The inertial sensors are, in particular, motion sensors, for example a speed sensor, a yaw rate sensor, a steering angle sensor, a transverse and / or longitudinal acceleration sensor. This comparison results in a very simple test of the entire stereo camera arrangement 1 , Is the image-based self-motion estimation part of the image analysis, z. B. because the determined quantities are more accurate than the inertial sensors, the test of the stereo camera assembly 1 be carried out with very little effort, since no significant additional processing power is required.

LIST OF REFERENCE NUMBERS

1

Stereo Camera System

1.1

camera

1.2

camera

1.3

processing unit

A

distance limit

B1, B2

image

BP1, BP2

pixel

D

disparity

G

speed limit

P1, P2

criminal term

QUOTES INCLUDE IN THE DESCRIPTION

This list The documents listed by the applicant have been automated generated and is solely for better information recorded by the reader. The list is not part of the German Patent or utility model application. The DPMA takes over no liability for any errors or omissions.

Cited patent literature

• - DE 10302671 A1 [0002]
• - DE 10149115 A1 [0003]

Cited non-patent literature

• H. Hirschmüller: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information; 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, pp. 807-814 [0016]
• H. Hirschmüller: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information; 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Volume 2, pp. 807-814 [0017]
• - Wedel et al .: Efficient Dense Scene Flow from Sparse or Dense Stereo Data; 10th European Conference on Computer Vision; Marseille [0020]
• - H. Badino: A Robust Approach for Ego-Motion Estimation Using a Mobile Stereo Platform; International Workshop on Complex Motion; Günzburg 2004, Germany [0024]

Claims (6)

Method for monitoring a stereo camera arrangement ( 1 ), whereby by means of two cameras ( 1.1 . 1.2 ) as image data pixels (BP1, BP2) are detected, which by means of a processing unit ( 1.3 ) are processed stereoscopically to each other corresponding pixels (BP1, BP2) of a pair of images, which are detected in the detection and / or processing mutually corresponding pixels (BP1, BP2) occurring errors, characterized in that in a plausibility check a real movement of the stereo camera assembly ( 1 ) is compared with a movement determined from the image data on the basis of disparity values and / or optical flow data, wherein deviations between the real movement and the determined movement are detected and monitored. A method according to claim 1, characterized in that at least two temporally successive image pairs are analyzed such that a change in distance of basis of temporally successive detected pixels (BP1, BP2) determined point distances to the cameras ( 1.1 . 1.2 ) and this change in distance is monitored for an exceeding of a predetermined distance change limit value (A) and / or a predetermined speed limit value (G) with which the change in distance takes place. A method according to claim 1 or 2, characterized in that a determined from the image data proper motion of the stereo camera arrangement ( 1 ) is estimated. A method according to claim 3, characterized in that the determined from the image data proper motion of the stereo camera arrangement ( 1 ) is compensated in the determination of the point distances and the change in distance. Method according to claim 3 or 4, characterized that determined from the image data own motion with a compared determined by at least one motion sensor own motion and corrected as appropriate. Use of a method according to one of the claims 1 to 5 in a vehicle or a robot.

Images