JP7232005B2 - VEHICLE DRIVING ENVIRONMENT DETECTION DEVICE AND DRIVING CONTROL SYSTEM - Google Patents

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a vehicle driving environment detection device and a driving control system for recognizing a vehicle driving environment based on an external image of the vehicle's external environment.

In vehicles such as automobiles, the camera captures the scenery outside the vehicle in front and recognizes the driving environment by image processing, avoiding collisions with obstacles in front, tracking control for the preceding vehicle, warning control and steering control for swaying and lane departure. Various support controls for drivers, such as vehicles, have been put into practical use, and automatic driving technology that does not require the driver's driving operation is being put into practical use.

In travel control including such automatic driving, it is important to accurately detect the distance to an object in the travel environment. For example, in Japanese Unexamined Patent Application Publication No. 2004-100000, an object that has moved outside the angle of view of a first sensor and has moved into the angle of view of a second sensor is calculated relative to the object on the basis of previously stored feature amounts of the object. A technique for accurately detecting objects around an automobile is disclosed.

JP 2008-123462 A

However, in travel control using camera images, it becomes difficult to continue control when an abnormality is detected in image recognition, and countermeasures are required from the viewpoints of redundancy and robustness. In particular, with a stereo camera that can calculate the distance to an object, if one of the stereo images shows an object that is not shown in the other image, for example, if the stereo camera is installed inside a vehicle, dirt on the glass can be detected. In some cases, raindrops that cannot be wiped off by the wiper may appear in one of the images, and an image abnormality may temporarily occur in which the corresponding positions between the images cannot be detected by stereo matching. If such an image abnormality occurs in the stereo camera, even if the image abnormality is temporary, the traveling control based on the distance information of the object is interrupted, and there is a possibility that the control stability is lost.

The present invention has been made in view of the above circumstances, and is capable of continuing running control based on distance information of an object even if a temporary image abnormality occurs in the image of the stereo camera, thereby improving control redundancy and robustness. It is an object of the present invention to provide a vehicle running environment detection device and a running control system that can be improved.

A vehicle running environment detection device according to a first aspect of the present invention is a vehicle running environment detection device for recognizing a vehicle running environment based on an outside image obtained by imaging the outside environment of the vehicle, wherein the outside image is a stereo image. a stereo image processing unit that performs stereo image processing on a pair of images captured by a camera and captured by the stereo camera, recognizes an object in the driving environment, and calculates distance information of the object; an image abnormality detection unit that detects an image abnormality that occurs in one of the pair of images obtained; an image feature storage unit that stores a feature amount on the image of the object recognized by the stereo image processing unit; When the image abnormality is detected by the image abnormality detection unit, distance information calculated by applying the feature amount of the object to the normal monocular image of the pair of images of the stereo camera, and the monocular image. and a monocular image processing unit for calculating distance information of the object based on the distance information calculated from only the object.
A vehicle running environment detection device according to a second aspect of the present invention is a vehicle running environment detection device for recognizing a vehicle running environment based on an outside image obtained by capturing the outside environment of the vehicle, wherein the outside image is a stereo image. a stereo image processing unit that performs stereo image processing on a pair of images captured by a camera and captured by the stereo camera, recognizes an object in the driving environment, and calculates distance information of the object; an image abnormality detection unit that detects an image abnormality that occurs in one of the pair of images obtained; an image feature storage unit that stores a feature amount on the image of the object recognized by the stereo image processing unit; When the image abnormality is detected by the image abnormality detection unit, the distance information of the object is obtained by applying the feature amount of the object to the normal monocular image of the pair of images of the stereo camera. and a monocular image processing unit for calculating, wherein the image feature storage unit stores the feature amount of the target object in association with the distance information of the target object.
A vehicle running environment detection device according to a third aspect of the present invention is a vehicle running environment detection device for recognizing a vehicle running environment based on an outside image obtained by capturing the outside environment of the vehicle, wherein the outside image is a stereo image. a stereo image processing unit that performs stereo image processing on a pair of images captured by a camera and captured by the stereo camera, recognizes an object in the driving environment, and calculates distance information of the object; an image abnormality detection unit that detects an image abnormality that occurs in one of the pair of images obtained; an image feature storage unit that stores a feature amount on the image of the object recognized by the stereo image processing unit; When the image abnormality is detected by the image abnormality detection unit, the distance information of the object is obtained by applying the feature amount of the object to the monocular image of the normal one of the pair of images of the stereo camera. and a monocular image processing unit that calculates the distance information calculated only from the monocular image with respect to the distance information calculated by applying the feature amount after the image abnormality is detected. The distance information of the object is calculated by weighting according to the elapsed time of .

A vehicle running control system according to one aspect of the present invention is a vehicle running environment detection device according to any one of the first to third aspects. Distance information calculated by a first travel control unit that performs travel control including deceleration and steering, and a monocular image processing unit in the vehicle travel environment detection device according to any one of the first to third aspects. and a second travel control unit that performs travel control including acceleration/deceleration and steering based on the control unit.

According to the present invention, even if a temporary image abnormality occurs in the image of the stereo camera, it is possible to continue the travel control based on the distance information of the object, and it is possible to improve the redundancy and robustness of the control. can.

Configuration diagram showing a vehicle running control system Explanatory diagram of distance measurement using a stereo camera Explanatory diagram showing projection of objects at different distances onto an imaging plane Explanatory diagram showing the relationship between object size and distance Explanatory diagram showing calculation of road surface height Explanatory diagram showing distance measurement of a monocular image combined with stereo image processing information Flowchart showing travel control switching process

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. In FIG. 1, reference numeral 100 denotes a vehicle (not shown) running control system, which includes a running environment detection device 10 for detecting the running environment of the vehicle, and driving assistance and automatic driving based on information from the running environment detection device 10. and a running control device 50 that executes running control including the main part. The driving environment detection device 10 processes an image of the scene outside the vehicle captured by the stereo camera 1, calculates the distance to the object, and obtains information such as the size and shape of the object to be recognized based on the calculated distance. Output to the travel control device 50 .

In the present embodiment, the driving environment detection device 10 includes a processing system that performs stereo image processing on the images of the two cameras 1a and 1b that constitute the stereo camera 1, and a processing system that performs stereo image processing when the stereo image processing cannot be performed normally. Furthermore, it has a processing system for performing monocular image processing using stereo image processing information stored in the normal state on the image of whichever of the cameras 1a and 1b is normal. Specifically, the driving environment detection device 10 includes a stereo camera 1, image detection units 2a and 2b, an image abnormality detection unit 3, a stereo image processing unit 4, an image feature storage unit 5, and a monocular image processing unit 6. .

The stereo camera 1 is composed of a pair of shutter-synchronized cameras 1a and 1b each having an imaging element such as a CCD or a CMOS. In this embodiment, the cameras 1a and 1b have their optical axes parallel to each other, and their imaging surfaces are aligned. are mechanically and electrically adjusted to the arrangement Such a stereo camera 1 is formed as a camera unit in which two cameras 1a and 1b are fixed with a predetermined base line length, and is installed, for example, in the vicinity of the rearview mirror inside the front window in the upper part of the passenger compartment.

The image detection units 2a and 2b perform preprocessing and various correction processes on the cameras 1a and 1b to detect original images for stereo image processing. This original image is represented by an ij coordinate system with the lower left corner of the image as the origin, the i coordinate axis in the horizontal direction, and the j coordinate axis in the vertical direction. , a pair of camera images (digital images) are stored at predetermined addresses in the input image memory.

For example, as preprocessing for the cameras 1a and 1b, the image detection units 2a and 2b include electronic shutter control of the cameras 1a and 1b, gain/offset control and adjustment of amplifiers, gamma correction using a lookup table (LUT), and the like. Brightness correction, shading correction, etc. are performed. The image detection units 2a and 2b also perform parameter setting for setting various parameters in image processing coordinate setting, image size adjustment, address control, etc., and optical positional deviation including lens distortion corresponding to a pair of captured images. Correction processing such as affine correction for correction and filter correction for noise removal processing is performed.

The image abnormality detection unit 3 detects whether there is an abnormality in each of the images captured by the cameras 1a and 1b detected by the image detection units 2a and 2b. The image anomaly detected here is an anomaly that makes it difficult to perform stereo image processing normally. An image is detected as an image anomaly in which mismatching exceeding the permissible range occurs in stereo matching.

The image abnormality detection unit 3 instructs the stereo image processing unit 4 to perform stereo image processing when no abnormality is detected in the images of the cameras 1a and 1b. On the other hand, when an abnormality is detected in the image of either one of the cameras 1a and 1b, the image abnormality detection unit 3 stops the stereo image processing of the stereo image processing unit 4, and causes the monocular image processing unit 6 to Execution of monocular processing using the image of the normal one of the cameras 1a and 1b is instructed.

The stereo image processing unit 4 performs stereo matching processing on the pair of images of the cameras 1a and 1b detected by the image detection units 2a and 2b, and calculates the distance to the object. As is well known, in distance measurement using the stereo method, as shown in FIG. dpL + pixel shift amount on the reference image side dpR), the optical axis distance (base line length) between the reference camera that captures the reference image and the comparison camera that captures the comparison image is L, and the focal length of the lens is f. As shown below, the distance Z to the object P can be determined by equation (2) derived from the relationship of equation (1) based on the principle of triangulation.
Z: L = f: dp (1)
Z=L×f/dp (2)

The stereo image processing unit 4 obtains the amount of pixel shift (parallax) at corresponding positions between the reference image and the comparison image by stereo matching processing, and generates a distance image having distance data. As the stereo matching process, for example, a well-known region search method is used to evaluate the degree of correlation between the reference image and the comparison image, and as the evaluation function, a small region (main block) of the reference image and a small region (sub block) of the comparison image are used. Calculates the sum of pixel value differences (absolute values) (SAD: Sum of Absolute Difference).

Then, the position where the SAD is the smallest is determined as the corresponding position (matching point) where the degree of correlation between the main block and the sub-block is the highest. The difference between the position of the main block and the position of the sub-block in direction) gives the disparity with a resolution of one pixel (pixel disparity). A set of parallaxes calculated for each block is stored as distance data forming a distance image.

As the pixel value, the brightness value of each pixel is generally used in many cases. In addition, since the resolution of distance information based on pixel parallax decreases as the distance to the object increases, processing is performed to obtain sub-pixel level parallax with a resolution of one pixel unit or less as necessary.

A point on the distance image is coordinate-transformed into a point on the real space with the vehicle width direction, that is, the lateral direction, as the X axis, the vehicle height direction, as the Y axis, and the vehicle length direction, that is, the distance direction, as the Z axis. The stereo image processing unit 4 uses the distance image and the original image to three-dimensionally recognize objects such as preceding vehicles and obstacles traveling in front of the own vehicle, and uses the distance information to determine the size of the object. to detect The size of an object is detected as the horizontal distance between equidistant parts of the same object. The thickness (width) of the white line is detected as the size of the white line.

The size (width) between equidistant portions of the same object can be accurately calculated using the distance Z. That is, as shown in FIG. 3, an object of size (width) X1 at a distance Z1 and a size (width) X2 at a distance Z2 (Z2>Z1) can be seen on the imaging surface of the imaging device of the camera. Since the object is projected as an image of the same size θ0, the distance must be known in order to know the size of the object.

As shown in FIG. 4, the distance Z0 and the size X0 of the object are equivalent to the relationship between the focal length f of the camera and the size .theta.0 of the captured image, and these relationships are shown in equation (3). Therefore, the size X0 can be accurately calculated for an object whose distance Z0 (parallax dp) is known by the formula (4) derived from the formula (3).
Z0:X0=f:θ0 (3)
X0=Z0×θ0/f=L×θ0/dp (4)

The stereo image processing unit 4 also performs processing for recognizing white lines (lanes) on the road on which the vehicle is traveling. A road white line as a lane is recognized by extracting a point group that is a candidate for a white line from an image and calculating a straight line or curve connecting the candidate points. For example, in a white line detection area set on an image, edges whose luminance changes by a predetermined amount or more are detected on a plurality of search lines set in the horizontal direction (vehicle width direction), and one set of white lines is detected for each search line. A start point and a white line end point are detected, and an intermediate area between the white line start point and the white line end point is extracted as a white line candidate point.

Then, the time-series data of the spatial coordinate positions of the white line candidate points based on the amount of vehicle movement per unit time is processed to calculate a model that approximates the left and right white lines, and the white line is recognized using this model. As the approximation model of the white line, it is possible to use an approximation model obtained by connecting straight line components obtained by Hough transform, or a model approximating with a curve such as a quadratic equation.

In this embodiment, objects on the road and objects such as white lines are detected based on a road surface model that represents the height of the road surface according to the distance. The height h (Y coordinate value) of the point P on the road surface at the distance Zh is shown in FIG. It can be obtained by a linear expression shown in the following equation (5) derived from the vertical positional relationship as shown.
h=Zh×wj×(j−jv)/f+CAM_H (5)
However, wj: pixel pitch (vertical direction)
jv : Optical axis position (j coordinate of point at infinity in front of own vehicle)
CAM_H: Height from origin to camera center (lens center)

(5) The coefficient of the linear expression is identified by applying the least squares method to the point sequence obtained by calculating the coordinates (y, z) of each point P, as shown in the following equation (6) A road surface model is formed by a linear equation with slope Ha and intercept Hb.
Y=Ha×Z+Hb (6)

Then, the stereo image processing unit 4 associates the distance to the target object with the feature amount on the image for the target object such as the object above the road surface height and the white line extracted using the road surface model described above, and creates an image. Stored in the feature storage unit 5 . For example, the distance to the preceding vehicle and the brightness, brightness, saturation, and shape of the image of the preceding vehicle are associated with each other and stored in the image feature storage unit 5. ), lane width, etc. are stored in association with the distance of each point.

A monocular image processing unit 6 uses a monocular image captured by one of the cameras 1a and 1b to recognize objects in front of the vehicle and white lines on the road. The monocular image processing unit 6 performs, in parallel, a process in which an image feature amount based on the stereo image processing result is applied to the monocular image, and a process of only the monocular image.

In the process applying the image feature amount to the monocular image, the distance of the object is calculated using the size X0 (distance Z0) of the object by the stereo image processing calculated immediately before the occurrence of the image abnormality. That is, as shown by the dashed line in FIG. 6, when an object of size X0 is detected by stereo image processing, the same object of size θ can be detected from the monocular image by specifying the same object from the captured image of the monocular camera. The distance Z of the object projected as an image is calculated by the following equation (8) derived from the following equation (7) according to the equation (3).
Z: X0 = f: θ (7)
Z=X0×f/θ (8)

Whether or not the object on the monocular image is the same as the object of size X0 detected by the stereo image processing is determined by pattern matching or the like using the image feature quantity stored in the image feature storage unit 5. FIG. For example, when the brightness, brightness, saturation, and shape of the preceding vehicle on the monocular image are substantially the same as the image feature values obtained from the stereo image, the preceding vehicle on the monocular image has the size X0 calculated from the stereo image. Judge that it is the same as the object. Note that the distance of the white line in the monocular image is calculated using a road surface model calculated by stereo image processing.

On the other hand, the distance of the object from only the monocular image can be calculated using a well-known technique. For example, by extracting the edge of the image, pattern matching, optical flow of the image, etc., the area of the target object on the monocular image is extracted, and from the installation position and angle of the camera, which distance position each pixel on the image corresponds to is used to convert the position of the region on the image to the position on the real space.

The processing in the monocular image processing unit 6 is switched according to the elapsed time after the image abnormality detection unit 3 detects an abnormality. After the image abnormality is detected, until a predetermined set time (gradually decreasing time) Te elapses, the distance Z3 calculated by applying the image feature amount to the monocular image, for example, as shown in the following equation (9). and the distance Z4 calculated only from the monocular image are weighted according to the elapsed time t to calculate the distance Z. After the gradual decrease time Te is reached, the distance Z4 is calculated only from the monocular image. is carried out.
Z=(Te−t)/Te×Z3+t/Te×Z4 (9)

In this gradual decrease process, the processing result of the stereo image processing is largely reflected at first, and a relatively accurate distance value can be obtained. The data of the image feature quantity in the image becomes old and becomes unsuitable for control. For this reason, the gradual decrease time Te for performing the gradual decrease process is set to an appropriate time in consideration of changes in the running environment, the running state of the vehicle, and the like.

In contrast to the above-described driving environment detection device 10, the driving control device 50 that performs driving control based on information from the driving environment detection device 10 corresponds to two image processing systems, stereo image processing and monocular image processing. , a first travel control unit 51 and a second travel control unit 52 .

The first travel control unit 51 performs well-known travel control including automatic driving based on information from the stereo image processing unit 4 . For example, the first travel control unit 51 performs collision avoidance control for obstacles recognized by stereo image processing, automatic travel control along the target route, follow-up travel control for following the target route and the preceding vehicle, and lane control. Lane departure prevention control to prevent departure, warning control for swaying and lane departure, etc. are executed.

The second travel control unit 52 basically performs the same control as the first travel control unit 51, but since the accuracy of the distance information is relatively low, it is limited compared to the first travel control unit 51 control. For example, the second travel control unit 52 performs processing such as restricting automatic driving or driving assistance to the driver to narrow the control operation range, or stopping substantial control to stop the warning.

Specifically, the travel control described above is switched by the travel control switching process shown in the flowchart of FIG. Next, cruise control switching processing in the cruise control system 100 will be described using the flowchart shown in FIG.

In the travel control switching process, first, in step S1, it is checked whether or not the images of the cameras 1a and 1b constituting the stereo camera 1 have an image abnormality that prevents normal stereo matching. As a result, if both the images of the cameras 1a and 1b are normal, the process proceeds from step S1 to step S2. move on.

When the process proceeds to step S9 due to an image abnormality, monocular image processing is performed using the normal camera image, and running control is performed based on the monocular image processing. On the other hand, if the images of the cameras 1a and 1b are both normal, in step S2, stereo image processing is performed using the images of the cameras 1a and 1b, and three-dimensional information of the object obtained by this stereo image processing is used. Carry out travel control.

Next, proceeding from step S2 to step S3, the feature amount on the image of the object is stored in association with the distance information of the object obtained by the stereo image processing. Then, in step S4, it is checked whether there is any abnormality in the images of the cameras 1a and 1b. If the images from the cameras 1a and 1b are both normal, the process returns to step S2 to implement driving control based on stereo image processing. , S5B are processed in parallel.

Step S5A is a process of calculating the distance by applying the image feature amount based on the stereo image processing obtained immediately before the occurrence of the image abnormality to the monocular image. Using the obtained object size X0, the distance to the object is calculated. Step S5B is a process of calculating the distance using only the monocular image, and the distance is calculated from image edge extraction, pattern matching, optical flow, and the like.

Then, it progresses to step S6 and implements traveling control based on the distance information calculated by the gradual decrease process. Then, in step S7, it is checked whether or not the image abnormality of the cameras 1a and 1b is a temporary abnormality, and whether or not they have returned to normal.

If the images of the cameras 1a and 1b have returned to normal, the process returns to step S2 to resume the stereo image processing. Checks whether the elapsed time of has reached the decrement time. If the gradual decrease time has not been reached, the process returns from step S8 to step S6 to continue the gradual decrease process.

As described above, in the present embodiment, when the image abnormality detection unit 3 detects an abnormality in the image of one of the cameras 1a and 1b constituting the stereo camera 1, the stereo image processing by the stereo image processing unit 4 is stopped. In addition, in the monocular image processing unit 6, the image feature amount based on the processing result of the stereo image stored in the image feature storage unit 5 is applied to the image of the normal one of the cameras 1a and 1b to obtain the target object. Calculate the distance. As a result, even if a temporary image abnormality occurs in the image of the stereo camera, it is possible to continue the travel control based on the distance information of the object, and it is possible to improve the redundancy and robustness of the control.

1 stereo camera 1a, 1b camera 2a, 2b image detection unit 3 image abnormality detection unit 4 stereo image processing unit 5 image feature storage unit 6 monocular image processing unit 10 driving environment detection device 50 driving control device 51 driving control unit 52 driving control unit 100 Travel control system

Claims (5)

A vehicle driving environment detection device for recognizing a vehicle driving environment based on an image outside the vehicle obtained by capturing the external environment of the vehicle,
a stereo image processing unit that captures the image outside the vehicle with a stereo camera, performs stereo image processing on a pair of images captured with the stereo camera, recognizes an object in the driving environment, and calculates distance information of the object;
an image abnormality detection unit that detects an image abnormality occurring in one of the pair of images captured by the stereo camera;
an image feature storage unit that stores a feature amount on the image of the object recognized by the stereo image processing unit;
When the image abnormality is detected by the image abnormality detection unit, distance information calculated by applying the feature amount of the object to the normal monocular image of the pair of images of the stereo camera, and the monocular image. and a monocular image processing unit that calculates the distance information of the object based on the distance information calculated from the distance information calculated from the distance information . A vehicle driving environment detection device for recognizing a vehicle driving environment based on an image outside the vehicle obtained by capturing the external environment of the vehicle,
a stereo image processing unit that captures the image outside the vehicle with a stereo camera, performs stereo image processing on a pair of images captured with the stereo camera, recognizes an object in the driving environment, and calculates distance information of the object;
an image abnormality detection unit that detects an image abnormality occurring in one of the pair of images captured by the stereo camera;
an image feature storage unit that stores a feature amount on the image of the object recognized by the stereo image processing unit;
When the image abnormality is detected by the image abnormality detection unit, the distance information of the object is obtained by applying the feature amount of the object to the normal monocular image of the pair of images of the stereo camera. A monocular image processing unit that calculates
The vehicle running environment detection device, wherein the image feature storage unit stores the feature amount of the object in association with distance information of the object. A vehicle driving environment detection device for recognizing a vehicle driving environment based on an image outside the vehicle obtained by capturing the external environment of the vehicle,
a stereo image processing unit that captures the image outside the vehicle with a stereo camera, performs stereo image processing on a pair of images captured with the stereo camera, recognizes an object in the driving environment, and calculates distance information of the object;
an image abnormality detection unit that detects an image abnormality occurring in one of the pair of images captured by the stereo camera;
an image feature storage unit that stores a feature amount on the image of the object recognized by the stereo image processing unit;
When the image abnormality is detected by the image abnormality detection unit, the distance information of the object is obtained by applying the feature amount of the object to the normal monocular image of the pair of images of the stereo camera. A monocular image processing unit that calculates
The monocular image processing unit weights the distance information calculated only from the monocular image with respect to the distance information calculated by applying the feature amount according to the elapsed time after the image abnormality is detected. and a vehicle traveling environment detection device, which calculates distance information of the object. The monocular image processing unit specifies an object on the monocular image based on the feature amount stored in the image feature storage unit before the image abnormality is detected, and calculates distance information of the specified object. 4. The vehicle driving environment detection device according to claim 1, wherein the vehicle driving environment detecting device calculates the driving environment. A first travel control that performs travel control including acceleration/deceleration and steering based on the distance information calculated by the stereo image processing unit in the vehicle travel environment detection device according to any one of claims 1 to 4. Department and
A second travel control that performs travel control including acceleration/deceleration and steering based on the distance information calculated by the monocular image processing unit in the vehicle travel environment detection device according to any one of claims 1 to 4. A vehicle travel control system comprising: a section;

Images