JP5863536B2 - Outside monitoring device - Google Patents

Description

  The present invention relates to an out-of-vehicle monitoring apparatus that recognizes an out-of-vehicle object based on a pair of images captured in stereo from different viewpoints.

  2. Description of the Related Art Conventionally, as a vehicle outside monitoring device for recognizing an object outside a vehicle, a stereo type vehicle outside monitoring device using a pair of in-vehicle cameras is known. In this type of out-of-vehicle monitoring device, first, a pixel block having a correlation with a pixel block in one image (reference image) is specified in the other image (comparison image) by stereo matching processing, and the relative relationship between the two is compared. The parallax that is the amount of deviation is calculated. Then, distance data is calculated from the calculated parallax for each pixel block using the principle of triangulation, and a known grouping process or the like is performed on the distance data, thereby obtaining an object ( Lane markings outside the vehicle, vehicles, traffic lights, etc.) are recognized.

  By the way, when performing various types of driving support control using information outside the vehicle recognized by the vehicle outside monitoring device, not only information about the position, size, range, etc. that each object occupies in the real space, but also color information of the object. It is desirable to be acquired also about etc. For example, when recognizing a vehicle as an object, if it is possible to acquire color information when an in-vehicle lamp is turned on, it is determined whether the lamp is a headlamp, tail lamp, brake lamp, or turn signal lamp. It is possible to easily determine whether the vehicle is an oncoming vehicle or a preceding vehicle, whether the vehicle is a right turn (left turn), or the like. Also, for example, when a traffic light is recognized as an object, if it is possible to acquire color information of a region that emits light on the traffic light, it is possible to accurately determine whether the traffic light is a red signal or a blue signal. . And if these recognition results are reflected in driving support control etc., it will become possible to implement more precise and precise control.

  On the other hand, when acquiring color information of a region that emits light with high brightness on an object such as a vehicle or a traffic light as described above, it is necessary to control the exposure amount of the in-vehicle camera to be small in order to prevent luminance saturation of the region. However, when the exposure amount is controlled to be small as described above, the recognition accuracy of other objects may be lowered. On the other hand, for example, as disclosed in Patent Document 1, it is conceivable to sequentially change the exposure timing according to the detection target.

JP 2008-207777 A

  However, when the exposure timing is changed for each detection target as in the technique disclosed in Patent Document 1 described above, the number of frames that can recognize the symmetry outside the vehicle as a whole may decrease. .

  The present invention has been made in view of the above circumstances, and is capable of accurately acquiring color information without causing a decrease in recognition accuracy or a decrease in the number of frames when recognizing an object outside the vehicle. An object is to provide an apparatus.

  A vehicle exterior monitoring apparatus according to an aspect of the present invention uses a first imaging unit and a second imaging unit to capture an object outside a vehicle from different viewpoints, and outputs a reference image and a comparative image having parallax to each other. An imaging unit, a color imaging unit that performs color imaging of an object outside the vehicle with an exposure amount lower than that of the first imaging unit and the second imaging unit, and outputs a reference image corresponding to the reference image; and the reference image A distance image generating means for generating a distance image in which distance information is given to pixels on the reference image by performing a stereo matching process between the pixel on the reference image and a pixel on the reference image. Color information providing means for providing color information.

  According to the vehicle exterior monitoring device of the present invention, it is possible to accurately acquire color information without causing a decrease in recognition accuracy or a decrease in the number of frames when recognizing an object outside the vehicle.

Schematic configuration diagram showing a vehicle driving support device The perspective view which shows schematic structure of a camera unit Flow chart showing image recognition processing routine Explanatory drawing which shows an example of a standard image and a reference image typically Explanatory drawing showing stereo matching processing Explanatory drawing of the evaluation method of the luminance edge in the horizontal direction in the pixel block Explanatory drawing when color information is given from the reference image to the distance image

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The drawings relate to an embodiment of the present invention, FIG. 1 is a schematic configuration diagram showing a vehicle driving support device, FIG. 2 is a perspective view showing a schematic configuration of a camera unit, and FIG. 3 is a flowchart showing an image recognition processing routine. 4 is an explanatory diagram schematically showing an example of a standard image and a reference image, FIG. 5 is an explanatory diagram showing a stereo matching process, FIG. 6 is an explanatory diagram of a method for evaluating a luminance edge in a horizontal direction in a pixel block, and FIG. It is explanatory drawing at the time of giving color information to a distance image from a reference image.

  In FIG. 1, reference numeral 1 denotes a vehicle such as an automobile (own vehicle), and a vehicle driving support device 2 is mounted on the vehicle 1. The driving support device 2 includes a stereo camera 3, a monocular color camera 4 provided along with the stereo camera 3, an image recognition device 5, and a control unit 6.

  Further, the vehicle 1 is provided with a vehicle speed sensor 11 for detecting the host vehicle speed V, a main switch 13 to which an ON-OFF signal of the driving support device is input, and the like, and the ON / OFF of the host vehicle speed V and the driving support control is provided. Signals and the like are input to the control unit 6.

  The stereo camera 3 includes a pair of left and right CCD cameras (color or monochrome cameras) 3a and 3b using a solid-state imaging device such as a charge coupled device (CCD) as a stereo optical system. These left and right CCD cameras 3a and 3b are respectively mounted at a certain interval in front of the ceiling in the vehicle interior, and images the object outside the vehicle from different viewpoints. In the present embodiment, the camera 3a is attached as a main camera on the right side in the traveling direction of the vehicle 1 and outputs a reference image. on the other hand. The camera 3b is attached as a sub camera on the left side in the traveling direction, and outputs a comparative image.

  The monocular color camera 4 is configured by a color camera using a solid-state imaging device such as a charge coupled device (CCD), for example. For example, as shown in FIG. 2, the monocular color camera 4 of the present embodiment constitutes a camera unit 7 that is held on the same stay 7a as the main camera 3a and the sub camera 3b. In this case, the monocular color camera 4 is fixed to the stay 7a at a position adjacent to the main camera 3a in the horizontal direction, for example. As a result, the monocular color camera 4 captures a color image of an object outside the vehicle from substantially the same viewpoint as the main camera 3a, and outputs a reference image corresponding to the reference image.

  Here, the main camera 3a, the sub camera 3b, and the monocular color camera 4 are synchronized with each other. Further, the exposure amount at the time of imaging by the stereo camera 3 (the main camera 3a and the sub camera 3b) and the monocular color camera 4 is variably set based on, for example, a preset map or the like. That is, the exposure amount (shutter speed, aperture amount, etc.) of the stereo camera 3 and the monocular color camera 4 is obtained by, for example, adding up the map values for each region obtained using the average luminance value of each region of the captured image as a parameter. It is determined. At this time, the exposure amount of the monocular color camera 4 is set to be relatively lower than the exposure amounts of the main camera 3a and the sub camera 3b, thereby suppressing the occurrence of a saturated luminance region on the reference image. For example, as shown in FIG. 4A, the reference image (and the comparison image) is also referred to when luminance saturation occurs in some light emitting areas such as the vehicle 100 and the traffic light 101 that are captured. On the image, saturation of luminance in the same region is suppressed (see FIG. 4B).

  Thus, in this embodiment, the stereo camera 3 implement | achieves the function as a stereo imaging means, and the monocular color camera 4 implement | achieves the function as a color imaging means.

  The image recognition device 5 generates a distance image in which distance information is added to pixels on the reference image by performing a stereo matching process between the reference image and the comparison image. Further, the image recognition device 5 gives color information of the corresponding pixel on the reference image to the pixel on the generated distance image. Then, the image recognition device 5 recognizes various three-dimensional objects and the like existing in front of the host vehicle 1 by performing a well-known grouping process or the like on the distance image to which the color information is given, and in front of the host vehicle 1. If there is a vehicle that satisfies a preset condition, the vehicle is detected as a preceding vehicle.

  Thus, in this embodiment, the image recognition device 5 realizes functions as a distance image generation unit and a color information addition unit.

  The control unit 6 performs, for example, cruise control with inter-vehicle distance control (ACC: Adaptive Cruise Control) as driving support control based on information outside the vehicle recognized by the image recognition device 5. Here, for example, the host vehicle speed V is input from the vehicle speed sensor 11 to the control unit 6, and a set vehicle speed Vset during constant speed driving is input through various switches provided alongside the main switch 13.

  In the state where the main switch 13 is turned on, the control unit 6 sets the set vehicle speed Vset set by the driver as the target vehicle speed during constant speed traveling control in which no preceding vehicle is detected, and the opening degree of the electronically controlled throttle valve 15 By performing control (engine output control), the own vehicle speed V is converged to the target vehicle speed. Further, when the control unit 6 determines that sufficient deceleration cannot be obtained only by engine output control, the control unit 6 also uses the output hydraulic pressure (automatic brake intervention control) from the active booster 16 to target the vehicle speed V. Let it converge to the vehicle speed.

  Further, when the control unit 6 detects the preceding vehicle on the own vehicle traveling path during the constant speed traveling control, the control unit 6 shifts to the following traveling control. When the control unit 6 shifts to the follow-up running control, the control unit 6 sets the target vehicle speed based on, for example, the follow-up target inter-vehicle distance that is variably set according to the own vehicle speed V, and controls the opening of the electronic throttle valve 15. Thus, the inter-vehicle distance is converged to the following target inter-vehicle distance. Furthermore, when the control unit 6 determines that sufficient deceleration cannot be obtained only by engine output control, the control unit 6 also uses control of the output hydraulic pressure from the active booster 16 to converge the inter-vehicle distance to the target tracking distance. .

  Next, image recognition processing executed in the image recognition device 5 will be described in detail according to the flowchart of the image recognition processing routine shown in FIG. This routine is repeatedly executed every set time. When the routine starts, the image recognition apparatus 5 first compares the reference image captured by the main camera 3a and the sub camera 3b of the stereo camera 3 in step S101. Perform stereo matching with the image.

  That is, the image recognition device 5 calculates distance data in units of pixel blocks composed of 4 × 4 pixels, for example, based on the reference image data and the comparison image data input from the stereo camera 3. More specifically, the image recognition device 5 extracts one pixel block PB1 from the reference image data, and searches for and specifies a pixel block having a correlation between the luminance characteristics of the pixel block PB1 and the comparison image data. (See FIG. 5). As is well known, the distance between objects displayed in a stereo image appears as a parallax in the stereo image, that is, a horizontal shift amount related to the positions of the reference image and the comparison image. Accordingly, when specifying a pixel block having a correlation with the target pixel block PB1, if a pixel block on the horizontal line (epipolar line) having the same value as the j coordinate value of the target pixel block PB1 is detected on the comparison image data, Good. The image recognition device 5 evaluates the correlation with all the comparison pixel blocks in a predetermined pixel range on the epipolar line by shifting the epipolar line one pixel at a time. Then, the image recognition device 5 obtains the parallax between the pixel block PB1 on the reference image and the pixel block for which the correlation between the pixel block PB1 is specified on the comparison image.

  When the process proceeds from step S101 to step S102, the image recognition apparatus 5 refers to the reference image data and evaluates the luminance edge (luminance change amount) in the horizontal direction of the pixel block, thereby filtering the parallax calculated by the stereo matching process. Apply processing. More specifically, as shown in FIG. 6, for example, the image recognition device 5 uses a pixel block that is a unit for calculating distance data in a luminance change amount (absolute value) ΔPn (1) of a pixel pair adjacent in the horizontal direction. ≦ Pn ≦ 16) is calculated. However, for the leftmost pixel column (P11, P12, P13, P14), the luminance change amount ΔP is calculated from the rightmost pixel column of the pixel block adjacent to the left side. Next, the image recognition apparatus 5 counts the number of luminance change amounts ΔPn that are greater than or equal to the luminance determination threshold DCDXth. Then, the image recognition apparatus 5 extracts a pixel block having a large luminance change amount by excluding parallax related to a pixel block having a luminance change amount ΔP equal to or greater than the luminance determination threshold DCDXth less than a set number (for example, four). Then, only the parallax related to the extracted pixel block is output as distance data. In other words, even if a pixel block having a small amount of luminance change in the horizontal direction and not having a characteristic of luminance is matched in stereo matching processing, the reliability of the parallax is often low. Therefore, the reliability of the distance data is improved by removing the calculated parallax for such a pixel block. Then, by obtaining the distance data in this way, the image recognition device 5 generates a distance image in which distance information is given to the pixels on the reference image.

  When the process proceeds from step S102 to step S103, the image recognition device 5 gives color information of the corresponding pixel on the reference image captured from substantially the same viewpoint to the pixel on the distance image (see FIG. 7).

  Then, when the process proceeds from step S103 to step S104, the image recognition apparatus 5 exits the routine after performing solid object recognition on the distance image. That is, the image recognition device 5 performs a known grouping process on the distance data on the generated distance image. Here, in the image recognition device 5, for example, frames (windows) such as three-dimensional road shape data, side wall data, and three-dimensional object data are preset and stored. The image recognition device 5 compares the grouped distance data group and each window, thereby extracting lane marking data, side wall data such as guardrails and curbs along the road, and 2D objects. It is classified and extracted into various other three-dimensional objects such as wheels, ordinary vehicles, large vehicles, pedestrians, utility poles, and traffic lights. Then, the image recognition device 5 gives color information obtained from the reference image to each extracted three-dimensional object.

  Further, the image recognition device 5 estimates the own vehicle travel route based on the recognized information such as the lane markings and the side walls, and further performs a detection process of the preceding vehicle on the estimated own vehicle travel route. That is, when a three-dimensional object (forward moving object) having a predetermined speed component or more in the same direction as the traveling direction of the vehicle exists on the traveling path of the vehicle, the forward moving object closest to the own vehicle 1 is selected from these. It detects (captures) a preceding vehicle and acquires various preceding vehicle information (such as the inter-vehicle distance, the preceding vehicle speed, and the preceding vehicle deceleration).

  According to such an embodiment, the stereo camera 3 (the main camera 3a and the sub camera 3b) is used to capture an image of an object outside the vehicle from different viewpoints, and output a reference image and a comparative image having parallax with each other. In an out-of-vehicle monitoring apparatus that generates a distance image in which distance information is given to pixels on a reference image by performing stereo matching processing of the image, a monocular color camera 4 that captures an image with an exposure amount lower than that of the main camera 3a and the sub camera 3b. By providing a stereo camera 3 together with the monocular color camera 4 to capture an object outside the vehicle and outputting a reference image, and adding color information of the pixel on the reference image to the pixel on the distance image, It is possible to accurately acquire color information without causing a reduction in recognition accuracy or a reduction in the number of frames when recognizing the target object.

  That is, by separately acquiring the color information of the object outside the vehicle with the monocular color camera 4 provided side by side, it is possible to capture the reference image and the comparative image with a sufficient exposure amount using the main camera 3a and the sub camera 3b. Various three-dimensional objects and the like can be accurately recognized from the distance image obtained using these. Further, by making the monocular color camera 4 share the acquisition of color information such as a three-dimensional object, it is possible to accurately recognize the situation outside the vehicle that changes from moment to moment without lowering the frame rate.

  In this case, by arranging the monocular color camera 4 adjacent to the main camera 3a so that the parallax between the monocular color camera 4 and the main camera 3a is substantially zero, the correspondence between the pixels of the reference image and the reference image The color information of the reference image can be given to the distance image without requiring complicated calculation or the like.

  In addition, by using the three-dimensional object information outside the vehicle configured including the color information in the above-described driving support control, it is possible to realize a precise driving support control with high accuracy. For example, when a traffic light is recognized in front of the host vehicle traveling path during the execution of the ACC, when the traffic signal is a red signal, the acceleration of the host vehicle 1 is limited to meet the driver's feeling. ACC can be realized. Similarly, for example, when a preceding vehicle is recognized, if a brake light that emits red light with high brightness is detected on the preceding vehicle, it is determined that the preceding vehicle is in a decelerating state and the vehicle 1 By limiting the acceleration, an ACC that matches the feeling of the driver can be realized.

1 ... Vehicle (own vehicle)
DESCRIPTION OF SYMBOLS 2 ... Vehicle driving assistance device 3 ... Stereo camera 3a ... Main camera 3b ... Sub camera 4 ... Monocular color camera 5 ... Image recognition device 6 ... Control unit 7 ... Camera unit 7a ... Stay 11 ... Vehicle speed sensor 13 ... Main switch 15 ... Electronically controlled throttle valve 16 ... Active booster

Claims (2)

Stereo imaging means for imaging an object outside the vehicle from different viewpoints using the first imaging means and the second imaging means, and outputting a reference image and a comparative image having parallax with each other;
Color imaging means for subjecting the object outside the vehicle to color imaging with an exposure amount lower than that of the first imaging means and the second imaging means, and outputting a reference image corresponding to the reference image;
Distance image generation means for generating a distance image in which distance information is given to pixels on the reference image by performing a stereo matching process between the reference image and the comparison image;
An out-of-vehicle monitoring device comprising: color information providing means for providing color information of a pixel on the reference image to a pixel on the distance image.   2. The out-of-vehicle monitoring device according to claim 1, wherein the color imaging unit is disposed adjacent to the first imaging unit.

Images