JP5145585B2 - Target detection device - Google Patents

Description

  The present invention relates to a target detection apparatus that detects a target using a laser radar and a stereo camera or an optical time-of-flight distance sensor.

  In order to solve problems such as traffic accidents, traffic jams, fuel consumption, exhaust gas, etc. in the automobile society, by providing advanced intelligence to automobiles, and supporting and automating driving by drivers, it is safe and smooth. There is a possibility that automobile traffic in harmony with the environment can be realized. Such intelligent and safety systems for automobiles are actively researched and developed.

  Pedestrian detection is an important basic technology in active safety of automobiles. If a pedestrian can be detected in advance, it is possible to reduce the speed of the automobile before the collision between the automobile and the pedestrian, operate a device that protects the pedestrian, and reduce the degree of injury to the pedestrian. . Also, in a road environment where there are pedestrians around the vehicle, it is possible to stop the automatic follow-up function with the risk of a pedestrian accident and switch to driver-driven driving.

  Conventionally, systems such as obstacle detection and on-vehicle distance maintenance using in-vehicle sensors such as radar and ultrasonic waves have been researched and developed, and technologies such as white line recognition using image sensors and detection of preceding vehicles are being put into practical use. . However, the radar sensor cannot classify objects (pedestrians, bicycles, signboards, stopped vehicles, etc.), and there is a problem that the running environment recognition technology is poor. Furthermore, since the detection area is narrow, it is difficult to measure overtaking vehicles from adjacent lanes or preceding vehicles on curved roads. On the other hand, the image sensor is excellent in the object recognition function such as whether the detection target is a vehicle, a pedestrian, or whether it is on the own lane, but the detection accuracy of the relative distance and the relative speed is low.

  In order to make up for the shortcomings of such a single sensor, various studies have been conducted on obstacle recognition by sensor fusion. Examples of sensor classification include millimeter wave radar, laser radar, ultrasonic sensor, monocular camera sensor, and stereo camera sensor.

  For example, the technique of Patent Document 1 includes image acquisition means using a millimeter wave radar and a stereo camera. In this technique, first, the image recognition area is specified based on the power output from the millimeter wave radar. Next, image processing such as edge detection and parallax calculation for object detection is executed only in the specified image recognition area to reduce the image processing time.

  Further, the technique of Patent Document 2 detects obstacles in traveling of a vehicle based on the output of a radar that detects the presence of objects around the vehicle, and determines the obstacle reference value detected by the radar using a stereo camera that captures the obstacles. Is provided with a judgment reference value changing means for changing. Thereby, based on the possibility of the presence of an obstacle, the judgment reference value is changed to enable appropriate obstacle recognition.

  In the technique of Patent Document 3, as much information as possible that can be acquired from the camera and the radar is used to eliminate misrecognition and perform more accurate sensor fusion. First, a fusion three-dimensional object at each sensor is set based on information from a stereo image and millimeter wave radar. Moreover, the same probability of the three-dimensional object acquired with the mutual sensor is calculated, and the fusion three-dimensional object by the combination of the image three-dimensional object and the millimeter wave three-dimensional object is determined. Next, in the fusion solid confirmation unit, determination is made for all three-dimensional objects of the fusion solid object by combining the three-dimensional fusion object of the image solid object, the fusion solid object of the millimeter wave solid object, and the combination of the image solid object and the millimeter wave solid object. Millimeter-wave solids that satisfy the conditions for ghost determination based on lateral responsiveness delay based on image information, multipath ghost determination based on wall reflection, and ghost determination based on reflected waves from the preceding vehicle The fusion solid object of a single object is determined to be a virtual image. The determination result of this virtual image is maintained for a set time, and the three-dimensional object determined as a virtual image in this way is excluded from the subsequent control targets.

  Moreover, in the technique of patent document 4, the three-dimensional coordinate of the point on the target object surface in a world coordinate system is calculated using a stereo sensor. By adding the number of projection points in the Z-axis direction, the feature quantity of the number of points is obtained in the evenly spaced grid occupation map. Further, it is determined whether or not a person exists using the feature amount.

  Further, in the technique of Patent Document 5, with respect to the resolution problem described above, measurement points on the object surface are projected onto a predetermined horizontal plane or vertical plane, respectively, and sensors that are defined in advance in the horizontal plane or vertical plane respectively. By accumulating the number of measurement points in the grid cell that are coarser than the measurement resolution, a clustering process is performed by binarization of the three-dimensional histogram to generate an obstacle map.

  In the technique of Patent Document 6, an obstacle is determined in consideration of the characteristics of the obstacle and the characteristics of the detection device, thereby enabling accurate determination of the object. By using a radar recognition result for an object with relatively high speed such as an oncoming vehicle or a preceding vehicle, these can be discriminated from a distance even when the visibility is poor due to rain or fog. For stationary objects (including low-speed moving objects), only objects recognized by both radar and image recognition are determined as obstacles, thereby preventing unnecessary detection from the viewpoint of the system. About a pedestrian, since it is difficult to detect with a radar, it can detect by determining using an image recognition result. Since pedestrians, particularly human bodies, clothes, and the like, do not have desirable millimeter wave reflection characteristics, it is difficult to stably detect them with millimeter wave radar. On the other hand, the image recognition means can detect stably. Therefore, what cannot be captured by millimeter waves and is recognized only by image recognition is secured as a single image target. Furthermore, the target itself is determined based on the moving speed and size. For example, when the traveling speed is less than 10 km / h and the settings such as the height of 50 to 250 cm and the width of 50 cm or less are satisfied, it is determined as a pedestrian. Moreover, regarding the object detected by both the millimeter wave and the image, erroneous detection is suppressed by making it an obstacle. This technology takes advantage of the advantages and disadvantages of millimeter waves and images by using the characteristics of these sensors to identify objects when detected using either millimeter wave radar or images. Sensor fusion can be performed.

JP 2001-296357 A JP-A-2005-345251 JP 2006-047057 A JP 2006-236025 A JP 2001-242934 A Japanese Patent Laid-Open No. 2005-202878

  However, with the technique of Patent Document 1, it is difficult to detect and recognize an object on which a millimeter wave radar signal such as a pedestrian is difficult to reflect. Moreover, the technique of patent document 2 recognizes only the presence of an obstacle on the road, and does not target a pedestrian or the like existing on a sidewalk or a roadside belt. In the technique of Patent Document 3, since a simple combination is performed without considering the detection accuracy of each sensor, it is difficult to detect an obstacle located far away. Further, due to the resolution of the radar sensor, obstacles that are close to each other cannot be separated. In the technique of Patent Document 4, since the resolution of stereo vision is inversely proportional to the distance due to the principle of triangulation, the projection accuracy on the occupation map of the object at a long distance is lowered and diverges. In the technique of Patent Document 5, similarly to the technique of Patent Document 4, the projection of objects at a long distance diverges, and the areas of the objects are connected on the occupation map, and cannot be easily divided. In the technique of Patent Document 6, in order to recognize as a single image target with high accuracy, an infrared image, stereo vision, and the like are used, and more information is required, which increases the cost and the calculation time. It becomes.

  The present invention has been made in view of such problems, and a first object thereof is to detect targets such as vehicles, pedestrians, and obstacles that are close to each other in a short time with a simple configuration. The object is to provide a possible target detection device. In addition, a second object is a target detection apparatus that can detect a target such as a vehicle, a pedestrian, and an obstacle existing in the distance of the vehicle with an accuracy equivalent to that of a target in the vicinity of the vehicle with a simple configuration. Is to provide.

A first target detection apparatus of the present invention includes a laser radar that acquires radar information in front of a vehicle, a stereo camera that acquires a stereo image in front of the vehicle, radar information acquired by a laser radar, and stereo acquired by a stereo camera. And a control unit for processing an image. Here, the control unit has the following means (A1) to (A7).
(A1) Each target candidate from the radar information, the stereo image, the first parallax derived from the radar information, and the second parallax derived from the stereo image or the second parallax corrected by the parallax correcting means The position, width, and height on the stereo image are derived, and the position etc. obtaining means for deriving the 3D coordinates of the 3D coordinate system in the stereo camera for each pixel of the stereo image (A2) Using the position, width, and height of each target candidate on the stereo image, the two-dimensional image of each target candidate region is cut out from the stereo image, and the parallax derived from the cut-out two-dimensional image is second After cutting out from parallax, out of the cut out two-dimensional image, out of the second parallax, the parallax derived from the cut out two-dimensional image and the cut out two-dimensional of the first parallax The stereo of each target candidate derived by the position acquisition means using the first partial image of the part that is the same or substantially the same as each other by comparing with the parallax of the target candidate in the region corresponding to the image Position etc. correction means for correcting the position, width and height on the image (A3) Type discrimination means for discriminating the type of each target candidate using the two-dimensional image or the first partial image cut out by the position etc. correction means. (A4) The position and width, height, type, two-dimensional image of the target candidate area of each target candidate whose type has been determined by the type determining unit and the target tracking unit, and a stereo camera Target list registration means for registering the three-dimensional coordinates of the three-dimensional coordinate system in the target list (A5) An area including the predicted position of each target candidate expected to exist in the newly acquired stereo image 2D images of Cut out from the newly acquired stereo image based on the position and width on the stereo image of each target candidate recorded in the list, the height, the type, and the 3D coordinates of the 3D coordinate system in the stereo camera. The correlation between each target candidate expected to be included in the cut-out two-dimensional image and each target candidate recorded in the target list is evaluated by a predetermined method. When it is evaluated that the correlation is extremely high, it is derived from the two-dimensional image of the region including the predicted position among the two-dimensional images of the region including the predicted position of the highly correlated target candidate (specific target). Of the first parallax derived from the newly acquired radar information and the parallax of the target candidate in the region corresponding to the two-dimensional image of the region including the predicted position, both are the same or substantially the same For the same parallax The position, width, and height of the specific target on the newly acquired stereo image are corrected using the corresponding second partial image, and the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera of the specific target are acquired. , Using the position and width and height on the newly acquired stereo image of the specific target obtained by the correction, cutting out the two-dimensional image of the target candidate area of the specific target from the newly acquired stereo image, In addition, if the correlation between the two is not very high but is evaluated as relatively high, the position of the target candidate (quasi-specific target) with relatively high correlation on the newly acquired stereo image and After acquiring the width and height, the two-dimensional image of the target candidate area, and the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera by performing the same process as the above process on the specific target Target tracking means for discriminating the type of the quasi-specific target using the acquired two-dimensional image of the target candidate area of the quasi-specific target (A6) In the target list, the specific target derived by the target tracking means Target list update means (A7) for updating the position and width on the stereo image, the height, the two-dimensional image of the target candidate area, and the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera. Of the second parallax derived from the stereo image, the first parallax derived from the parallax derived from the two-dimensional image of the target candidate area of the specific target derived by the target tracking means and the newly acquired radar information. A parallax in which both of them are the same or almost the same as the parallax of the target candidates in the area corresponding to the two-dimensional image of the target candidate area of the specific target clipped by the target tracking means. Parallax correcting means for changing to zero

  In the first target detection apparatus of the present invention, a laser radar is used in addition to the stereo camera in order to detect the target. As a result, it is possible to detect and recognize a target such as a pedestrian that is difficult to reflect a millimeter wave radar signal. Further, the position, width, and height of each target candidate derived by the position etc. acquiring unit on the stereo image are corrected using the first partial image. Here, since the first partial image corresponds to the two-dimensional image of the target candidate area obtained by removing the background image, the target candidate is mixed into the background image, and there is an error in the position of the target candidate. It can be eliminated. Further, when tracking the temporal change of each target candidate, not only the position and width of the target candidate, the height, and the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera are used, Since the type is also used, the detection and recognition of the target can be performed with high accuracy. In addition, when it is evaluated that the correlation is high, the two-dimensional image of the target candidate region of the specific target extracted by the target tracking unit out of the second parallax derived from the newly acquired stereo image Target candidates in the region corresponding to the two-dimensional image of the target candidate region of the specific target extracted by the target tracking means out of the parallax derived from the first parallax derived from the newly acquired radar information The parallax in which both are the same or substantially the same as each other is changed to zero. As a result, the specific target is deleted in the second parallax derived from the newly acquired stereo image, and that portion becomes the background. It is possible to exclude it from the object from which it is derived.

The second target detection apparatus of the present invention is acquired by a laser radar that acquires radar information in front of the vehicle, an optical time-of-flight distance sensor that acquires a single image in front of the vehicle and the distance to the subject, and laser radar. And a control unit for processing the radar information and the single image acquired by the optical time-of-flight distance sensor and the distance to the subject. Here, the control unit has the following means (B1) to (B7).
(B1) Radar information, a single image, a first parallax derived from the radar information, and a second parallax derived from the single image and the distance to the subject or the second after being corrected by the parallax correcting means The position, width, and height of each target candidate on the single image are derived from the parallax, and the three-dimensional coordinates of the three-dimensional coordinate system in the optical time-of-flight distance sensor are derived for each pixel of the single image. Position etc. acquisition means (B2) Using the position, width and height of each target candidate on the single image derived by the position etc. acquisition means, a two-dimensional image of each target candidate area is cut out from the single image. In addition, after the cut-out two-dimensional image and the parallax derived from the distance to the subject corresponding to the cut-out two-dimensional parallax are cut out from the second parallax, to this The parallax derived from the distance to the corresponding subject and the parallax of the target candidate in the area corresponding to the cut out two-dimensional image of the first parallax are compared with each other and are the same or substantially the same. Using the first partial image, position candidate correction means (B3) for correcting the position, width and height of each target candidate derived by the position acquisition means on the single image was cut out by the position correction means. Type discriminating means for discriminating the type of each target candidate using the two-dimensional image or the first partial image (B4) on the single image of each target candidate whose type is discriminated by the type discriminating means and the target tracking means. Target list registration means for registering the position, width, height, type, two-dimensional image of the target candidate area, and the three-dimensional coordinates of the three-dimensional coordinate system in the time-of-flight distance sensor in the target list ( B5) In a newly acquired single image The position, width, height, and type of a stereo image of each target candidate recorded in the target list is converted into a two-dimensional image of the area including the predicted position of each target candidate that is expected to exist. And a target candidate that is expected to be included in the cut out two-dimensional image from the newly acquired single image based on the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera. When the correlation with each target candidate recorded in the list is evaluated by a predetermined method and, as a result, the correlation between both is evaluated to be extremely high, the target candidate (specific target with high correlation) ) Of the region including the predicted position in (2), the parallax derived from the two-dimensional image of the region including the predicted position and the corresponding distance to the subject and the newly acquired radar information. The expected position of one parallax The specific target is newly acquired using the second partial image corresponding to the parallax that is the same or almost the same as each other by comparing the parallax of the target candidate in the area corresponding to the two-dimensional image of the including area While correcting the position, width, and height on a single image, the three-dimensional coordinates of the three-dimensional coordinate system in the time-of-flight distance sensor of the specific target are acquired, and the new specific target obtained by the correction is newly acquired. Using the position, width and height on the acquired single image, a two-dimensional image of the target candidate area of the specific target is cut out from the newly acquired single image, and the correlation between both is extremely high If it is evaluated as relatively high although it cannot be said, the position and width and height of the newly acquired target image (quasi-specific target) on the single image that has a relatively high correlation and the target candidate 2D image of area and time-of-flight distance After acquiring the three-dimensional coordinates of the three-dimensional coordinate system in the sensor by performing the same processing as the above-described processing for the specific target, the two-dimensional image of the target candidate area of the acquired semi-specific target is used as a quasi-standard. Target tracking means for discriminating the type of the specific target (B6) In the target list, the position and width of the specific target derived by the target tracking means on the single image, the height, and the target candidate area Target list update means (B7) for updating the two-dimensional image and the three-dimensional coordinates of the three-dimensional coordinate system in the time-of-flight distance sensor. The second image derived from the newly acquired single image and the distance to the subject. Among the parallaxes, the second derived from the two-dimensional image of the target candidate area of the specific target derived by the target tracking means and the parallax derived from the distance to the subject corresponding thereto and the newly acquired radar information Target pursuit out of one parallax The parallax that compares the parallax of the target candidates in the area corresponding to the two-dimensional image of the target candidate area of the specific target cut out by the means and changes the parallax that is the same or substantially the same as each other to zero Correction means

  In the second target detection apparatus of the present invention, a laser radar is used in addition to the optical time-of-flight distance sensor to detect the target. As a result, it is possible to detect and recognize a target such as a pedestrian that is difficult to reflect a millimeter wave radar signal. Further, the position, width and height of each target candidate derived by the position etc. acquiring unit on the single image are corrected using the first partial image. Here, since the first partial image corresponds to the two-dimensional image of the target candidate area obtained by removing the background image, the target candidate is mixed into the background image, and there is an error in the position of the target candidate. It can be eliminated. Further, when tracking the temporal change of each target candidate, not only using the position and width of the target candidate, the height, and the three-dimensional coordinates of the three-dimensional coordinate system in the optical time-of-flight distance sensor, Since the types of target candidates are also used, it is possible to detect and recognize the target with high accuracy. If the correlation is evaluated to be high, the target of the specific target derived by the target tracking means out of the newly acquired single image and the second parallax derived from the distance to the subject. The target of the specific target cut out by the target tracking means out of the parallax derived from the two-dimensional image of the candidate area and the distance to the subject corresponding thereto and the first parallax derived from the newly acquired radar information Contrast with the parallax of the target candidate in the area corresponding to the two-dimensional image of the target candidate area, the parallax where both are the same or substantially the same is changed to zero. As a result, the specific target is deleted in the second parallax derived from the newly acquired single image, and that portion becomes the background. Therefore, the position and width of the specific target on the newly acquired single image In addition, the height can be excluded from the target to be derived.

The third target detection apparatus of the present invention includes a laser radar that acquires radar information in front of the vehicle, a stereo camera that acquires a stereo image in front of the vehicle, radar information acquired by the laser radar, and stereo acquired by the stereo camera. And a control unit for processing an image. Here, the control unit has the following means (C1) to (C9).
(C1) First position information acquisition means for acquiring the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera for each pixel of the stereo image (C2) The second parallax and stereo derived from the stereo image for each pixel in the stereo image Among the plurality of cells in which the ordinate of the image is divided at a predetermined interval, the image is distributed to cells including the second parallax of each pixel and the ordinate of the stereo image, the second parallax is set as the x axis, and the ordinate of the stereo image is The z-axis value is binarized in the three-dimensional distribution when the y-axis and the number of pixels per cell are the z-axis, and the road surface area intersecting the straight line L2 where the second parallax is zero in the xy plane equivalent linear acquisition means (C3) for detecting the inclined straight line L1 which derives the x coordinate V ₀ which the intersection of the straight lines L1 and L2, and the x-coordinate V _0, with the focal length f of the stereo camera Depression angle information acquisition means (C4) for deriving the depression angle α of the teleo camera The second parallax and x coordinate of each point on the straight line L1 and the second parallax of each pixel having the parallax and x coordinate of each point near the straight line L1 are made zero Using the second parallax corrected by the parallax information deleting unit (C5) and the abscissa of the three-dimensional coordinates obtained by the first position information acquiring unit, each pixel in the stereo image is corrected. The second parallax and the three-dimensional coordinate abscissa are distributed to cells including the second parallax and the three-dimensional coordinate abscissa of each pixel among a plurality of cells divided by a predetermined interval. The maximum ordinate of the three-dimensional coordinates of the pixels included in each cell in the three-dimensional distribution with the x-axis, the second parallax as the y-axis, and the number of pixels (occupancy) for each cell as the z-axis Values and average values for each cell Occupation degree information acquisition means for calculating (C6) In the three-dimensional distribution derived by the occupancy degree information acquisition means, the z-axis value is smoothed and binarized, and the z-axis value is higher than a predetermined threshold value. Using the width and area in the x-axis direction and the maximum value and average value of the cells included in the mass, the mass is extracted as a target candidate according to a predetermined rule determined for each detection target, and on the xy plane First target acquisition means (C7) for acquiring the center position and width After correcting the radar information obtained by the laser radar using the depression angle α, the first parallax and the three-dimensional in the stereo camera are obtained from the corrected radar information. Second position information acquisition means (C8) for acquiring the three-dimensional coordinates of the coordinate system The abscissa of the three-dimensional coordinates obtained by the second position information acquisition means is the x-axis and is obtained by the second position information acquisition means. Let the first parallax be the y-axis Second target acquisition means (C9) for acquiring the center position and width of each target candidate on the xy plane in accordance with a predetermined rule, the xy plane of each target candidate obtained by the first target acquisition means Each target obtained by the first target acquisition means by comparing the upper center position and width with the center position and width on the xy plane of each target candidate obtained by the second target acquisition means. The presence / absence of the candidate is determined, and the position and width of each target candidate determined to be present on the stereo image are obtained, the center position and width on the xy plane obtained by the first target acquisition means, and the second target acquisition. The center position and width on the xy plane obtained by the means are weighted and averaged according to a predetermined rule, and then the center position and width on the xy plane obtained by the weighted average are converted into a position and width on the stereo image. Further, the height of each target candidate determined to be present The third target deriving means for obtaining from the ordinate of the three-dimensional coordinates obtained in location information acquiring means

In the third target detection apparatus of the present invention, a laser radar is used in addition to the stereo camera in order to detect the target. As a result, it is possible to detect and recognize a target such as a pedestrian that is difficult to reflect a millimeter wave radar signal. In addition, the second parallax is obtained by binarizing the z-axis value in a three-dimensional distribution when the stereo image has the ordinate of the y-axis and the number of pixels per cell is the z-axis. The second parallax and the x coordinate of each point on the straight line L1 and the second parallax of each pixel having the parallax and x coordinate of each point near the straight line L1 are corrected to zero. Thereby, since the road surface parallax is deleted, it is possible to prevent the target candidate from being mixed with the road surface as the background image and causing an error in the position of the target candidate. Further, the radar information is corrected using the stereo camera depression angle α derived using the x coordinate V ₀ of the intersection of the straight line L1 and the straight line L2 and the focal length f of the stereo camera. Thereby, it is possible to eliminate an error when the radar information is converted into the first parallax or the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera. Also, the center position and width of each target candidate obtained by the first target acquisition means on the xy plane, and the center position and width of each target candidate obtained by the second target acquisition means on the xy plane. And the presence / absence of each target candidate obtained by the occupancy level information obtaining means is determined. Here, since the depth information on the xy plane is represented by parallax, no error divergence occurs in the distance.

The fourth target detection apparatus of the present invention is acquired by a laser radar that acquires radar information in front of the vehicle, an optical time-of-flight distance sensor that acquires a single image in front of the vehicle and the distance to the subject, and laser radar. And a control unit for processing the radar information and the single image acquired by the optical time-of-flight distance sensor and the distance to the subject. Here, the control unit has the following means (D1) to (D9).
(D1) First position information acquisition means for acquiring the three-dimensional coordinates of the three-dimensional coordinate system in the optical time-of-flight distance sensor for each pixel of the single image (D2) The second parallax derived from the distance to and the ordinate of the single image is distributed to cells including the second parallax of each pixel and the ordinate of the single image among a plurality of cells separated by a predetermined interval; When the second parallax is the x axis, the ordinate of the single image is the y axis, and the number of pixels per cell is the z axis, the z axis value is binarized in the xy plane. Straight line acquisition means (D3) for detecting an oblique straight line L1 corresponding to a road surface area intersecting with the straight line L2 where the second parallax is zero. The x coordinate V ₀ of the intersection of the straight line L1 and the straight line L2 is derived, and this x coordinate and V _0, the focal length of the optical time-of-flight distance sensor Depression angle information acquisition means (D4) for deriving the depression angle α of the optical time-of-flight distance sensor using f and the second parallax and x coordinate of each point on the straight line L1 and the parallax and x coordinate of each point near the straight line L1 The parallax information deleting means for correcting the second parallax of each pixel to zero (D5) The second parallax corrected by the parallax information deleting means and the abscissa of the three-dimensional coordinates obtained by the first position information acquiring means Using each pixel in the single image, the second parallax and the abscissa of the three-dimensional coordinate of each pixel are included in a plurality of cells in which the abscissa of the second parallax and the three-dimensional coordinate is divided at a predetermined interval. Included in each cell in a three-dimensional distribution when the abscissa of the three-dimensional coordinates is the x-axis, the second parallax is the y-axis, and the number of pixels (occupancy) for each cell is the z-axis. Maximum value of the three-dimensional coordinate ordinate of the pixel Occupation degree information acquisition means (D6) for calculating the average value for each cell In the three-dimensional distribution derived by the occupancy degree information acquisition means, the z-axis value is smoothed and binarized, and the z-axis value is a predetermined value. Extracts clusters as target candidates using predetermined rules determined for each detection target using the x-axis width and area of the clusters of cells higher than the threshold and the maximum and average values of the cells included in the clusters. Then, first target acquisition means (D7) for acquiring the center position and width on the xy plane, the radar information obtained by the laser radar is corrected using the depression angle α, and then the first radar information is obtained from the corrected radar information. Second position information acquisition means (D8) for acquiring the parallax and the three-dimensional coordinates of the three-dimensional coordinate system in the optical time-of-flight distance sensor. The abscissa of the three-dimensional coordinates obtained by the second position information acquisition means is the x-axis. In the second position information acquisition means Second target acquisition means (D9) for acquiring the center position and width of each target candidate on the xy plane according to a predetermined rule when the obtained first parallax is the y-axis. The first object is obtained by comparing the center position and width of the obtained target candidates on the xy plane with the center position and width of the target candidates obtained by the second target acquisition means on the xy plane. The presence or absence of each target candidate obtained by the target acquisition means is determined, and the position and width on the single image of each target candidate determined to be present on the xy plane obtained by the first target acquisition means The center position and width on the xy plane obtained by the weighted average after the weighted average of the center position and width and the center position and width on the xy plane obtained by the second target acquisition means are weighted and averaged. Each target acquired by converting to a position and width on one image and determined to exist The third target deriving means for obtaining the height of the complement, the ordinate of the three-dimensional coordinates obtained by the first position information obtaining means

In the fourth target detection apparatus of the present invention, a laser radar is used in addition to the optical time-of-flight distance sensor to detect the target. As a result, it is possible to detect and recognize a target such as a pedestrian that is difficult to reflect a millimeter wave radar signal. Also obtained by binarizing the z-axis value in a three-dimensional distribution where the second parallax is the x-axis, the ordinate of the single image is the y-axis, and the number of pixels per cell is the z-axis. The second parallax and x coordinate of each point on the straight line L1 and the second parallax of each pixel having the parallax and x coordinate of each point near the straight line L1 are corrected to zero. Thereby, since the road surface parallax is deleted, it is possible to prevent the target candidate from being mixed with the road surface as the background image and causing an error in the position of the target candidate. Also, radar information is obtained by using the depression angle α of the optical time-of-flight distance sensor derived using the x coordinate V ₀ of the intersection of the straight line L1 and the straight line L2 and the focal length f of the optical time-of-flight distance sensor. It is corrected. Thereby, it is possible to eliminate the occurrence of an error when the radar information is converted into the first parallax or the three-dimensional coordinates of the three-dimensional coordinate system in the optical time-of-flight distance sensor. Also, the center position and width of each target candidate obtained by the first target acquisition means on the xy plane, and the center position and width of each target candidate obtained by the second target acquisition means on the xy plane. And the presence / absence of each target candidate obtained by the height information acquisition means is determined. Here, since the depth information on the xy plane is represented by parallax, no error divergence occurs in the distance.

  According to the first target detection apparatus of the present invention, in order to detect a target, a laser radar is used in addition to a stereo camera, and a millimeter wave radar signal such as a pedestrian as well as a vehicle and an obstacle is detected. Since it is possible to detect and recognize a target that is difficult to reflect, it is possible to detect a target such as a vehicle, a pedestrian, and an obstacle located far away from the vehicle with a simple configuration. In addition, when the correlation is evaluated to be high, the specific target is deleted in the second parallax derived from the newly acquired stereo image, and thus the specific target is newly acquired. It is possible to exclude the upper position, width, and height from the objects to be derived. As a result, it is possible not only to greatly reduce the time required for calculation processing compared to the case where a specific target is a target for deriving a position, etc., but also when target candidates are close to each other on a stereo screen. Even if it exists, each position, width, height, type, etc. can be derived. Therefore, in the present invention, targets such as vehicles, pedestrians, and obstacles that are close to each other can be detected in a short time with a simple configuration.

  According to the second target detection apparatus of the present invention, in order to detect a target, a laser radar is used in addition to the optical time-of-flight distance sensor, and not only a vehicle or an obstacle but also a pedestrian. Targets such as vehicles, pedestrians, and obstacles that are far away from the vehicle can be detected with a simple configuration because it is possible to detect and recognize targets that are difficult to reflect millimeter-wave radar signals. it can. In addition, when it is evaluated that the correlation is high, the specific target is deleted in the second parallax derived from the newly acquired single image and the distance to the subject. It is possible to exclude the position, width, and height on the newly acquired single image from the targets to be derived. This makes it possible not only to significantly reduce the time required for calculation processing compared to the case where a specific target is the target for deriving the position, etc., but also when target candidates are close to each other on a single image Even so, the position, width, height, type, etc. of each can be derived. Therefore, in the present invention, targets such as vehicles, pedestrians, and obstacles that are close to each other can be detected in a short time with a simple configuration.

  According to the third target detection apparatus of the present invention, in order to detect a target, a laser radar is used in addition to a stereo camera, and not only a vehicle or an obstacle but also a millimeter wave radar signal such as a pedestrian. Since it is possible to detect and recognize a target that is difficult to reflect, it is possible to detect a target such as a vehicle, a pedestrian, and an obstacle located far away from the vehicle with a simple configuration. Also, the center position and width of each target candidate obtained by the first target acquisition means on the xy plane, and the center position and width of each target candidate obtained by the second target acquisition means on the xy plane. In contrast, since the presence or absence of each target candidate obtained by the height information acquisition means is determined, the divergence of the error does not occur far away. Therefore, according to the present invention, it is possible to detect a target such as a vehicle, a pedestrian, and an obstacle that are located far away with a simple configuration with the same accuracy as a target near the vehicle.

  According to the fourth target detection apparatus of the present invention, in order to detect a target, a laser radar is used in addition to the optical time-of-flight distance sensor, and not only a vehicle or an obstacle but also a pedestrian. Targets such as vehicles, pedestrians, and obstacles that are far away from the vehicle can be detected with a simple configuration because it is possible to detect and recognize targets that are difficult to reflect millimeter-wave radar signals. it can. Also, the center position and width of each target candidate obtained by the first target acquisition means on the xy plane, and the center position and width of each target candidate obtained by the second target acquisition means on the xy plane. In contrast, since the presence or absence of each target candidate obtained by the height information acquisition means is determined, the divergence of the error does not occur far away. Therefore, according to the present invention, it is possible to detect a target such as a vehicle, a pedestrian, and an obstacle that are located far away with a simple configuration with the same accuracy as a target near the vehicle.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic configuration of a sensor fusion (target detection apparatus) according to an embodiment of the present invention. This sensor fusion is a system for detecting a forward traveling vehicle such as an automobile or a motorcycle, a target such as a pedestrian or an obstacle, and is mounted on the automobile C, for example.

  This sensor fusion includes a laser radar 1, a stereo camera 2, a control unit 3, and a storage unit 4.

  The control unit 3 is configured by, for example, a DSP (Digital Signal Processor) and the like, and processes information obtained by the dalaser radar 1 and the stereo camera 2 to determine the position and type of the target ahead of the car C by a predetermined calculation. It has come to identify. The storage unit 4 includes, for example, a RAM (Random Access Memory) and an HD (hard disk), and the storage unit 4 stores a program for calibrating sensor fusion, a program for detecting a target, and the like. The calculation results obtained by the control unit 3 (for example, a target list to be described later) are stored as needed.

  The laser radar 1 is attached to, for example, a central portion of a bumper of the automobile C, and radiates laser light in front of the automobile C, detects reflected light of the laser light emitted in front of the automobile C, and From the detected reflected light, the azimuth θ, the distance d, the relative speed v, and the like of the target existing in front of the automobile C are measured.

  In general, the laser radar has a higher detection capability than the millimeter wave radar, and can measure a target such as a pedestrian that hardly reflects radio waves. In general, the laser radar is superior to the millimeter wave radar in terms of viewing angle, resolution, and followability. In terms of cost, laser radar is generally cheaper than millimeter wave radar. Therefore, in the present embodiment, the laser radar 1 is used with emphasis on pedestrian detection and cost.

  The stereo camera 2 includes a right camera 21 and a left camera 22. The right camera 21 and the left camera 22 are composed of, for example, a charge coupled device (CCD), and are, for example, the inner walls of the windshield of the automobile C, are spaced from each other by a predetermined distance, and are mounted at the same height from the road surface. ing. As a result, the stereo camera 2 captures the front of the car C from different viewpoints, and obtains a stereo image composed of a two-dimensional image captured by the right camera 21 and a two-dimensional image captured by the left camera 22. It is supposed to be. In addition, it is preferable that the optical axes of the right camera 21 and the left camera 22 are parallel to each other, and the respective image planes are on the same plane.

  Next, a method for detecting a target in the sensor fusion according to the present embodiment will be described.

  FIG. 2 shows each functional block executed by a program for detecting a target stored in the storage unit 4, and FIG. 3 shows the position of each function shown in FIG. Each functional block included in the equal acquisition means 10 is represented.

  First, before describing each functional block shown in FIGS. 2 and 3, (1) a three-dimensional coordinate system of the stereo camera 2 and the laser radar 1, and (2) an occupancy fusion map (OFM) (3) The U-Disparity plane and the V-Disparity plane will be described.

(3D coordinate system of stereo camera 2 and laser radar 1)
For example, a small area on the target T in front of the car C is projected onto the two-dimensional coordinates (Xr, Yr) on the screen of the right camera 21 and the two-dimensional coordinates (Xl, Yl) on the screen of the left camera 22. ). At this time, when these two-dimensional coordinates (Xr, Yr), (Xl, Yl) are converted into three-dimensional coordinates (X, Y, Z) of the three-dimensional coordinate system of the stereo camera 2, the following formula ( 1) to (4) are used.

X = ((Xl + Xr) / 2) × (L / dc) (1)
Y = Yl × L / dc (2)
Z = f × L / dc (3)
dc = X1-Xr (4)

  Here, dc (= X1−Xr) is a shift between the left and right images, and is referred to as binocular parallax (or simply parallax). L is the baseline length of the stereo camera 2, that is, the distance between the center positions of the right camera 21 and the left camera 22. f is the focal length of the stereo camera 2. Since these L and f are known values, the three-dimensional coordinates (X, Y, Z) of the minute region can be obtained from the projected coordinates on the left and right images by the above equations (1) to (4). I can do it. Further, the origin of the three-dimensional coordinate system of the stereo camera 2 is a midpoint between the center positions of the right camera 21 and the left camera 22. It should be noted that the depth Zc can be obtained by calculation if the parallax is known from the equation (3). A method for obtaining the depth Zc from the parallax dc of the stereo image and detecting the position of the target in the three-dimensional space is called stereo vision.

  Further, it is assumed that a minute area on the target T in front of the automobile C is captured by the laser radar 1. At this time, when converting the radar information (θ, dc, v) received by the laser radar 1 into the three-dimensional coordinates (Xm, Ym, Zm) of the radar coordinate system, the following equations (5) to (7) are used. Use.

Xm = dc × sin θ (5)
Ym = dc × cos θ × sin ξ (6)
Zm = dc × cos θ (7)

  Here, ξ is the depression angle of the laser radar 1. Note that the origin of the three-dimensional coordinate system of the laser radar 1 is the center position of the laser radar 1.

  As described above, the origins of the three-dimensional coordinate systems of the laser radar 1 and the stereo camera 2 are slightly deviated due to the difference in location where the laser radar 1 and the stereo camera 2 are installed. However, by using the transformation matrix N that corrects the deviation, the three-dimensional coordinates (Xm, Ym, Zm) of the radar coordinate system are changed to the three-dimensional coordinates (X, Y, Z) of the three-dimensional coordinate system of the stereo camera 2. Can be converted. The formula used at this time is shown below.

  λ × [X, Y, Z] = N [Xm, Ym, Zm] (8)

  Here, λ is a coefficient for the Saisei camera coordinate system. Further, the conversion matrix N can be calculated by calibrating the laser radar 1 and the stereo camera 2.

(Occupied Fusion Map (OFM))
OFM is generally arranged in a grid formed within a horizontal plane by projecting measurement points onto the horizontal plane based on the three-dimensional coordinates of each point on the target surface and dividing the coordinate axes at equal intervals. It is obtained by accumulating the number of measurement points projected in the cell, and a target can be extracted by performing a clustering process on this OFM.

  However, conventionally, when acquiring an OFM, the horizontal plane was used as a projection fusion map, so when projecting and adding to a cell, it diverges in the distance due to the difference in stereo vision perspective error, When the position of a target is measured with a stereo camera alone, it is difficult to specify a distant obstacle area, and there is a problem that the areas of the targets are easily connected and it is difficult to divide the targets. Therefore, in the present embodiment, when the OFM is acquired, the U-Disparity plane is used as a projection fusion map.

(U-Disparity plane and V-Disparity plane)
The X-Disparity plane is an xy plane in which the horizontal axis x is the abscissa of the three-dimensional coordinate system of the stereo camera 2 and the vertical axis y is parallax (FIGS. 5A and 5B). 6 (A) and (B)). Cells arranged in a grid formed in the xy plane by dividing coordinate axes at equal intervals have unequal sizes in the y-axis direction in the real world. Therefore, the closer to the car C, the larger the resolution and the smaller the actual size.

  In addition to the X-Disparity plane, there is an xy plane called a V-Disparity plane. The V-Disparity plane is an xy plane in which the horizontal axis x is parallax and the vertical axis y is the ordinate of the three-dimensional coordinate system of the stereo camera 2 (see FIGS. 4A and 4B). .

  By projecting a parallax image derived from a stereo image onto the U-Disparity plane and the V-Disparity plane, the three-dimensional distribution can be expressed by a straight line, so that the analysis of the three-dimensional distribution is simplified.

  Next, a method for detecting a target will be described with reference to FIG.

  First, the control unit 3 acquires the radar information A0 from the laser radar 1, acquires the stereo image B0 from the stereo camera 2, and then derives the first parallax C0 from the radar information A0 and the second from the stereo image B0. The parallax D0 is derived.

  Next, in the position etc. acquiring means 10, each of the radar information A0, the stereo image B0, the first parallax C0, and the second parallax E0 after being corrected by the second parallax D0 or the parallax correcting means 80 described later. The position F0, the width G0, and the height H0 of the target candidate T on the stereo image B0 are derived, and the three-dimensional coordinate J0 of the three-dimensional coordinate system in the stereo camera 2 is derived for each pixel of the stereo image B0.

  Next, the position correction means 20 uses the position F0, the width G0, and the height H0 on the stereo image B0 of each target candidate T derived by the position etc. acquisition means 10 to obtain 2 of each target candidate area. The dimensional image K0 is cut out from the stereo image B0, and the parallax derived from the cut out two-dimensional image K0 is cut out from the second parallax D0.

  Next, in the cut-out two-dimensional image B0, the disparity derived from the cut-out two-dimensional image B0 in the second disparity D0 and the region corresponding to the cut-out two-dimensional image B0 in the first disparity A0 On the stereo image B0 of each target candidate T derived by the position etc. acquiring means 10 using the first partial image N0 of the part that is the same or almost the same as each other in comparison with the parallax of the target candidate. The position F0, the width G0, and the height H0 are corrected to obtain a new position F1, the width G1, and the height H1.

  Next, the type discriminating unit 30 discriminates the type P0 of each target candidate T using the two-dimensional image K0 or the first partial image N0 cut out by the position correcting unit 20.

  For example, model matching is performed as an initial determination for determining whether or not the target candidate T is a pedestrian. For example, multiple types of typical pedestrian contours and their symmetrical contour sets are prepared, and each set of perspective contours is also prepared, and these are used for initial judgment by edge-based model matching. I do. As a result, the target candidate T having a high matching rate is sent to the target tracking means 60, and the target candidate T having a low matching rate is sent to the next detailed determination, and the matching rate is low. The target candidate T is deleted. In the initial determination, when the first partial image N0 is used, the influence of the background can be eliminated, and the matching rate can be improved accordingly.

  In the detailed determination, for example, identification is performed using the edge size and direction of each pixel of the two-dimensional image K0 or the first partial image N0. Specifically, using the probability of being a pedestrian and the probability of being a non-pedestrian at each edge, the probability of being a pedestrian and the probability of being a non-pedestrian in the entire image is calculated, and the pedestrian is calculated based on the ratio. Can be identified.

  Next, in the target list registration unit 40, the position F1 and the width G1, the height H1, the type P0, and the target on the stereo image B0 of each target candidate T whose type has been determined by the type determination unit 30. The two-dimensional image K0 of the candidate area and the three-dimensional coordinate M0 of the three-dimensional coordinate system in the stereo camera 2 are registered in the target list 50.

  Next, in the target tracking unit 60, a two-dimensional image of a region including the predicted position of each target candidate T that is predicted to exist in the newly acquired stereo image B1 is recorded in the target list 50. Based on the position F2 and width G2 on the stereo image B0 of each target candidate T, the height H2, the type P2, and the three-dimensional coordinates M2 of the three-dimensional coordinate system in the stereo camera 2 The correlation between each target candidate T that is cut out from the acquired stereo image B1 and is expected to be included in the cut out two-dimensional image and each target candidate T that is recorded in the target list 50 is determined in advance. Evaluate by method.

  Here, it is possible to derive the predicted position using, for example, a velocity vector obtained from the position of each target candidate T in a plurality of stereo images acquired in the past. Further, the correlation is derived from, for example, the two-dimensional image of each target candidate T included in the target list 50 and converted to a gray scale model, and this gray scale model and the target tracking means 60. It is possible to obtain a correlation score with a two-dimensional image of a predetermined size centered on the predicted position of each target candidate T and evaluate it according to the magnitude of this correlation score.

  As a result of the evaluation, when it is evaluated that the correlation between the two is extremely high, the predicted position in the two-dimensional image of the region including the predicted position of the target candidate T (specific target T1) having a high correlation is obtained. Target candidates in a region corresponding to the two-dimensional image of the region including the predicted position among the parallax derived from the two-dimensional image of the region including the first parallax C1 derived from the newly acquired radar information A1 Using the second partial image corresponding to the parallax in which both of them are the same or substantially the same as the parallax of T, the position F2 and the width G2 and the height H2 of the specific target T1 are newly acquired. The position is corrected to a position F3, a width G3 and a height H3 on the image B1. Furthermore, the three-dimensional coordinate M3 of the three-dimensional coordinate system in the stereo camera 2 of the specific target T1 is acquired, and the position F3 and the width G3 on the newly acquired stereo image B1 of the specific target T1 obtained by the correction and the high The two-dimensional image K3 of the target candidate area of the specific target T1 is cut out from the newly acquired stereo image B1 using the height H3.

  In addition, as a result of the evaluation, if the correlation between the two is not very high but is evaluated as relatively high, a target candidate T (quasi-specific target T2) having a relatively high correlation is newly acquired. The position F4 and width G4 and height H4 on the stereo image B1, the two-dimensional image K4 of the target candidate region, and the three-dimensional coordinate M4 of the three-dimensional coordinate system in the stereo camera 2 are processed for the specific target. After obtaining by performing the same processing as, the type P4 of the semi-specific target T2 is determined using the two-dimensional image K4 of the target candidate area of the obtained semi-specific target T2.

  Next, in the target list registration means 40, the position F4 and width G4 on the stereo image B1 of each target candidate T (order specific target T2) whose type has been determined by the target tracking means 60, and the height. H4, the type P4, the two-dimensional image K4 of the target candidate area, and the three-dimensional coordinates M4 of the three-dimensional coordinate system in the stereo camera 2 are registered in the target list 50.

  When a target for which a corresponding area with high correlation is not found in the target tracking unit 60 among the targets included in the target list 50 satisfies a predetermined condition, the target is deleted from the target list 50. It is preferable.

  Next, the target list update unit 70 includes a position F3 and a width G3 on the stereo image B1 of the specific target T1 derived by the target tracking unit 60 in the target list 50, a height H3, and a target. The two-dimensional image K3 of the candidate area and the three-dimensional coordinate M3 of the three-dimensional coordinate system in the stereo camera 2 are updated.

  Next, in the parallax correction means 80, the 2D image K3 of the target candidate area of the specific target T1 cut out by the target tracking means 60 out of the second parallax D1 derived from the newly acquired stereo image B1. Corresponding to the two-dimensional image K3 of the target candidate area of the specific target T1 cut out by the target tracking means 60 out of the first parallax C1 derived from the newly acquired radar information A1. Contrast with the parallax of the target candidate T in the area is changed to zero so that both are the same or substantially the same.

  At this time, among the second parallax D1 derived from the newly acquired stereo image B1, the parallax derived from the two-dimensional image K4 of the target candidate area of the semi-specific target T2 cut out by the target tracking means 60. And an object in an area corresponding to the two-dimensional image K4 of the target candidate area of the semi-specific target T2 extracted by the target tracking means 60 out of the first parallax C1 derived from the newly acquired radar information A1. Contrast with the parallax of the target candidate T may be changed to zero when both are the same or substantially the same.

  Next, the details of the position etc. acquiring means 10 of FIG. 2 will be described with reference to FIG.

  First, the first position information acquisition unit 110 acquires the three-dimensional coordinate m0 of the three-dimensional coordinate system in the stereo camera 2 for each pixel of the stereo image B0.

  Next, in the straight line acquisition unit 111, each pixel in the stereo image B0 is converted into the second parallax D0 derived from the stereo image B0 or the second parallax E0 and the stereo image B0 after being corrected by the parallax correcting unit 80 described later. Among a plurality of cells whose ordinates are separated by a predetermined interval, the cells are distributed to cells including the second parallax D0, E0 of each pixel and the ordinate of the stereo image B0.

  Subsequently, in the three-dimensional distribution (see FIG. 4A) where the second parallax is the x axis, the ordinate of the stereo image B0 is the y axis, and the number of pixels per cell is the z axis (see FIG. 4A), The value is binarized, and an oblique straight line L1 corresponding to a road surface area intersecting with the straight line L2 (y axis) where the second parallaxes D0 and E0 are zero in the xy plane is detected.

Next, the depression angle information acquisition unit 112 derives the x coordinate V ₀ of the intersection of the straight line L1 and the straight line L2 (y axis) (see FIG. 4B), and the x coordinate V ₀ and the focal point of the stereo camera. The depression angle α of the stereo camera is derived using the distance f.

  Next, in the parallax information deletion unit 113, the second parallax of each pixel having the second parallax D0, E0 and x coordinate of each point on the straight line L1 and the parallax D0, E0 and x coordinate of each point near the straight line L1 is obtained. Correct to zero.

  Next, the occupancy degree information acquisition unit 114 uses the second parallax Q0 corrected by the parallax information deletion unit 113 and the abscissa m1 of the three-dimensional coordinates obtained by the first position information acquisition unit 110 to perform stereo. Each pixel in the image B0 includes the second parallax Q0 and the abscissa m1 of the three-dimensional coordinate of each pixel among a plurality of cells in which the second parallax Q0 and the abscissa m1 of the three-dimensional coordinate are separated at a predetermined interval. Sort into cells. Subsequently, in the three-dimensional distribution R0 where the abscissa m1 of the three-dimensional coordinates is the x-axis, the second parallax Q0 is the y-axis, and the number of pixels (occupancy) for each cell is the z-axis, each cell The maximum value S0 and the average value S1 of the ordinate m1 of the three-dimensional coordinates of the pixels included in are calculated for each cell.

  Next, in the first target acquisition unit 115, in the three-dimensional distribution R0 derived by the occupation level information acquisition unit 114, the z-axis value is smoothed and binarized, and the z-axis value is higher than a predetermined threshold value. Using the width and area of the cell mass in the x-axis direction and the maximum value S0 and the average value S1 of the cells included in the mass, the mass is converted into a target candidate T according to a predetermined rule determined for each detection target. To obtain the center position U0 and the width U1 on the xy plane (see FIGS. 5A and 5B).

  Here, in the first target acquisition unit 115, for example, an empirical value is used as the initial value of the predetermined threshold, and a value corresponding to the degree of occupancy obtained after the recognition of each target candidate T is obtained as needed. It is possible to use.

  Next, in the second position information acquisition unit 116, the radar information A0 obtained by the laser radar 1 is corrected using the depression angle α, and then the first parallax C0 and 3 in the stereo camera 2 are determined from the corrected radar information A1. The three-dimensional coordinate V0 of the dimensional coordinate system is acquired.

  Next, in the second target acquisition unit 117, the abscissa of the three-dimensional coordinate V0 obtained by the second position information acquisition unit 116 is the x axis, and the first parallax C0 obtained by the second position information acquisition unit 116 is obtained. Is the y-axis, the center position U2 and the width U3 of each target candidate T on the xy plane are acquired according to a predetermined rule (see FIGS. 6A and 6B).

  Next, the third target deriving unit 118 obtains the center position U0 and the width U1 on the xy plane of each target candidate T obtained by the first target obtaining unit 115 and the second target obtaining unit 117. The presence or absence of each target candidate T obtained by the first target acquisition means 115 is determined by comparing the center position U2 and the width U3 on the xy plane of each target candidate T obtained. As a result, the position F0 and the width G0 of each target candidate T determined to be present on the stereo image B0, the center position U0 and the width U1 on the xy plane obtained by the first target acquisition means 115, and the second The center position U2 and the width U3 on the xy plane obtained by the target acquisition means 117 are weighted and averaged according to a predetermined rule, and the center position and width on the xy plane obtained by the weighted average are positions on the stereo image B0. And by converting to width. Further, the height H0 of each target candidate T determined to be present is acquired from the ordinate m2 of the three-dimensional coordinate obtained by the first position information acquisition unit 110.

  Note that the third target deriving unit 118 can determine a weighted average rule according to the measurement resolution of the laser radar 1 and the stereo camera 2 on the xy plane of the first target acquiring unit 110.

  The third target deriving unit 118 may detect a radar ghost. Specifically, among the target candidates T acquired by the second target acquisition means 116, the positions and widths of the residual target candidates on the stereo image B0 excluding the target candidates T determined to exist, A two-dimensional image of the residual target candidate is cut out from the stereo image B0 using the specified height H0. Subsequently, among the cut-out two-dimensional images, the target candidates in the region corresponding to the cut-out two-dimensional image of the first parallax C0 and the parallax derived from the cut-out two-dimensional image of the second parallax D0. In contrast to the parallax of T, when the number of pixels in a portion where both are the same or substantially the same is smaller than a predetermined threshold value, it is determined that the residual target candidate exists. Then, the position F0 and width G0 on the stereo image B0 of the residual target candidate determined to exist are displayed on the stereo image B0, and the center position U2 and width U3 on the xy plane obtained by the second target acquisition unit 117 are displayed on the stereo image B0. Obtained by converting to position and width.

  In the sensor fusion of the present embodiment, the laser radar 1 is used in addition to the stereo camera 2 in order to detect a target. This makes it possible to detect and recognize targets such as pedestrians that are difficult to reflect millimeter-wave radar signals, so it is possible to select targets such as vehicles, pedestrians, and obstacles that are far away from the vehicle with a simple configuration. Can be detected. In addition, using the first partial image described above, the position F2 and the width G2 and the height H2 of the specific target T1 are corrected to the position F3 and the width G3 and the height H3 on the newly acquired stereo image B1. . Here, since the first partial image corresponds to the two-dimensional image of the target candidate region obtained by removing the background image, the target candidate T is confused with the background image, and the position of the target candidate T is determined. It is possible to eliminate an error.

  In the present embodiment, the position and width of the target candidate T, the height, and the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera 2 are used when tracking the temporal change of each target candidate T. In addition, since the types of target candidates are also used, it is possible to detect and recognize the target with high accuracy.

  In the present embodiment, when the correlation is evaluated to be high, the target of the specific target extracted by the target tracking unit out of the second parallax derived from the newly acquired stereo image. Corresponds to the two-dimensional image of the target candidate area of the specific target extracted by the target tracking means out of the parallax derived from the two-dimensional image of the candidate area and the first parallax derived from the newly acquired radar information Contrast with the parallax of the target candidates in the area is changed to zero, which is the same or almost the same as each other. For example, the configuration is changed from FIG. 7A to FIG. 7B. As a result, the specific target is deleted in the second parallax derived from the newly acquired stereo image, and that portion becomes the background. It is possible to exclude it from the object from which it is derived. As a result, it is possible not only to greatly reduce the time required for calculation processing compared to the case where a specific target is a target for deriving a position, etc., but also when target candidates are close to each other on a stereo screen. Even if it exists, each position, width, height, type, etc. can be derived.

  Therefore, in the present embodiment, targets such as vehicles, pedestrians, and obstacles that are close to each other can be detected in a short time with a simple configuration.

[Modification]
In the above embodiment, the stereo camera 2 is used, but instead, for example, an optical time-of-flight distance sensor that acquires a single image ahead of the vehicle and a distance to the subject can be used. However, in this case, it is necessary to replace “stereo image” with “single image” in the above embodiment. In the above embodiment, “parallax derived from a two-dimensional image cut out from a stereo image” can be derived from a two-dimensional image cut out from a single image and a distance to the corresponding subject. .

[Example]
In order to verify the effectiveness of the target detection by the sensor fusion according to the above embodiment and its modification, pedestrian detection was performed in a simulation environment (a parking lot with few obstacles). The target pedestrian performed an operation as shown in Table 1. In addition, a total of 10 scenes including the difference between shooting in a state in which a vehicle equipped with a sensor is stopped and shooting in a low-speed traveling state are targeted. In addition, as shown in Table 2, experiments were conducted on various pedestrians existing in an actual road environment. A total of 10 scenes of pedestrians such as traffic lights, narrow paths, and sidewalks were selected from scenes shot while traveling in the city, and used as evaluation scenes.

  The experimental results in the simulation environment are shown in Tables 3 and 4. In the simulation environment, there were few false obstacles in the vicinity, so a high detection rate and recognition rate could be obtained.

  Tables 5 and 6 show the results of experiments in an actual road environment. The ROI average detection rate of the road experiment was 85.19%, and the recognition rate was 72.09%. When the accuracy of the original image and the parallax image is reduced, the recognition rate is lower than in the simulation environment. If the brightness value fluctuates up or down due to the relationship between sunlight and shadows, the image itself will deteriorate, so the parallax may not be measured due to the texture loss of the object in the stereo camera, or it may contain a lot of noise. There may be a parallax image. In these scenes, the reduction in the accuracy of the parallax images directly leads to a reduction in the recognition rate. In order to solve this problem, it is necessary to perform automatic measurement of luminance values and automatic correction based on the result as preprocessing of parallax calculation after image acquisition by camera photography. Alternatively, in order to avoid a load due to these calculations, a solution means that a camera having an automatic brightness adjustment function is used can be considered.

  In addition, delay in response to the lateral movement of the radar is one of the factors that reduce the recognition rate in the road environment. When the radar detection point deviates from an object such as a pedestrian, the ROI region may not be accurately taken and the pedestrian may not be recognized. Since the detection point shift occurs when the detection point is delayed from the actual position, the actual position is predicted by examining the locus of the detection point of the radar, and the parallax image is used to predict the particle corresponding to the radar distance. It is considered that the ROI region can be corrected by finding the position.

  FIG. 8 shows a processing speed distribution diagram in the simulation experiment, and FIG. 9 shows a processing speed distribution map in an actual road environment. The unit of the horizontal axis in these figures is 10 milliseconds. In the simulation experiment, an average processing speed of 175 milliseconds / frame was achieved. However, since the number of object candidates is small, it is considered that the OFM construction, the number of ROI detections, and the number of ROIs for identification are small. On the other hand, since there are many obstacle candidates in an actual road environment, it took time for ROI detection and target matching, and the average processing time for one frame was 265 milliseconds.

  While the present invention has been described with reference to the embodiment and examples, the present invention is not limited to these, and various modifications are possible.

It is a schematic block diagram of the sensor fusion which concerns on one embodiment of this invention. It is a functional block diagram for demonstrating the detection method of the target in the sensor fusion of FIG. It is a functional block diagram in the position etc. acquisition means of FIG. It is a conceptual diagram for demonstrating a V-Disparity plane. It is a conceptual diagram for demonstrating the U-Disparity plane in a stereo camera coordinate system. It is a conceptual diagram for demonstrating the U-Disparity plane in a laser radar coordinate system. It is the schematic diagram which represented typically a mode that the target candidate was deleted from 2nd parallax. It is a distribution map of processing speed in a simulation experiment. It is a distribution map of processing speed in a real road environment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Laser radar, 2 ... Stereo camera, 21 ... Right camera, 22 ... Left camera, 3 ... Control part, 4 ... Memory | storage part.

Claims (6)

A laser radar for acquiring radar information in front of the vehicle;
A stereo camera for acquiring a stereo image in front of the vehicle;
A control unit that processes radar information acquired by the laser radar and a stereo image acquired by the stereo camera, and
The control unit includes first position information acquisition means, straight line acquisition means, depression angle information acquisition means, parallax information deletion means, occupation degree information acquisition means, first target acquisition means, and second position information acquisition. Means, second target acquisition means, and third target acquisition means,
The first position information acquisition means acquires the three-dimensional coordinates of the three-dimensional coordinate system in the stereo camera for each pixel of the stereo image,
The straight line acquisition unit is configured to determine each pixel in the stereo image as a second parallax of a plurality of cells in which a second parallax derived from the stereo image and a ordinate of the stereo image are divided at a predetermined interval. 3 when the parallax and the ordinate of the stereo image are allocated to the cell, the second parallax is the x axis, the ordinate of the stereo image is the y axis, and the number of pixels per cell is the z axis. Binarizing the z-axis value in the dimensional distribution, detecting an oblique straight line L1 corresponding to a road surface area intersecting with the straight line L2 where the second parallax is zero in the xy plane,
The depression angle information acquisition means derives the x coordinate V0 of the intersection of the straight line L1 and the straight line L2, and derives the depression angle α of the stereo camera using the x coordinate V0 and the focal length f of the stereo camera. And
The parallax information deleting unit corrects the second parallax and the x coordinate of each point on the straight line L1 and the second parallax of each pixel having the parallax and x coordinate of each point near the straight line L1 to zero,
The degree-of-occupation information acquisition unit uses the second parallax corrected by the parallax information deletion unit and the abscissa of the three-dimensional coordinates obtained by the first position information acquisition unit, so that each pixel in the stereo image Are allocated to cells including the second parallax of each pixel and the abscissa of the three-dimensional coordinate among the plurality of cells in which the second parallax and the abscissa of the three-dimensional coordinate are separated at a predetermined interval, In the three-dimensional distribution in which the abscissa of the dimensional coordinate is the x-axis, the second parallax is the y-axis, and the number of pixels (occupancy) per cell is the z-axis, the 3 of the pixels included in each cell. Calculate the maximum and average ordinates of dimensional coordinates for each cell,
The first target acquisition means smoothes and binarizes the z-axis value in the three-dimensional distribution derived by the occupancy degree information acquisition means, and the cell target has a z-axis value higher than a predetermined threshold. Using the width and area in the x-axis direction and the maximum value and the average value of the cells included in the mass, the mass is extracted as a target candidate according to a predetermined rule determined for each detection target, Obtain the center position and width on the xy plane,
The second position information acquisition unit corrects the radar information obtained by the laser radar using the depression angle α, and then calculates the first parallax and the three-dimensional coordinate system of the stereo camera from the corrected radar information. Get 3D coordinates,
The second target acquisition means uses the abscissa of the three-dimensional coordinates obtained by the second position information acquisition means as the x axis and the first parallax obtained by the second position information acquisition means as the y axis. Sometimes, the center position and width of each target candidate on the xy plane is acquired according to a predetermined rule.
The third target deriving unit includes a center position and a width on the xy plane of each target candidate obtained by the first target obtaining unit, and each target candidate obtained by the second target obtaining unit. Is compared with the center position and the width on the xy plane to determine the presence or absence of each target candidate obtained by the first target acquisition means, and on the stereo image of each target candidate determined to be present Predetermined rules for the position and width are the center position and width on the xy plane obtained by the first target acquisition means and the center position and width on the xy plane obtained by the second target acquisition means. Is obtained by converting the center position and width on the xy plane obtained by the weighted average to the position and width on the stereo image, and further, the height of each target candidate determined to exist is obtained. , The vertical seat of the three-dimensional coordinates obtained by the first position information acquisition means It is obtained from target detection device object according to claim. The first target acquisition means uses an empirical value as an initial value of the predetermined threshold, and uses a value according to the degree of occupancy obtained after recognition of each target candidate as needed. The target detection apparatus according to claim 1. 2. The third target deriving unit determines a weighted average rule according to respective measurement resolutions of the laser radar and the stereo camera on the xy plane of the first target acquiring unit. The target detection apparatus described in 1. A laser radar for acquiring radar information in front of the vehicle;
A time-of-flight distance sensor that acquires a single image in front of the vehicle and the distance to the subject;
A control unit for processing radar information acquired by the laser radar and a single image acquired by the optical time-of-flight distance sensor and a distance to the subject,
The control unit includes first position information acquisition means, straight line acquisition means, depression angle information acquisition means, parallax information deletion means, occupation degree information acquisition means, first target acquisition means, and second position information acquisition. Means, second target acquisition means, and third target acquisition means,
The first position information acquisition means acquires the three-dimensional coordinates of the three-dimensional coordinate system in the optical time-of-flight distance sensor for each pixel of a single image,
The straight line acquisition means includes a plurality of cells in which each pixel in the single image is divided into a second parallax derived from a distance to the single image and a subject and a ordinate of the single image at predetermined intervals. Are distributed to cells including the second parallax of each pixel and the ordinate of the single image, the second parallax is the x-axis, the ordinate of the single image is the y-axis, and the pixel for each cell The z-axis value is binarized in the three-dimensional distribution when the number of is the z-axis, and an oblique straight line L1 corresponding to a road surface area intersecting the straight line L2 where the second parallax is zero in the xy plane Detect
The depression angle information acquisition means derives the x coordinate V0 of the intersection of the straight line L1 and the straight line L2, and uses the x coordinate V0 and the focal distance f of the optical time-of-flight distance sensor to calculate the optical flight time. Deriving the depression angle α of the mold distance sensor,
The parallax information deleting unit corrects the second parallax and the x coordinate of each point on the straight line L1 and the second parallax of each pixel having the parallax and x coordinate of each point near the straight line L1 to zero,
The occupancy degree information acquisition means uses the second parallax corrected by the parallax information deletion means and the abscissa of the three-dimensional coordinates obtained by the first position information acquisition means, Distributing the pixels to a cell including the second parallax of each pixel and the abscissa of the three-dimensional coordinate among a plurality of cells in which the second parallax and the abscissa of the three-dimensional coordinate are divided at a predetermined interval, In the three-dimensional distribution in which the abscissa of the three-dimensional coordinate is the x-axis, the second parallax is the y-axis, and the number of pixels (occupancy) per cell is the z-axis, the pixels included in each cell Calculate the maximum and average 3D coordinate ordinates for each cell,
The first target acquisition means smoothes and binarizes the z-axis value in the three-dimensional distribution derived by the occupancy degree information acquisition means, and the cell target has a z-axis value higher than a predetermined threshold. Using the width and area in the x-axis direction and the maximum value and the average value of the cells included in the mass, the mass is extracted as a target candidate according to a predetermined rule determined for each detection target, Obtain the center position and width on the xy plane,
The second position information acquisition means corrects the radar information obtained by the laser radar using the depression angle α, and then calculates the first parallax and the time-of-flight distance sensor 3 from the corrected radar information. Get the 3D coordinates of the 3D coordinate system,
The second target acquisition means uses the abscissa of the three-dimensional coordinates obtained by the second position information acquisition means as the x axis and the first parallax obtained by the second position information acquisition means as the y axis. Sometimes, the center position and width of each target candidate on the xy plane is acquired according to a predetermined rule.
The third target deriving unit includes a center position and a width on the xy plane of each target candidate obtained by the first target obtaining unit, and each target candidate obtained by the second target obtaining unit. On the xy plane, the presence or absence of each target candidate obtained by the first target acquisition means is determined, and the target candidate determined to be present on the single image The center position and width on the xy plane obtained by the first target acquisition means and the center position and width on the xy plane obtained by the second target acquisition means It is obtained by converting the center position and width on the xy plane obtained by weighted average after the rule to the position and width on the single image, and further, the height of each target candidate determined to exist Is obtained from the ordinate of the three-dimensional coordinates obtained by the first position information acquisition means. Target detection device which is characterized in that to. The first target acquisition means uses an empirical value as an initial value of the predetermined threshold, and uses a value according to the degree of occupancy obtained after recognition of each target candidate as needed. The target detection apparatus according to claim 4 . The third target deriving unit determines a weighted average rule according to each measurement resolution of the laser radar and the optical time-of-flight distance sensor in the xy plane of the first target acquiring unit. The target detection apparatus according to claim 4 .

Images