JP4918676B2 - Calibration apparatus and calibration method - Google Patents

Description

The present invention relates to a calibration device and a calibration method for calibrating a target detection device that detects a target using coordinates detected by a radar and coordinates detected by a stereo camera.

  In recent years, anti-collision devices that prevent collisions with targets such as cars and motorcycles, pedestrians, and obstacles, and inter-vehicle distance control to keep the distance between targets within a certain range. Various driving support devices have been developed to improve the safety and comfort of driving such as automobiles and motorcycles. In these driving support devices, it is extremely important to detect a target, and the driving support device is equipped with a target detection device for detecting such a target. The target detection device includes, for example, a radar that detects a target with laser light or millimeter wave electromagnetic waves, a camera that detects a target from analysis of a captured image, a sensor fusion that includes the radar and the camera, and the like. It is configured.

  Here, in general, the radar can not only measure the distance to the target and the relative speed with high accuracy, but also can detect the target even in bad weather such as dense fog. Although it has the advantage that it is possible, detection of the boundary of the target is not easy because the resolution in the lateral direction is low, and the field of view is narrow because the angle in the vertical direction of the detection range is small. Cameras generally have the advantage of easy detection of the boundary of a target due to its high planar resolution and a wide viewing angle. However, the accuracy of the distance to the target and the relative speed is not so high, and the weather and lighting Since it is easily affected by conditions, it has a drawback that the ability to detect a target is likely to deteriorate in bad weather or at night. On the other hand, sensor fusion can compensate for the shortcomings of both radar and camera with the advantages of both, and has become the mainstream of recent developments. For example, the target detection apparatus described in Patent Literatures 1 and 2 includes a sensor fusion including one radar and one camera, and the coordinates of the target detected by the radar, The target is detected using the coordinates of the target detected by the camera.

JP 2003-315441 A JP 2003-315442 A

  However, in such a device, since the coordinates obtained from the camera do not include accurate information about the distance to the target, the coordinates obtained from the radar and the coordinates obtained from the camera are There is a problem that it is not easy to use and match (match) the targets.

The present invention has been made in view of such problems, and the object thereof is to facilitate the use of target coordinates obtained from a radar and the coordinates of a target obtained from a camera. and to provide a calibration device and a calibration method thereof for calibrating the standard detection equipment possible ones that match (matching).

The calibration apparatus of the present invention includes a radar for acquiring radar information in front of a vehicle, a stereo camera having a right camera and a left camera for acquiring a stereo image in front of the vehicle, radar information acquired by the radar, and right And a control unit for processing a stereo image including a two-dimensional image acquired by the camera and a two-dimensional image acquired by the left camera. Here, the control unit has the following means (A) to ( F ).
(A) The three-dimensional coordinates of the radar coordinate system of each target are acquired from the radar information obtained by the radar, and the three-dimensional coordinates of the radar coordinate system of each target acquired from the radar information are distributed on one plane. If not, the radar side three-dimensional coordinate acquisition means (B) corrects the three-dimensional coordinates using the depression angle and the deviation angle of the radar, and selects each target from the two-dimensional image obtained by the right camera. Camera-side two-dimensional coordinate acquisition means (C) for extracting the two-dimensional coordinates of the right camera coordinate system of the target from the two-dimensional image obtained by the left camera. Using the parallax information acquisition means (D) parallax information for acquiring parallax information from the two-dimensional coordinates of the right camera coordinate system of the target and the two-dimensional coordinates of the left camera coordinate system of each target, the right camera coordinates of each target 2D coordinates of the system and left camera seat Radar of each target obtained based on camera side 3D coordinate acquisition means (E) radar information for converting 2D coordinates of the system into 3D coordinates of the right camera coordinate system and 3D coordinates of the left camera coordinate system Using the three-dimensional coordinates of the coordinate system and the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system of each target obtained based on the stereo image, the three-dimensional coordinates of the radar coordinate system are obtained. Conversion coefficient acquisition means for obtaining a conversion coefficient for conversion into three-dimensional coordinates in the right camera coordinate system and three-dimensional coordinates in the left camera coordinate system
(F) The radar-side three-dimensional coordinate acquisition means performs linear approximation on the three-dimensional coordinate of the radar coordinate system of each target acquired from the radar information by using the least square method, and obtains 3 of each target obtained thereby. A first straight line including dimensional coordinates and a second straight line that intersects perpendicularly in a plane perpendicular to the one plane are obtained, and an angle at which the second straight line intersects the vehicle axle is obtained as a radar deviation angle.

The calibration method of the present invention includes the following steps (A) to ( H ).
(A) A radar for acquiring radar information in front of the vehicle, a stereo camera having a right camera and a left camera for acquiring a stereo image in front of the vehicle, radar information acquired by the radar, and acquired by the right camera A step of preparing a control unit for processing a stereo image including a two-dimensional image and a two-dimensional image acquired by a left camera; (B) three-dimensional coordinates of a radar coordinate system of each target from radar information obtained by radar (C) The two-dimensional coordinates of the right camera coordinate system of each target from the two-dimensional image obtained by the right camera, and the left of each target from the two-dimensional image obtained by the left camera Step of extracting the two-dimensional coordinates of the camera coordinate system (D) The parallax information is acquired from the two-dimensional coordinates of the right camera coordinate system of each target and the two-dimensional coordinates of the left camera coordinate system of each target. Step (E) Using the parallax information, the two-dimensional coordinates of the right camera coordinate system and the two-dimensional coordinates of the left camera coordinate system of each target are converted into the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system. (F) The three-dimensional coordinates of the radar coordinate system of each target are acquired based on the radar information, and the three-dimensional coordinates of the radar coordinate system of each target acquired from the radar information are on one plane. If the distribution angle is not distributed, the step of correcting the three-dimensional coordinates using the depression angle and deviation angle of the radar
(G) In the above correction, the three-dimensional coordinates of each target obtained from the radar information are linearly approximated using the least square method, and the three-dimensional coordinates of each target obtained by the approximation are included. A step of obtaining a second straight line that intersects perpendicularly in a plane perpendicular to the first plane and the one plane, and obtaining an angle at which the second straight line and the axle of the automobile intersect as a deviation angle of the radar.
(H) The three-dimensional coordinates of the radar coordinate system of each target obtained based on the radar information and the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system of each target obtained based on the stereo image. A step of obtaining a conversion coefficient for converting the three-dimensional coordinates of the radar coordinate system into the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system using the three-dimensional coordinates.

  In the calibration device and the calibration method of the present invention, since the parallax information related to the distance from the target is obtained from the stereo image obtained from the stereo camera, the two-dimensional coordinates of the right camera coordinate system obtained from the stereo camera and Two-dimensional coordinates in the left camera coordinate system can be converted into three-dimensional coordinates in the right camera coordinate system and three-dimensional coordinates in the left camera coordinate system. Thus, the coordinates of each target obtained from the radar and the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system converted based on the parallax information are used. Conversion coefficients for converting the dimensional coordinates into the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system can be obtained.

The target detection apparatus according to the reference example includes a radar, a stereo camera, and a control unit, as in the above calibration apparatus, and further converts the three-dimensional coordinates of the radar coordinate system into the three-dimensional coordinates and the left camera coordinates of the right camera coordinate system. A storage unit is provided in which conversion coefficients for conversion into the three-dimensional coordinates of the system are stored. Here, the control unit has the following means (A) to (D).
(A) Radar side three-dimensional coordinate acquisition means for acquiring the three-dimensional coordinates of the radar coordinate system of each target from radar information obtained by the radar. (B) 3 of the radar coordinate system of each target using a conversion coefficient. The three-dimensional coordinates are converted into three-dimensional coordinates in the right camera coordinate system and three-dimensional coordinates in the left camera coordinate system, and the three-dimensional coordinates in the right camera coordinate system and the left camera coordinate system of each converted target are converted. Radar side two-dimensional coordinate acquisition means for converting coordinates into two-dimensional coordinates in the right camera coordinate system and two-dimensional coordinates in the left camera coordinate system. (C) Right of each target from two-dimensional images obtained by the right camera. The camera-side two-dimensional coordinate acquisition means (D) radar information for extracting the two-dimensional coordinates of the left camera coordinate system of each target from the two-dimensional image obtained by the left camera. Each obtained based on 2D coordinates of the right camera coordinate system and 2D coordinates of the left camera coordinate system, and 2D coordinates of the right camera coordinate system and 2D coordinates of the left camera coordinate system of each target obtained based on the stereo image Target identification means to identify the target ahead of the vehicle

The target detecting device according to the reference example, since to use a transform coefficient, from the radar information obtained by the radar, the two-dimensional coordinates and the left-right camera coordinate system of each target object including the depth information substantially The two-dimensional coordinates of the camera coordinate system can be obtained.

  According to the calibration apparatus and the calibration method of the present invention, since the parallax information related to the distance from the target is obtained from the stereo image obtained from the stereo camera, the three-dimensional coordinates of the radar coordinate system are converted into the right camera coordinate system. Conversion coefficients for conversion to the three-dimensional coordinates of the left camera coordinate system and the three-dimensional coordinates of the left camera coordinate system can be obtained. As a result, when the position of an obstacle is identified during traveling, the coordinates of the target obtained from the radar and the target obtained from the stereo camera are used using the conversion coefficient obtained as described above. Since the coordinates can be contrasted, it is possible to specify a target having a high possibility of existence. Therefore, the targets can be easily matched using the coordinates of the target obtained from the radar and the coordinates of the target obtained from the stereo camera.

According to the target detection apparatus according to the reference example, since to use a transform coefficient, from the radar information obtained by the radar, the two-dimensional coordinates of the right camera coordinate system of each target object including the depth information substantially And the two-dimensional coordinates of the left camera coordinate system can be obtained. Thus, obtained based on radar information, the two-dimensional coordinates of the two-dimensional coordinates and the left camera coordinate system of the right camera coordinate system of each target object that substantially contains depth information obtained on the basis of the stereo image and as it can be contrasted with the two-dimensional coordinates of the two-dimensional coordinates and the left camera coordinate system of the right camera coordinate system of each target object can be identified with high target object presence possibilities. Therefore, the targets can be easily matched using the coordinates of the target obtained from the radar and the coordinates of the target obtained from the stereo camera.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic configuration of a sensor fusion (target detection device) according to an embodiment of the present invention. This sensor fusion is a system that detects a forward traveling vehicle such as an automobile or a motorcycle, a target such as a pedestrian or an obstacle, and is mounted on the automobile C, for example.

  This sensor fusion includes a radar 1, a stereo camera 2, a control unit 3, a storage unit 4, and a display unit 5. The radar 1 includes, for example, a laser radar that emits laser light or a millimeter wave radar that emits millimeter wave electromagnetic waves, and is attached to a central portion of a bumper of the automobile C. The radar 1 emits laser light and millimeter wave electromagnetic waves in front of the automobile C, detects reflected light of laser light and reflected waves of millimeter wave electromagnetic waves emitted in front of the automobile C, and further detects the reflected light detected. Further, the azimuth θ, the distance d, the relative speed v, and the like of the target existing in front of the automobile C are measured from the reflected wave. The stereo camera 2 includes a right camera 21 and a left camera 22. The right camera 21 and the left camera 22 are made of, for example, a charge coupled device (CCD), are inner walls of the windshield of the automobile C, are spaced apart from each other by a predetermined distance, and are mounted at the same height from the road surface. Yes. The stereo camera 2 captures the front of the car C from different viewpoints and acquires a stereo image composed of a two-dimensional image captured by the right camera 21 and a two-dimensional image captured by the left camera 22. It has become.

  The control unit 3 is configured by, for example, a DSP (Digital Signal Processor) or the like, and processes information obtained by the radar 1 and the stereo camera 2 to determine the position and type of an obstacle ahead of the vehicle C by a predetermined calculation. It has come to identify. The storage unit 4 includes, for example, a RAM (Random Access Memory) and an HD (hard disk). The storage unit 4 includes a program for calibrating sensor fusion, a program for detecting an obstacle, and the control unit 3. Stores the calculation results obtained in. The display unit 5 is, for example, a liquid crystal display device, and displays information based on the calculation result obtained by the control unit 3.

  Next, a procedure for calibrating sensor fusion will be described. However, prior to this, the reflector 6 used for the calibration will be described.

(Reflector 6)
2 and 3 show an example of the reflecting plate 6. FIG. 2 illustrates a reflecting plate when the radar 1 is a laser radar, and FIG. 3 illustrates a reflecting plate when the radar 1 is a millimeter wave radar. It is a thing. FIG. 4 shows an arrangement example of the reflector 6 when calibrating the sensor fusion.

  The reflection plate 6 in FIG. 2 has a lattice pattern in which reflection blocks 6A and absorption blocks 6B are alternately arranged. The reflection block 6A is made of a material that strongly reflects laser light, and the absorption block 6B is made of a material that strongly absorbs laser light. On the other hand, the reflecting plate 6 in FIG. 3 includes a flat plate 6C and a reflector 6D. The flat plate 6C is made of a material that transmits millimeter-wave electromagnetic waves (cloth, foamed polystyrene, etc.), and the reflector 6D is fixed on the back surface. The reflector 6D is made of, for example, a triangular pyramid-shaped metal plate, and is arranged in a lattice pattern on the back surface of the reflector 6D. The reflector 6D reflects millimeter wave electromagnetic waves incident from the opening side.

  Here, the portion of the reflection block 6A or the flat plate 6C where the reflector 6D is disposed strongly reflects the laser beam or the millimeter wave electromagnetic wave, so that a target having the same width as the width D1 of the reflection block 6A or the like is obtained. The radar 1 can recognize that it exists there. On the other hand, the portion of the absorption block 6B or the flat plate 6C where the reflector 6D is not disposed hardly reflects laser light or millimeter wave electromagnetic waves, and therefore there exists a space having the same width as the width D2 of the absorption block 6B or the like. The radar 1 can recognize it.

  The size of each reflection block 6A and each absorption block 6B is set according to the resolution of the radar 1 and the distance between the radar 1 and the reflection plate 6. The horizontal width D1 of the reflection block 6A and the like needs to be at least about twice the resolution of the radar 1 because it is difficult for the radar 1 to recognize a target having the same size as the resolution. The horizontal width D2 of the absorption block 6B or the like is at least three times the resolution of the radar 1 because it becomes difficult for the radar 1 to clearly distinguish a plurality of targets as separate targets when they are arranged close to each other. It is necessary to be a degree. The vertical size H of the reflection block 6A and the like and the absorption block 6B and the like clearly distinguishes them as separate targets when there are a plurality of targets within an error range in a direction perpendicular to the scan plane of the radar 1. Therefore, it is necessary that the error is at least about a direction perpendicular to the scan plane of the radar 1.

  The horizontal size of the reflecting plate 6 is about the same as the range scanned by the radar 1, and the vertical size of the reflecting plate 6 is large enough to arrange a plurality of reflecting blocks 6A in the vertical direction. If it becomes the size beyond this, it will not be restrict | limited in particular. Further, for the sake of calibration, it is assumed that the lowermost reflection block 6A and the like are arranged at the same height as the radar 1 mounted on the automobile C.

(Radar 1 calibration, FIGS. 4 to 8)
First, the worker P installs the reflecting plate 6 (step S1). Specifically, as shown in FIG. 4, the reflector 6 intersects with the axis X1 of the automobile C perpendicularly, and in front of the automobile C (in front of the radar 1 and the stereo camera 2 installed in the automobile C). Arranged at a predetermined distance. Subsequently, in the case where the radar 1 is a laser radar, the cover that strongly absorbs the laser beam is left in the remaining part of the lattice-like pattern of the reflection plate 6 except for the pattern closest to the ground (see FIG. In the case where the radar 1 is a millimeter wave radar, only the bottom reflector 6D closest to the ground is left out of the reflectors 6D attached to the flat plate 6C, and the other reflectors 6D are removed from the flat plate 6C. .

  Next, the worker P requests the control unit 3 to execute calibration of the radar 1 itself (step S2). Then, the control unit 3 measures the three-dimensional coordinates of each target (reflecting block 6A, etc.) included in the pattern in the bottom row, the three-dimensional coordinates of the radar 1, the depression angle ξ, and the shift angle φ.

  Here, the depression angle ξ is the scan plane A of the radar 1 and the (Xm, Zm) plane B in the radar coordinate system (Xm, Ym, Zm) as shown in FIG. 5 (side view of FIG. 1). (Generally a plane parallel to the road surface) is an angle (small deviation angle) that intersects. When the radar 1 is a laser radar, the laser beam has a certain spread angle, and the plane that divides the spread angle into two equal parts is defined as a scan plane A. As shown in FIG. 6 (top view of FIG. 1), the deviation angle φ is perpendicular to a straight line X2 (described later) in a plane parallel to the (Xm, Zm) plane of the radar coordinate system (Xm, Ym, Zm). Is the angle at which the straight line X3 intersecting the vehicle and the axle X1 of the automobile C intersect.

  Specifically, first, the control unit 3 requests the radar 1 to scan (step S3). When the radar 1 receives the scan request (step S4), the radar 1 scans the pattern on the bottom row to detect the azimuth θi of the target (such as the reflection block 6A) existing in the scan plane, and also detects the bottom row detected. After calculating the distance di and the relative speed vi for each target (step S5), the radar information (θi, di, vi) of each target in the bottom row is transmitted to the control unit 3 (step S6). Note that the suffix i of θ, d, and v is an integer of 1 or more and n or less of the number of reflection blocks 6A included in the pattern in the bottom row.

  After receiving the radar information (θi, di, vi) of each target in the bottom row from the radar 1 (step S7), the control unit 3 determines whether or not all the targets that should be detected by the radar 1 have been detected. Is determined (step S8). When all the targets that should be detected by the radar 1 are detected, the control unit 3 uses the following equation (1) to detect the radar information (θi, di, vi) is converted into the three-dimensional coordinates (Xmi, Ymi, Zmi) of the radar coordinate system (Xm, Ym, Zm) (step S9), and the radar information (θi, di, vi) of each target in the bottom row ) And the three-dimensional coordinates (Xmi, Ymi, Zmi) of each target in the bottom row are stored in the storage unit 4 (step S10).

  Expression (1) is an expression for converting measurement data obtained by the radar into three-dimensional coordinates in the radar coordinate system.

  On the other hand, when all the targets that should be detected by the radar 1 are not detected, the control unit 3 requests the display unit 5 to display the fact (step S11). When receiving the request, the display unit 5 displays that all targets are not detected (step S12). The worker P confirms the display, and after adjusting the direction (the depression angle ξ and the deviation angle φ) of the radar 1 so that all the targets in the lowermost row are detected (step S13), the step P is performed again. S1 to S8 are executed. That is, aiming is executed. At this time, in order to detect all targets on the bottom row, for example, the intensity of the reflected light or the reflected wave corresponding to the target placed at the center among the targets on the bottom row is the highest. The depression angle ξ and the deviation angle φ are adjusted so as to increase.

  As a result, when all the targets in the lowermost row are detected by the radar 1, the control unit 3 sequentially executes the above steps S9 and S10. Step S11 is executed, and finally step 10 is executed.

  Next, the control unit 3 sets the three-dimensional coordinates (Xmi, Ymi, Zmi) of each target converted to the radar coordinate system on the (Xm, Ym) plane of the radar coordinate system (Xm, Ym, Zm). It is determined whether it is distributed (step S14). When the three-dimensional coordinates (Xmi, Ymi, Zmi) of each target are not distributed on the (Xm, Ym) plane, the three-dimensional coordinates (Xmi, Ymi, Zmi) of each target are on the reflector 6. In other words, that is, the radar 1 is not in real shape with respect to the reflecting plate 6, so that the radar 1 can be detected with respect to the reflecting plate 6 by accurately determining the inclination of the radar 1. It is possible to obtain the three-dimensional coordinates (Xmi ″, Ymi ″, Zmi ″) (described later) of the radar coordinate system (Xm, Ym, Zm) of each target when the object is true.

  As a result of the determination, if it is not distributed on the (Xm, Ym) plane, the control unit 3 sets, for example, the three-dimensional coordinates (Xmi, Ymi, Zmi) (black circles in FIG. 6) of each target to the minimum two. A linear approximation is performed using multiplication, and the three-dimensional coordinates (Xmi ′, Ymi ′, Zmi ′) (white circles in FIG. 6) of each target obtained by the least square method and the three-dimensional coordinates (Xmi ′, Ymi). ′, Zmi ′) and a straight line X3 perpendicular to the straight line X2 in the (Xm, Zm) plane of the radar coordinate system (Xm, Ym, Zm) are obtained. Subsequently, using the triangulation method, the coordinates, depression angle ξ, and deviation angle φ of the radar 1 are calculated (step S15).

  Here, the coordinates of the radar 1 are (Xm, Zm) planes as shown in FIG. 6, for example, and the three-dimensional coordinates (Xmi ′, Ymi ′, Zmi ′) of each target on the straight line X2. ), The three-dimensional coordinates (Xm1 ′, Ym1 ′, Zm1 ′), (Xmn ′, Ymn) of the targets at both ends of the three-dimensional coordinates (Xmi ′, Ymi ′, Zmi ′) of each target. ′, Zmn ′), centering on the position, draw arcs with radii of distances d1 and dn corresponding to the coordinates, and calculate the coordinates corresponding to the point on the car C side between the two points where both arcs intersect Is required. As described above, the depression angle ξ is adjusted so that, for example, the intensity of the reflected light or the reflected wave corresponding to the target placed at the center among the targets in the lowermost row is maximized, and is almost 0 °. Therefore, for example, the default may be 0 °. The shift angle φ is obtained, for example, by calculating the angle at which the straight line X3 perpendicular to the straight line X2 and the axle X1 of the automobile C intersect in the (Xm, Zm) plane of the radar coordinate system (Xm, Ym, Zm). It is done.

  Next, the control unit 3 uses the depression angle ξ and the deviation angle φ to distribute the three-dimensional coordinates (Xmi, Ymi, Zmi) or (Xmi ′, Ymi ′, Zmi ′) of each target on the reflector 6. Thus, the three-dimensional coordinates (Xmi, Ymi, Zmi) or (Xmi ′, Ymi ′, Zmi ′) are corrected (step S16).

  On the other hand, when the three-dimensional coordinates (Xmi, Ymi, Zmi) of each target converted to the radar coordinate system are distributed on the (Xm, Ym) plane, the three-dimensional coordinates (Xmi) of each target , Ymi, Zmi) are distributed on the reflector 6, it can be assumed that the depression angle ξ and the deviation angle φ of the radar 1 are almost 0 °. Therefore, in this case, since it is not necessary to correct the three-dimensional coordinates (Xmi, Ymi, Zmi), the three-dimensional coordinates (Xmi, Ymi, Zmi) are not corrected. Of course, the depression angle ξ and the deviation angle φ of the radar 1 may be obtained, but in this case, the above steps S15 and S16 are executed.

  In the following, the procedure when the three-dimensional coordinates (Xmi, Ymi, Zmi) are corrected will be described. However, this procedure is performed for the three-dimensional coordinates (Xmi, Ymi, Zmi) that are not corrected. Is of course applicable.

  Next, the three-dimensional coordinates (Xmi ″, Ymi ″, Zmi ″), the depression angle ξ and the shift angle φ of each target in the bottom row obtained by the correction in the above step 16 are stored in the storage unit 4. Store (step S17).

  Next, the control unit 3 uses the three-dimensional coordinates (Xmi ″, Ymi ″, Zmi ″) of each target in the bottom row to coordinate the targets (reflection block 6A) other than the bottom row. Is estimated (step S18). For example, since the heights of the reflection block 6A and the absorption block 6B are both H, by adding H to Ymi every time one line rises from the bottom row, the coordinates (Xmi ″, Ymi ′) of each target other than the bottom row are added. '+ H × (s−1), Zmi ″). Here, s is the number of blocks counted from the bottom row (s = 1 in the bottom row). Thereafter, the coordinates (Xmi ″, Ymi ″ + H × (s−1), Zmi ″) of each target other than the bottom row are stored in the storage unit 4 (step S19).

  Next, the control unit 3 requests the display unit 5 to output calibration completion of the radar 1 itself (step S20). When receiving the request, the display unit 5 outputs that the calibration of the radar 1 itself has been completed (step S21). In this way, the calibration of the radar 1 itself is completed.

(Stereo camera 2 calibration itself, Fig. 9)
Next, the worker P adjusts the reflecting plate 6 (step S22). Specifically, when the radar 1 is a laser radar, the cover that covers the reflection plate 6 is removed to expose the entire lattice pattern of the reflection plate 6, while when the radar 1 is a millimeter wave radar, the first step is performed. The reflector 6D removed in step 1 is attached to the original position. Subsequently, the worker P requests the control unit 3 to calibrate the stereo camera 2 itself (step S23). Then, the control unit 3 requests the first shooting from the stereo camera 2 (step S24). When the stereo camera 2 receives the first imaging request (step S25), it captures a stereo image of the reflector 6 (step S26).

  Next, the control part 3 requests | requires the display part 5 to display the instruction | command which changes the angle with respect to the axle shaft X1 of the reflecting plate 6 to a predetermined angle with respect to the operator P (step S27). When receiving the display request from the control unit 3, the display unit 5 displays the command (Step S28). The worker P changes the angle of the reflecting plate 6 according to the command (step S29). Subsequently, the control unit 3 requests the stereo camera 2 to perform the next shooting (step S30). When the stereo camera 2 receives the next photographing request (step S31), the stereo camera 2 photographs a stereo image of the reflector 6 (step S32).

  Subsequently, after steps S27 to S33 are repeated a plurality of times while gradually changing the angle of the reflecting plate 6, when the number of times of photographing reaches a predetermined number (step S33), the control unit 3 transmits all of the photographed stereo images. Is requested to the stereo camera 2 (step S34). When the stereo camera 2 receives the request (step S35), it transmits all captured stereo images to the control unit 3 (step S36).

  When the control unit 3 receives the stereo image (step S37), the internal parameters (for example, the focal length F, the optical center position, and the optical distortion correction coefficient) of the stereo camera 2 and the external parameters are received using the received stereo image. (For example, the parallel progression and rotation determinant between the right camera 21 and the left camera 22) are calculated (step S38), and the internal and external parameters and the first stereo image taken are stored. Stored in the unit 4 (step S39).

  Next, the control unit 3 requests the display unit 5 to output calibration completion of the stereo camera 2 itself (step S40). Upon receiving the request, the display unit 5 outputs a message indicating that the calibration of the stereo camera 2 itself has been completed (step S41). In this way, calibration of the stereo camera 2 itself is completed.

(Calibration between radar 1 and stereo camera 2, FIG. 10)
The control unit 3 reads the first stereo image stored in the storage unit 4, and from the stereo image, the two-dimensional coordinates (Xrj, Yrj) of the right camera image coordinate system of each target and the left camera image The two-dimensional coordinates (Xlj, Ylj) of the sitting system are extracted (step S42). Specifically, four corners of the reflection block 6A are extracted, and the coordinates of the center of the four corners are extracted as the coordinates of the target. Note that the suffix j of Xr, Yr, Xl, and Yl is an integer that is 1 or more and less than or equal to the number of targets detected by the stereo camera 2.

  Subsequently, the control unit 3 uses the two-dimensional coordinates (Xrj, Yrj), (Xlj, Ylj) and the following formulas (2) to (7), and the three-dimensional of the right camera coordinate system of each target. The coordinates (XRj, YRj, ZRj) and the three-dimensional coordinates (XLj, YLj, ZLj) of the left camera coordinate system are obtained (step S43).

  Here, b is the base line length of the stereo camera 2, that is, the distance between the center positions of the right camera 21 and the left camera 22. Further, d (Xrj, Yrj) is the parallax information of the two-dimensional coordinates of the right camera, and d (Xlj, Ylj) is the parallax information of the two-dimensional coordinates of the left camera. The parallax information d (Xrj, Yrj) and d (Xlj, Ylj) relate to the distance to the target, and are a two-dimensional image captured by the right camera 21 and a two-dimensional image captured by the left camera 22. It can be obtained using images. Specifically, the right camera 21 and the left camera 22 are arranged at the same height, the two-dimensional coordinates (Xrj, Yrj) of each target in the two-dimensional image photographed by the right camera 21, and the left Of the two-dimensional coordinates (Xlj, Ylj) of each target in the two-dimensional image photographed by the camera 22, the Y coordinates of the same target coincide with each other. The absolute values | Xrj−Xlj | are the parallax information d (Xrj, Yrj) and d (Xlj, Ylj). Accordingly, the parallax information d (Xrj, Yrj) and d (Xlj, Ylj) in this case are values that are unrelated to the Y coordinate.

  Next, the control unit 3 obtains the coordinates (Xmi ″, Ymi ″ + H × (s−1), Zmi ″) of each target measured or estimated by the radar 1 from the storage unit 4. After reading, the three-dimensional coordinates (Xmi ″, Ymi ″ + H × (s−1), Zmi ″) of the radar coordinate system obtained from the radar 1 and the right camera coordinate system obtained from the stereo camera 2 The three-dimensional coordinates (XRj, YRj, ZRj) and the three-dimensional coordinates (XLj, YLj, ZLj) of the left camera coordinate system are substituted into the following equations (8) and (9), and Rrj for each target , Rlj, Trj, Tlj are obtained (step S44). Here, Rrj and Rlj are rotation determinants from the radar coordinate system to the camera coordinate system (XR, YR, ZR) and (XL, YL, ZL), and Trj and Tlj are the coordinates from the radar coordinate system to the camera coordinate system. It is a parallel progression type.

  Expression (8) is an expression for converting the three-dimensional coordinates of the radar coordinate system to the three-dimensional coordinates of the right camera coordinate system. Expression (9) is an expression for converting the three-dimensional coordinates of the radar coordinate system to the three-dimensional coordinates of the right camera coordinate system. This is an expression to convert to coordinates.

  Subsequently, the control unit 3 obtains average values Rr, Rl, Tr, and Tl of Rrj, Rlj, Trj, and Tlj for each target (step S45), and stores these in the storage unit 4 (step S46). .

  Thus, in the present embodiment, the coordinates (Xmi ″, Ymi ″ + H × (s−1), Zmi ″) of each target obtained by correcting the radar information from the radar 1, The three-dimensional coordinates of the radar coordinate system using the three-dimensional coordinates (XRj, YRj, ZRj) of the right camera coordinate system and the three-dimensional coordinates (XLj, YLj, ZLj) of the left camera coordinate system obtained from the stereo camera 2 Rr, Rl, Tr, and Tl for mutually converting the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system can be obtained.

  Next, the control unit 3 requests the display unit 5 to output the completion of sensor fusion calibration (step S47). When receiving the request, the display unit 5 outputs a message indicating that the sensor fusion calibration has been completed (step S48). In this way, calibration between the radar 1 and the stereo camera 2 is performed, and sensor fusion calibration is completed.

  As described above, in this embodiment, the coordinates of each target are obtained from the radar 1 and the coordinates of each target are obtained from the stereo camera 2, so that not only the radar 1 itself or the stereo camera 2 itself but also the radar. Calibration between the 1 and the stereo camera 2 can also be performed collectively. That is, the sensor fusion including the radar 1 and the stereo camera 2 can be simultaneously calibrated.

  Next, a procedure for specifying the presence, position, and type of an obstacle present in front of the automobile C while the automobile C is traveling will be described.

(Identification of obstacle position, FIGS. 11 to 14)
First, the control unit 3 sends a scan request to the radar 1 and a shooting request to the stereo camera 2 in synchronization with each other (step S49). When the radar 1 receives the scan request (step S50), the radar 1 scans the scan plane to detect the orientation θj of the target existing in the scan plane, and for each detected target, the distance dj and After calculating the relative speed vj (step S51), the measured radar information (θi, di, vi) is transmitted to the control unit 3 (step S52). Here, θ and the suffix i of d and v are integers of 1 or more and less than or equal to the number of targets detected by the radar 1. On the other hand, when the stereo camera 2 receives a request for shooting (step S53), the stereo camera 2 captures a stereo image in front of the car C (step S54) and transmits the stereo image to the control unit 3 (step S55).

  Next, the control unit 3 receives the radar information (θi, di, vi) from the radar 1 and the stereo image from the stereo camera 2 (step S56). Then, first, the control unit 3 uses the equation (1) and the following equations (10) to (13) to convert the received radar information (θi, di, vi) into the three-dimensional coordinates (XmRi) of the right camera coordinate system. , YmRi, ZmRi) and three-dimensional coordinates (XMri, YMLi, ZmLi) of the left camera coordinate system, and two-dimensional coordinates (Xmri, Ymri) of the right camera coordinate system and two-dimensional coordinates (Xmli) of the left camera coordinate system , Ymli) (step S57). Subsequently, the control unit 3 extracts the corners of the target from the received stereo image, so that the two-dimensional coordinates (Xrj, Yrj) of the right camera coordinate system and 2 of the left camera coordinate system of each target are extracted. Dimensional coordinates (Xlj, Ylj) are obtained (step S58). Here, the suffix j of Xr, Yr, Xl, Yl is an integer of 1 or more and less than the total number of targets detected from the stereo image.

  Expression (10) is an expression for converting the three-dimensional coordinates of the radar coordinate system into the three-dimensional coordinates of the right camera coordinate system, and Expression (11) is the three-dimensional coordinate of the radar camera coordinate system. This is an expression to convert to coordinates. Expression (12) is an expression for converting the three-dimensional coordinates obtained in Expression (10) into two-dimensional coordinates, and Expression (13) is for converting the three-dimensional coordinates obtained in Expression (11) into two-dimensional coordinates. It is a formula. Here, λ is a coefficient for the Saisei camera coordinate system, (Xr, Yr) is a two-dimensional coordinate of the right camera coordinate system, (Xl, Yl) is a two-dimensional coordinate of the left camera coordinate system, ( XR, YR, ZR) are the three-dimensional coordinates of the right camera coordinate system, (XL, YL, ZL) are the three-dimensional coordinates of the left camera coordinate system, and fxr, fyr are the focal lengths of the right camera along the pixels. Fxl and fyl are values obtained by dividing the focal length of the left camera by the vertical or vertical size of the pixels, and uo and vo are the right camera and the left camera, respectively. The image coordinates of the optical center position. Note that fxl, fyl, uo, and vo are obtained in the calibration of the stereo camera 2 itself.

  Next, the control unit 3 determines whether the one or more targets detected by the radar 1 are included in the one or more targets detected by the stereo camera 2. That is, matching is performed between one or more targets detected by the radar 1 and one or more targets detected by the stereo camera 2 (step S59).

  Specifically, first, the two-dimensional coordinates (Xmri, Ymri), (Xmri, Ymri) obtained from the radar 1 and the two-dimensional coordinates (Xrj, Yrj), (Xlj, Ylj) obtained from the stereo camera 2 are firstly displayed. Is compared with each other, it is determined whether or not there are coordinates that are the same or approximate to each other. As a result, when there are coordinates that are the same or approximate to each other, there is a high possibility that a target exists at those coordinates. Subsequently, the distance di corresponding to the two-dimensional coordinates (Xmri, Ymri), (Xmri, Ymli) obtained from the radar 1 is the same as the target detected by the radar 1 obtained from the stereo camera 2. The depth ZLj obtained by Expression (7) of the two-dimensional coordinates (Xrj, Yrj) and (Xlj, Ylj) of the target to be compared is compared with the target having a high possibility of existence. As a result, when the distance di and the depth ZLj match or approximate (match), the target detected by the radar 1 and the target detected by the stereo camera 2 are the same target. On the other hand, if they do not match, the target detected by the radar 1 and the target detected by the stereo camera 2 are different from each other. The control unit 3 extracts the target matched in this way as an obstacle candidate (step S60).

  Next, when the control unit 3 repeatedly performs step S50 to step 59, the control unit 3 compares the rate of change of the distance di of the obstacle candidate target with the rate of change of the depth ZLj (step S61). . As a result, when the rate of change of the distance di and the rate of change of the depth ZLj are completely different from each other, it is highly possible that the target does not exist, and therefore the target is excluded from the obstacle candidates ( Step S62). In the case where they are almost equal to each other, it is highly possible that the target exists, so that the target is left as an obstacle candidate.

  Next, the control unit 3 determines whether or not the obstacle candidate target is within the obstacle existence possible area. Here, the obstacle possible area is an area where an obstacle can exist, and includes, for example, the traveling road surface S1 and the roadside S2 as shown in FIG. The traveling road surface S1 is a road surface on which the automobile C can travel, and the road side S2 is a road surface provided along the traveling road surface S1. In FIG. 14, the traveling road surface S <b> 1 and the road side S <b> 2 are separated by a lane line L.

  As a result of the determination, if the target of the obstacle candidate is on the traveling road surface S1, as shown in FIG. 14, for example, the target is a list of obstacles that can exist on the traveling road surface S1 (for example, , Automobile, motorcycle, bicycle, fallen object), on the other hand, if the target of distance di is on the roadside S2, the target is on the roadside S2. This is one obstacle in a list of possible obstacles (eg, guardrails, road signs). Therefore, when the target of the obstacle candidate is outside the obstacle possible area, the target is a target that cannot be an obstacle for the car C. In that case, The target is excluded from obstacle candidates (step S63). In this way, the target of the obstacle candidate is specified, and the position of the target is specified.

(Identification of type of obstacle 7, FIGS. 12 to 14)
Next, the control unit 3, for example, two-dimensional coordinates (Xrj, Yrj), (Xlj, Ylj) obtained from the stereo camera 2, and two-dimensional coordinates (Xmri, Ymri), (Xmli) obtained from the radar 1. , Ymli) is estimated from the distance di corresponding to the obstacle obstacle candidate target shape and size (step S64). Subsequently, an obstacle corresponding to the estimated shape and size and relative speed vi is specified from the list (step S65). For example, when the obstacle candidate candidate is the obstacle 7 in FIG. 14 and the obstacle 7 is traveling at the relative speed vi, a list of obstacles that can exist on the traveling road surface S1 (for example, , Automobile, motorcycle, bicycle, fallen object). In this way, the type of obstacle is specified.

  As described above, in the present embodiment, since the parallax information related to the distance from the target is obtained from the stereo image obtained from the stereo camera 2, 2 in the right camera coordinate system obtained from the stereo camera 2 is obtained. The two-dimensional coordinates of the dimensional coordinates and the left camera coordinate system can be converted into the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system. As a result, the coordinates of each target obtained from the radar 1 and the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system converted based on the parallax information are used. Rr, Rl, Tr, and Tl for converting the three-dimensional coordinates into the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system can be obtained. As a result, the coordinates of the target obtained from the radar 1 and the stereo camera are obtained by using the Rr, Rl, Tr, and Tl obtained as described above when specifying the position of the obstacle during traveling. The coordinates of the target obtained from 2 can be compared. As a result, it is possible to identify a target that is highly likely to exist.

  Therefore, in the present embodiment, when the position of an obstacle is specified during traveling, the coordinates of the target detected by the radar 1 and the coordinates of the target detected by the stereo camera 2 are used. The marks can be easily matched with each other.

  In particular, in the present embodiment, since the parallax information related to the distance to the target is obtained from the stereo image obtained from the stereo camera 2, the angle of depression of the stereo camera 2 is the car C. The two-dimensional coordinates of the right camera coordinate system and the two-dimensional coordinates of the left camera coordinate system obtained from the stereo camera 2 are obtained regardless of the value within the range in which the target in front of the camera can be photographed. Thus, it can always be accurately converted into the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system. As a result, the coordinates of each target obtained from the radar 1 and the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system converted based on the parallax information are used. Rr, Rl, Tr, and Tl for converting the three-dimensional coordinates into the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system can always be accurately obtained. As a result, when the position of the obstacle is identified during traveling, for example, even when the depression angle of the stereo camera 2 fluctuates with time due to the suspension of the car C, it is obtained as described above. By using Rr, Rl, Tr, and Tl, it is possible to accurately compare the coordinates of the target obtained from the radar 1 and the coordinates of the target obtained from the stereo camera 2. As a result, it is possible to accurately identify a target that is highly likely to exist.

In the present embodiment, since Rr, Rl, Tr, and Tl are used when specifying the position of an obstacle during traveling, the three-dimensional radar coordinate system of each target obtained by the radar 1 is used. coordinates can be converted into two-dimensional coordinates of the two-dimensional coordinates and the left camera coordinate system of the right camera coordinate system that substantially contains depth information. Thus, obtained from the radar 1, the two-dimensional coordinates of the two-dimensional coordinates and the left camera coordinate system of the right camera coordinate system of each target object including the depth information substantially each obtained based on the stereo image Since the two-dimensional coordinates of the right camera coordinate system of the target and the two-dimensional coordinates of the left camera coordinate system can be compared, it is possible to specify a target that is highly likely to exist. Therefore, the targets can be easily matched using the coordinates of the target obtained from the radar and the coordinates of the target obtained from the stereo camera.

  Furthermore, in the present embodiment, when the position of the obstacle is specified during traveling, the depth ZLj of the target is obtained from the suggestion information, so the distance di from the target obtained from the radar 1; It is possible to compare the depth ZLj of the target that is considered to be the same as the target detected by the radar 1 and obtained from the stereo camera 2. Thereby, for example, when the two-dimensional coordinates of the target obtained from the radar 1 and the two-dimensional coordinates of the target obtained from the stereo camera 2 are close to each other, the distance di and the depth ZLj are greatly different from each other. Can be recognized as separate targets. On the other hand, for example, the two-dimensional coordinates of the target obtained from the radar 1 and the two-dimensional coordinates of the target obtained from the stereo camera 2 are equal or close to each other, and the distance di And the depth ZLj are equal or close to each other, they can be recognized as the same target.

  In this way, when specifying the position of an obstacle during traveling, only the two-dimensional coordinates of the target obtained from the radar 1 and the two-dimensional coordinates of the target obtained from the stereo camera 2 are compared with each other. In addition, since the distance di and the depth ZLj are compared with each other, the two-dimensional coordinates of the target obtained from the radar 1 and the two-dimensional coordinates of the target obtained from the stereo camera 2 are close to each other. Even if it is, it can be specified correctly whether they are different from each other or the same.

  Further, in the present embodiment, in the calibration, when the three-dimensional coordinates (Xmi, Ymi, Zmi) of each target detected by the radar 1 are not distributed on the reflecting plate 6, it is obtained by calculation. Since the three-dimensional coordinates (Xmi, Ymi, Zmi) are corrected by using the depression angle ξ and the deviation angle φ of the radar 1, errors due to the orientation of the radar 1 are removed from Rr, Rl, Tr, Tl. Can do. As a result, the coordinates of the target obtained from the radar 1 and the coordinates of the target obtained from the stereo camera 2 can be compared very accurately, so that the position of the obstacle is specified during traveling. In this case, the coordinates of the target detected by the radar 1 and the coordinates of the target detected by the stereo camera 2 can be used to match the targets very accurately.

  Further, in the present embodiment, when the position of an obstacle is specified during traveling, the rate of change in the distance di from the target obtained from the radar 1 and the radar 1 obtained from the stereo camera 2 are used. Since the ratio of the change in the depth ZLj of the target that seems to be the same as the detected target is compared, the two-dimensional coordinates of the target obtained from the radar 1 and the target obtained from the stereo camera 2 are used. Even when the two-dimensional coordinates of the mark are close to each other, it can be very accurately specified whether they are different from each other or the same. Thereby, when specifying the position of an obstacle during traveling, the coordinates of the targets detected by the radar 1 and the coordinates of the targets detected by the stereo camera 2 are used to accurately identify the targets. Can be matched.

  In the present embodiment, after specifying the position and type of the obstacle 7, the radar information (θi, di, vi) of the obstacle 7 detected by the radar 1 and the obstacle detected by the stereo camera 2 are used. Even if one of the 7 stereo images has some trouble (for example, noise mixing) and it becomes impossible to use any one of the information, only the information that can be used is used. It can be used to track the position of the obstacle 7. In this case, from the time when the position of the obstacle 7 is tracked using both pieces of information, the position of the obstacle 7 is switched at the moment of switching to track the position of the obstacle 7 using only one piece of information. Does not fluctuate discontinuously. When the position of the obstacle 7 is tracked, since the equations (1), (7), (10) to (13) including constants obtained by calibration are used, the object detected by the radar 1 is used. This is because the coordinates of the target and the coordinates of the target detected by the stereo camera 2 can be converted into almost the same coordinates in one coordinate system.

  While the present invention has been described with reference to the embodiment and examples, the present invention is not limited to these, and various modifications are possible.

It is a schematic block diagram of the target detection apparatus which concerns on one embodiment of this invention. It is a figure for demonstrating the example of 1 structure of the reflecting plate of FIG. It is a figure for demonstrating the other structural example of the reflecting plate of FIG. FIG. 5 is a layout diagram when performing calibration of the target detection device. It is a side view for explaining a depression angle ξ. It is a top view for demonstrating deviation | shift angle (phi). It is a flowchart for demonstrating the calibration procedure of a target detection apparatus. It is a flowchart for demonstrating the procedure following FIG. It is a flowchart for demonstrating the procedure following FIG. 10 is a flowchart for explaining a procedure following FIG. 9. It is a flowchart for demonstrating the procedure about the detection of an obstruction. FIG. 12 is a flowchart for explaining the procedure following FIG. 11. It is a flowchart for demonstrating the procedure following FIG. It is a perspective view for demonstrating the procedure about the detection of an obstruction.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Radar, 2 ... Stereo camera, 21 ... Right camera, 22 ... Left camera, 3 ... Control part, 4 ... Memory | storage part, 5 ... Display part, 6 ... Reflector plate, 6A ... Reflection block, 6B ... Absorption block, 6C ... Flat plate, 6D ... Reflector, 7 ... Obstacle, A ... Scan plane, B ... (Xm, Zm) plane, C ... Automobile, d1, d2 ... Radius, S1 ... Traveling road surface, S2 ... Non-traveling road surface, X1 ... Axle , X2, X3 ... line segment, ξ ... depression angle, φ ... deviation angle.

Claims (3)

A radar for acquiring radar information in front of the vehicle;
A stereo camera having a right camera and a left camera for acquiring a stereo image in front of the vehicle;
A control unit for processing a radar image acquired by the radar, a stereo image including a two-dimensional image acquired by the right camera and a two-dimensional image acquired by the left camera;
With
The controller is
The three-dimensional coordinates of the radar coordinate system of each target are acquired from the radar information obtained by the radar, and the three-dimensional coordinates of the radar coordinate system of each target acquired from the radar information are distributed on one plane. If not, radar side three-dimensional coordinate acquisition means for correcting the three-dimensional coordinates using the radar depression and deviation angles;
The two-dimensional coordinates of the right camera coordinate system of each target from the two-dimensional image obtained by the right camera, and the left camera coordinate system of each target from the two-dimensional image obtained by the left camera. Camera-side two-dimensional coordinate acquisition means for respectively extracting the two-dimensional coordinates of
Parallax information acquisition means for acquiring parallax information from the two-dimensional coordinates of the right camera coordinate system of each target and the two-dimensional coordinates of the left camera coordinate system of each target;
Using the parallax information, the 2D coordinates of the right camera coordinate system and the 2D coordinates of the left camera coordinate system of each target are converted into 3D coordinates of the right camera coordinate system and 3D coordinates of the left camera coordinate system. Camera side three-dimensional coordinate acquisition means,
The three-dimensional coordinates of the radar coordinate system of each target obtained based on the radar information, and the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system of each target obtained based on the stereo image. by using the dimension coordinates, have a conversion coefficient acquisition means for obtaining a conversion coefficient for converting the 3-dimensional coordinates of the radar coordinate system into the three-dimensional coordinates and three-dimensional coordinates of the left camera coordinate system of the right camera coordinate system,
The radar-side three-dimensional coordinate acquisition means performs linear approximation on the three-dimensional coordinates of the radar coordinate system of each target acquired from the radar information using the least square method, and the three-dimensional of each target obtained thereby A first straight line including coordinates and a second straight line that intersects perpendicularly in a plane perpendicular to the one plane are obtained, and an angle at which the second straight line and the axle of the vehicle intersect is obtained as a deviation angle of the radar. Calibration device to do. The calibration apparatus according to claim 1 , wherein the right camera and the left camera are arranged at the same height. A radar for acquiring radar information in front of the vehicle, a stereo camera having a right camera and a left camera for acquiring a stereo image in front of the vehicle, radar information acquired by the radar, and 2 acquired by the right camera A control unit for processing a stereo image composed of a two-dimensional image acquired by a two-dimensional image and the left camera,
Obtaining the three-dimensional coordinates of the radar coordinate system of each target from the radar information obtained by the radar;
The two-dimensional coordinates of the right camera coordinate system of each target from the two-dimensional image obtained by the right camera, and the left camera coordinate system of each target from the two-dimensional image obtained by the left camera. Each of the two-dimensional coordinates of
Parallax information is acquired from the two-dimensional coordinates of the right camera coordinate system of each target and the two-dimensional coordinates of the left camera coordinate system of each target;
Using the parallax information, the 2D coordinates of the right camera coordinate system and the 2D coordinates of the left camera coordinate system of each target are converted into 3D coordinates of the right camera coordinate system and 3D coordinates of the left camera coordinate system. And
When the three-dimensional coordinates of the radar coordinate system of each target are acquired based on the radar information, and the three-dimensional coordinates of the radar coordinate system of each target acquired from the radar information are not distributed on one plane. Corrects the three-dimensional coordinates using the depression angle and deviation angle of the radar,
At the time of the correction, a linear approximation is performed on the three-dimensional coordinates of the radar coordinate system of each target acquired from the radar information using the least square method, and a first straight line including the three-dimensional coordinates of each target obtained by this is obtained. Obtaining a second straight line perpendicular to the plane perpendicular to the one plane, and obtaining an angle at which the second straight line and the axle of the vehicle intersect as a deviation angle of the radar;
The three-dimensional coordinates of the radar coordinate system of each target obtained based on the radar information, and the three-dimensional coordinates of the right camera coordinate system and the left camera coordinate system of each target obtained based on the stereo image. A calibration method characterized in that, using the three-dimensional coordinates, a conversion coefficient for converting the three-dimensional coordinates of the radar coordinate system into the three-dimensional coordinates of the right camera coordinate system and the three-dimensional coordinates of the left camera coordinate system is obtained.

Images