JP2015159397A - Image processing apparatus, image processing system, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing apparatus that quickly and accurately detects parallax.SOLUTION: An image processing apparatus 100 forms a parallax image including parallax information by using images photographed with two imaging means 11 and 12, and includes a plurality of parallax detection means 24 and 25 for detecting parallax in a unit of less than one pixel with different accuracies, and switching means 23 for causing one of the plurality of parallax detection means to detect parallax in a unit of less than one pixel, where the plurality of parallax detection means operate in parallel to each other.

Description

  The present invention relates to an image processing apparatus that creates a parallax image including parallax information using images captured by two imaging units.

  In a distance measurement system using a stereo camera, the corresponding points of two images are searched by block matching or the like while shifting the corresponding points in the horizontal direction, and the parallax between corresponding points with the highest similarity is obtained. Since this parallax is a parallax obtained from corresponding points between pixels, it becomes a parallax with integer pixel accuracy. On the other hand, in order to improve the measurement accuracy of the distance to a distant subject where the parallax becomes smaller, the distance measurement system may determine the parallax with sub-pixel accuracy.

  As a method of obtaining the parallax between the subjects of the left and right images with sub-pixel accuracy, several similarities between the two blocks calculated for each search position are extracted centering on the maximum similarity, and data of these similarities is obtained. Methods for performing operations such as parabolic fitting are known. By such calculation, the sub-pixel accuracy parallax can be obtained.

  However, for example, the method of obtaining parallax by parabola fitting has the following inconvenience because it is based on the premise that data of several similarities can be approximated to a quadratic curve such as a parabola shape (parabola). That is, since it may be difficult to approximate to a quadratic curve depending on the accuracy of integer pixel parallax for obtaining sub-pixel accuracy parallax, the accuracy of sub-pixel accuracy parallax obtained by approximation is reduced. .

  On the other hand, a technique for improving the accuracy of parallax with subpixel accuracy is known (see, for example, Patent Document 1). In Patent Document 1, two sub-pixel corresponding position calculation means including a first sub-pixel corresponding position calculation means and a second sub-pixel corresponding position calculation means are provided, and the reliability of the similarity at the similarity peak position is obtained. Accordingly, an image motion detection apparatus that switches between first and second subpixel corresponding position calculation means is disclosed.

  However, the image motion detection apparatus described in Patent Document 1 has a problem that it takes time to search for corresponding points. That is, in Patent Document 1, since the POC (phase-only correlation method) is used as the second subpixel corresponding position calculation means that is a highly accurate calculation method, it takes time to calculate the parallax with subpixel accuracy. Therefore, even if the second subpixel corresponding position calculation means is used only when the reliability of the corresponding point search result is low, it still takes time to obtain the parallax from the image.

  In view of the above problems, an object of the present invention is to provide an image processing apparatus that performs high-precision parallax detection at high speed.

  In view of the above problems, the present invention is an image processing apparatus that creates a parallax image including parallax information using images captured by two imaging units, and has different parallax in units of less than one pixel. A plurality of parallax detection means for detecting with accuracy, and a switching means for causing one of the plurality of parallax detection means to detect parallax in units of less than one pixel, and the plurality of parallax detection means It is characterized by operating in parallel.

  An image processing apparatus that can detect parallax with high accuracy at high speed can be provided.

It is an example of the figure explaining schematic operation | movement of an image processing apparatus. It is an example of the hardware block diagram of an image processing apparatus. It is an example of the functional block diagram of a parallax calculation part. It is a figure which shows an example of a reference | standard image and a comparison image. It is an example of the figure explaining the calculation of integer pixel parallax. It is an example of the figure which shows the calculation result of the similarity within a search range. It is an example of the figure which shows the similarity of 5 points | pieces centering on the search position of the maximum similarity. It is a figure which shows an example of the calculation result of the subpixel parallax by parabolic fitting. It is a figure which shows an example of the calculation result of the subpixel parallax calculated by a high-order polynomial. It is an example explaining the switching of the calculation method of the sub pixel parallax based on the integer pixel parallax and the threshold value T. It is an example of the timing chart figure of the parallel processing by a high-order polynomial estimation calculating part and a parabolic fitting calculating part. FIG. 3 is an example of a diagram schematically illustrating parallel operations of a high-order polynomial estimation calculation unit and a parabolic fitting calculation unit. It is an example of the sequence diagram which shows the procedure in which an image processing apparatus calculates subpixel parallax. (Example 2) which is an example of the functional block diagram of a parallax calculation part. It is an example of the figure explaining switching of the calculation method of the sub pixel parallax based on integer pixel parallax and threshold value T1,2. It is an example of the figure which illustrates typically selection of the calculation method by an estimation system branch part. (Example 4) which is an example of the functional block diagram of a parallax calculation part. It is an example of the figure explaining the change of the threshold value T in the daytime and nighttime. It is an example of the figure explaining the change of the threshold value T according to a vehicle speed. FIG. 10 is an example of a hardware configuration diagram of an image processing apparatus (Example 5). It is a figure which shows an example of an object pixel range typically.

  DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings.

  FIG. 1 is an example of a diagram illustrating a schematic operation of the image processing apparatus according to the present embodiment. The image processing apparatus 100 performs block matching with the other image while shifting one of the two images in the horizontal direction, and calculates the degree of similarity within a predetermined search range. As a result, the search position with the highest similarity is determined, and integer pixel precision parallax (hereinafter referred to as integer pixel parallax) is obtained. Sub-pixel parallax can be obtained by calculating several similarities around integer pixel parallax.

  In sub-pixel parallax calculation, calculation time is sacrificed if sub-pixel parallax calculation is performed with high accuracy but high calculation cost. In addition, when sub-pixel parallax calculation is performed at high speed but low in calculation cost, sub-pixel accuracy is lowered.

  Therefore, the image processing apparatus 100 according to the present embodiment prepares at least two sub-pixel precision parallax calculation methods (hereinafter referred to as sub-pixel parallax), and the distance to the subject (examples 1, 2, and 4). The calculation method is switched according to the image range (Examples 3 and 5). Each operation is performed in parallel.

For example, in the figure, three calculation methods are possible: high-order polynomial estimation (6th order), high-order polynomial estimation (4th order), and parabolic fitting. Each calculation cost and subpixel accuracy are as follows as an example.
High-order polynomial estimation (sixth-order): Calculation cost (high) Subpixel accuracy (high)
High-order polynomial estimation (4th order) ... Calculation cost (Medium) Subpixel accuracy (Medium)
Parabolic fitting: computational cost (low) Subpixel accuracy (low)
For example, when the calculation method is switched according to the distance to the subject, the image processing apparatus 100 switches the calculation method as follows.
When the distance is short ... When the parabolic fitting distance is medium ... Higher order polynomial estimation (4th order)
When the distance is long: Higher order polynomial estimation (6th order)
Since the parallax decreases with increasing distance, a highly accurate sub-pixel parallax is required. For example, in this example, the subpixel parallax is calculated with a calculation method with higher subpixel accuracy as the distance is longer. Can be sought. In addition, when the distance is short, the subpixel parallax can be obtained by a calculation method with low calculation cost even if the accuracy is not high.

  In addition, since each calculation is performed in parallel, the time for obtaining the parallax of the entire image can be reduced compared to the case where all the images are obtained by a calculation method with low calculation cost (difference in calculation cost of each calculation method and distant Depending on how much of the pixel ratio).

  Therefore, the image processing apparatus according to the present embodiment can measure a distance that balances processing speed and accuracy.

  FIG. 2 shows an example of a hardware configuration diagram of the image processing apparatus. The image processing apparatus 100 includes a left camera 11, a right camera 12, and a parallax calculation unit 13. Each of the left camera 11 and the right camera 12 has a horizontal and parallel optical axis, and is fixed to a housing (not shown).

  In this embodiment, the left camera 11 captures a comparative image and the right camera 12 captures a reference image, but this relationship may be reversed. In block matching, a reference image is used as a reference and compared with a comparison image while shifting the pixel position in the horizontal direction.

  The left camera 11 and the right camera 12 include an imaging device such as a CMOS or a CCD, an optical system, and a control unit that controls gain, shutter speed, and the like. Examples of the resolution (number of pixels) of the image sensor include VGA, SVGA, HD, and full HD. The left camera 11 and the right camera 12 are cameras that acquire luminance information, but may be cameras that can acquire color information by including a color filter.

  The left camera 11 and the right camera 12 output a reference image and a comparison image every predetermined cycle. General image correction is performed on the reference image and the comparison image. By this image correction, the reference image and the comparison image are corrected so that no difference other than parallax occurs. Image correction is possible by prior calibration. When the left camera 11 and the right camera 12 are installed, for example, a calibration subject (for example, a checkered chart) is photographed. A LUT (Look for Geometric Transformation) that compares two images and transforms the image data so that hardware internal error factors such as camera lens distortion, optical axis deviation, focal length deviation, and image sensor distortion are minimized. Up Table) is generated. The image processing apparatus 100 performs image correction with reference to the LUT.

  The parallax calculation unit 13 calculates the parallax from the reference image and the comparison image. The parallax calculation method will be described later. The parallax calculation unit 13 outputs a parallax image in which parallax information (or distance information) may be associated with each pixel or pixel block of the reference image to the external device 14.

  The external device 14 performs some control using the parallax image. An example of the external device 14 is an in-vehicle device. That is, the image processing apparatus 100 is mounted on a vehicle (may be a moving body other than the vehicle), and the in-vehicle apparatus performs driving assistance using the parallax image.

  The external device 14 as an in-vehicle device recognizes a preceding vehicle, a pedestrian, and the like and detects a distance from the preceding vehicle or the pedestrian. In addition, a collision prediction time is calculated by detecting a relative speed with respect to a preceding vehicle or a pedestrian, and warning, seat belt hoisting, automatic braking, and the like are performed according to the collision prediction time.

  In addition to detecting the distance, the external device 14 may recognize lines and shoulders that divide the traveling lane. In this case, the external device 14 can prevent the vehicle from deviating from the travel lane or can control to travel near the center of the travel lane.

  Note that the control performed by the external device 14 using a parallax image may be any control, and is not limited to the above-described content. Moreover, you may mount other than a mobile body.

[Configuration example]
FIG. 3 is an example of a functional block diagram of the parallax calculation unit 13. The stereo image acquisition unit 21 acquires a comparative image from the left camera 11 and a reference image from the right camera 12 that are installed in parallel in the left-right direction. The reference image and the comparison image are acquired at a timing that can be regarded as simultaneous or simultaneous.

  The similarity calculation unit 22 calculates the similarity between the reference image and the comparison image within a predetermined search range. An integer pixel parallax is obtained by this similarity. Details are described below. The subpixel parallax is obtained using the similarity calculated by the similarity calculation unit 22.

  The estimation method branching unit 23 switches the subpixel parallax calculation method (subpixel estimation method) according to the integer pixel parallax (that is, according to the distance to the subject). The estimation method branching unit 23 selects either the high-order polynomial estimation calculation unit 24 with high accuracy but high calculation cost, or the parabolic fitting calculation unit 25 with slightly lower accuracy but low calculation cost. Details will be described later. Note that there may be three or more sub-pixel parallax calculation units as shown in FIG.

  The high-order polynomial estimation calculation unit 24 calculates subpixel parallax by high-order polynomial estimation based on, for example, a quartic equation, and the parabola fitting calculation unit 25 calculates subpixel parallax by parabola fitting. The high-order polynomial estimation calculation unit 24 and the parabolic fitting calculation unit 25 are each implemented by hardware such as LSI, and can be processed in parallel.

  Note that the parallax image generation unit 26 generates a parallax image by associating the sub-pixel parallax obtained for each pixel or each pixel block with the pixel or the pixel block of the reference image.

[Calculation of integer pixel parallax]
FIG. 4 is an example of a reference image and a comparative image, and FIG. 5 is an example of a diagram for explaining integer pixel parallax calculation. In the reference image and the comparison image, the position of the subject is imaged in the horizontal direction due to the parallax. When the left camera 11 is a comparison image and the right camera 12 is a reference image, the subject of the comparison image is shifted in the right direction with reference to the reference image. Therefore, in block matching, the similarity may be calculated while moving (searching) the reference image only in the right direction.

  For example, the parallax of the truck of the preceding vehicle is d1, and the parallax of the right pile is d2. The closer the distance to the subject, the greater the parallax between the reference image and the comparative image, and the farther the distance to the subject, the smaller the parallax between the reference image and the comparative image. Therefore, as is apparent from FIG. 4, parallax d1 <parallax d2.

The distance z to the subject is obtained from the following equation using a known parameter, the base line length B between the left camera 11 and the right camera 12, the focal length f of the camera lens, and the calculated parallax d. .
z = Bf / d
Next, calculation of integer pixel parallax will be described.

  (i) The parallax calculation unit 13 determines the pixel of interest (x, y) in order from the top left vertex of the reference image, for example.

  (ii) The parallax calculation unit 13 calculates the luminance value of each pixel of the reference image in a pixel near the target pixel (x, y) (for example, a window such as 5 × 5 to 11 × 11 centered on the target pixel). A process of comparing the luminance value of each pixel of the comparison image is performed. Specifically, for each pixel of the reference image window p and each pixel of the comparison image window q at the same position as the reference image window p, a luminance difference is calculated for each pixel and summed (or the difference) Sum the squares). Therefore, the smaller the sum of luminance differences is, the higher the similarity (matching degree) is.

  (iii) The parallax calculation unit 13 moves the window q of the comparison image by one pixel in the horizontal direction from the coordinates (x, y) as a base point, and similarly calculates the sum of luminance differences from the window p of the reference image. To do.

  (iv) The parallax calculation unit 13 repeatedly calculates the sum of luminance differences by moving the window q of the comparison image in the horizontal direction pixel by pixel. Then, a comparison is made when the similarity is maximum (the sum of luminance differences is minimum) within a predetermined horizontal search range D (for example, 64 pixels as an example in this embodiment as will be described later). The amount of horizontal movement of the image window p is output as an integer pixel parallax d.

  The similarity calculated in this way may be referred to as cost. Further, the sum of the luminance value differences for each pixel in the windows p and q is called SAD (Sum of Absolute Differences), and the sum of the squares of the luminance value differences is called SSD (Sum of Squared Differences).

  FIG. 6 is an example of a diagram illustrating a calculation result of the similarity within the search range. The search range D is 64 pixels. When the parallax is obtained in the range of 0 to 64 pixels, the actual movement range of the window q is “−2 to 66”. That is, an extra search is performed 2 pixels to the left of the leftmost pixel in the search range D, and an extra search is performed 2 pixels to the right of the rightmost pixel in the search range D.

  The margin of ± 2 pixels is to secure a total of five similarities centering on the search position having the maximum similarity, which is necessary for calculation of higher-order polynomial estimation described later.

  When sub-pixel parallax is calculated only by parabolic fitting, not by high-order polynomial estimation, a margin of ± 1 is sufficient. In this embodiment, a margin of ± 2 pixels is set so that the sub-pixel parallax may be obtained by high-order polynomial estimation in any pixel.

  In the search range designated in the horizontal direction (−2 <d <66), the similarity C (x, y, d) is calculated for each search position (horizontal position). x and y are the coordinates of the reference image on which the parallax is calculated, d is the integer pixel parallax, and C is the similarity. The similarity C (x, y, d) becomes smaller as the windows p and q are more similar.

  As shown in FIG. 6, the search position having the highest similarity is set as the integer pixel parallax d of the comparison image with respect to the reference image. Sub-pixel parallax is calculated from this parallax d and the data of the similarity of five points (or three points) in the vicinity thereof. In FIG. 6, since the search position (horizontal movement position of the window q) is 16 in the highest similarity, the integer pixel parallax d is 16.

[Calculation of sub-pixel parallax]
The sub-pixel parallax calculation will be described with reference to FIGS. FIG. 7 shows the similarity of five points near the search position with the maximum similarity. Accordingly, in FIG. 7, the five similarities of search positions 14, 15, 16, 17, and 18 are scored. Among these, the parabola fitting calculation unit 25 uses 3 points, and the high-order polynomial estimation calculation unit 24 uses 5 points to calculate the sub-pixel parallax.

  FIG. 8 is a diagram illustrating an example of a calculation result of sub-pixel parallax by parabolic fitting. The subpixel parallax s in which the quadratic curve obtained by parabola fitting takes the minimum value has a similarity of R (0) for the search position 16, R (-1) for the similarity of the search position 15, and a similarity of the search position 17. If the degree is R (1), it can be obtained from equation (1).

図９は、高次多項式により演算されるサブピクセル視差の演算結果の一例を示す図である。以下手順を説明する。
(i)まず、高次多項式推定演算部２４は、最小二乗法により４次曲線へのフィッティングを行う。
ｙ＝ax⁴+bx³＋cx²+dx
(ii) 類似度が最大になる探索位置１６の類似度をR(0), 探索位置１４の類似度をR(-2)、探索位置１５の類似度をR(-1) 、探索位置１７の類似度をR(1)、探索位置１８の類似度をR(2)とする。これらを使ってｙｐ、ｙｍを次のように定義する。
ｙｐ＝R(-1)＋R(1)
ｙｍ＝R(1)-R(-1)
ｙｐ、ｙｍによりa,b,c,dは以下のように表すことができる。
a＝｛R(-2)＋R(2)-４yp＋6R(0)｝／24
b＝｛R(2)-R(-2)-2ym＋6R(0)｝／12
c=（yp/2）-R(0)-a
d=（ym/2）-b
(iii)４次曲線の最小値を算出するために、ｘについて偏微分し勾配を求め、その勾配が０（ゼロ）になるところをニュートン法で求める。偏微分の結果は以下のようになる。

FIG. 9 is a diagram illustrating an example of the calculation result of the sub-pixel parallax calculated by a high-order polynomial. The procedure will be described below.
(i) First, the high-order polynomial estimation calculation unit 24 performs fitting to a quartic curve by the least square method.
y = ax ⁴ + bx ³ + cx ² + dx
(ii) The similarity of the search position 16 that maximizes the similarity is R (0), the similarity of the search position 14 is R (-2), the similarity of the search position 15 is R (-1), and the search position 17 Is R (1), and the similarity at the search position 18 is R (2). Using these, yp and ym are defined as follows.
yp = R (-1) + R (1)
ym = R (1) -R (-1)
a, b, c, and d can be expressed as follows using yp and ym.
a = {R (-2) + R (2) -4yp + 6R (0)} / 24
b = {R (2) -R (-2) -2ym + 6R (0)} / 12
c = (yp / 2) -R (0) -a
d = (ym / 2) -b
(iii) In order to calculate the minimum value of the quartic curve, partial differentiation is performed with respect to x to obtain a gradient, and a place where the gradient becomes 0 (zero) is obtained by the Newton method. The result of partial differentiation is as follows.

(iv)勾配がゼロになることは、式（２）の値が、予め定められた閾値Ｔ以下になることで判断される。または、決められた計算回数まで勾配を計算したら、勾配がゼロになったと推定する。勾配がゼロになったと見なされた探索位置がサブピクセル視差ｓである。

(iv) It is determined that the gradient becomes zero when the value of the equation (2) becomes equal to or less than a predetermined threshold value T. Alternatively, when the gradient is calculated up to a predetermined number of calculations, it is estimated that the gradient has become zero. The search position at which the gradient is regarded as zero is the sub-pixel parallax s.

なお、xnに初期値が必要であるが、初期値は適当な値（例えば、整数ピクセル視差など）を設定する。なお、高次多項式において勾配がゼロとなる場所の求め方は、ニュートン法に限られず、多様なアルゴリズムを用いてよい。

Note that an initial value is required for xn, but an appropriate value (for example, integer pixel parallax) is set as the initial value. Note that the method for obtaining the location where the gradient is zero in the high-order polynomial is not limited to the Newton method, and various algorithms may be used.

[Switching calculation method]
The subpixel parallax obtained by high-order polynomial estimation is often more accurate than the subpixel parallax obtained by parabolic fitting. However, the calculation cost of high-order polynomial estimation is high for parabolic fitting.

  Therefore, the estimation method branching unit 23 switches the sub-pixel parallax calculation method according to the integer pixel parallax calculated by the similarity calculation unit 22.

  As shown in FIG. 4, as the distance from the subject increases, the parallax decreases. Therefore, as the distance from the subject increases, the accuracy of subpixel parallax needs to be improved. In other words, when the distance to the subject is short, the parallax is large, so even if the accuracy of the subpixel parallax is somewhat worse, the accuracy of the distance does not decrease so much.

  Therefore, as shown in FIG. 10, a threshold T is provided, and the sub-pixel parallax calculation method is switched according to the comparison result between the integer pixel parallax and the threshold T. FIG. 10 is an example of a diagram illustrating switching of the sub-pixel parallax calculation method based on the integer pixel parallax and the threshold T.

  When the parallax is equal to or smaller than the threshold T (when the distance to the subject is far), the estimation method branching unit 23 uses a high-order polynomial estimation calculation unit 24 with high accuracy. If the parallax is larger than the threshold T (when the distance to the subject is short), the parabola fitting calculation unit 25 having a low calculation cost is used.

  As a result, the calculation time is increased only in pixels for which it is preferable to obtain a highly accurate sub-pixel parallax, so that the processing speed and accuracy can be balanced.

[About parallel processing]
FIG. 11 is an example of a timing chart of parallel processing by the high-order polynomial estimation calculation unit 24 and the parabolic fitting calculation unit 25. The high-order polynomial estimation calculation unit 24 and the parabolic fitting calculation unit 25 operate in parallel and have different calculation times. “Calculation time of high-order polynomial estimation calculation unit> Calculation time of parabolic fitting calculation unit”. In the figure, the calculation time of the high-order polynomial estimation calculation unit 24 is about twice as long as the calculation time of the parabolic fitting calculation unit 25. However, the calculation time is only an example and may be less than twice or more than twice.

  Assuming that the calculation time t1 of one pixel of the higher-order polynomial estimation calculation unit 24 is about twice the calculation time t2 of one pixel of the parabolic fitting calculation unit 25, the parallax of N pixels (for example, the number of pixels of one image) The calculation time will be described.

The calculation time Tp when sub-pixel parallax is calculated by parabolic fitting is as follows.
Tp = N × t2
The calculation time Th when sub-pixel parallax is calculated by high-order polynomial estimation is as follows.
Th = N × t1 = Tp × 2
The calculation time Tave when N / 2 pixels are calculated by parabolic fitting and N / 2 pixels are calculated by high-order polynomial estimation, respectively, is calculated as follows.
Tave = (N / 2) × t2 + (N / 2) × t1 = (Tp + Th) /2=1.5 Tp
When high-order polynomial estimation and parabola fitting are performed in order instead of in parallel, if there is even one pixel that calculates sub-pixel parallax by high-order polynomial estimation, sub-pixel parallax of all pixels is calculated by parabola fitting. The calculation time becomes longer.

  However, in this embodiment, high-order polynomial estimation and parabolic fitting are operated in parallel, so that depending on the number of pixels calculated by high-order polynomial estimation, the calculation time of all pixels can be reduced, and the sub-pixel parallax of all pixels can be calculated by parabolic fitting. Can be suppressed to less than the time for calculating.

  FIG. 12 is an example of a diagram schematically illustrating the parallel operation of the high-order polynomial estimation calculation unit 24 and the parabolic fitting calculation unit 25. The similarity calculation unit 22 sequentially calculates a search position having the maximum similarity for each pixel of interest and five similarities centered on the search position, and inputs them to the estimation method branching unit 23.

  The estimation method branching unit 23 compares the threshold T with the search position, selects a subpixel parallax estimation method, and calculates the similarity of five points centered on the input search position by the high-order polynomial estimation calculation unit 24. The three points of similarity are stored in the buffer 42 in the buffer 41 of the parabolic fitting calculation unit 25, respectively. Each buffer has a queue structure.

  If the buffer is not empty, the high-order polynomial estimation calculation unit 24 and the parabola fitting calculation unit 25 calculate the subpixel parallax with respect to the stored similarity and store it in the line buffer 43 in the subsequent stage. Note that the parabolic fitting calculation unit 25 may calculate the subpixel accuracy of pixels adjacent in the horizontal direction by skipping one to several. An object at a short distance does not require accuracy as far away, and can be obtained by interpolation from adjacent pixels.

  The line buffer 43 stores the sub-pixel parallax of pixels for one line in the horizontal direction of the parallax image. When the sub-pixel parallax is stored in the line buffer 43, the high-order polynomial estimation calculation unit 24 and the parabola fitting calculation unit 25 store the sub-pixel parallax at a position corresponding to the pixel on which the parallax calculation is performed.

  As described above, the high-order polynomial estimation calculation unit 24 and the parabolic fitting calculation unit 25 can calculate the sub-pixel parallax with respect to the line buffer 43 in parallel. Note that the parabolic fitting and high-order polynomial estimation do not necessarily fill in the line buffer 43 sequentially from the left because the computation time is different.

  When one of parabolic fitting and high-order polynomial estimation is completed earlier in one line, the high-order polynomial estimation calculating unit 24 or the parabolic fitting calculating unit 25 that has not yet finished calculating the subpixel parallax is: Wait until the calculation results for one line are complete. When the calculation results for one line are obtained, the parallax for one line stored in the line buffer 43 is output (stored in one line of the parallax image). Thereafter, calculation of the subpixel parallax of each pixel of the next line is started.

  When computing in parallel as shown in FIG. 12, the computation time of all the pixels N is as follows. It is assumed that the subpixel parallax is calculated by parabolic fitting of N / 2 pixels and high-order polynomial estimation of N / 2 pixels, respectively. Since the calculation times of Tp / 2 and Th / 2 are required and the calculation time is Tp / 2 <Th / 2, the longer calculation time, Th / 2 (= Tp), is required. Therefore, it takes about the same time as when all the pixels are estimated by parabolic fitting.

  It is assumed that the calculation time t1 of one pixel of the high-order polynomial estimation calculation unit 24 is about three times the calculation time t2 of one pixel of the parabolic fitting calculation unit 25. In this case as well, when sub-pixel parallax is calculated by parabolic fitting for 2N / 3 pixels and high-order polynomial estimation for N / 3 pixels, the same degree as when all pixels are calculated by parabolic fitting. Takes time.

  Therefore, by performing parabolic fitting and high-order polynomial estimation in parallel, an increase in calculation time can be suppressed, and highly accurate sub-pixel parallax can be obtained for pixels in which distant subjects are imaged.

[Operation procedure]
FIG. 13 is an example of a sequence diagram illustrating a procedure by which the image processing apparatus 100 calculates subpixel parallax.
S1: The left camera 11 outputs the captured comparison image to the stereo image acquisition unit 21.
S2: The right camera 12 outputs the captured reference image to the stereo image acquisition unit 21. The reference image and the comparison image are taken simultaneously or almost simultaneously.
S3, S4: The stereo image acquisition unit 21 outputs the reference image and the comparison image to the similarity calculation unit 22, respectively.
S5: The similarity calculation unit 22 calculates the window similarity by block matching, for example.
S6: The similarity calculation is not synchronized with the sub-pixel parallax calculation, and the similarity calculation unit 22 calculates the window similarity one after another, and the search position (integer pixel parallax) with the highest similarity is calculated. ) And five similarities centered on the search position are output to the estimation method branching unit 23.
S7: The estimation method branching unit 23 compares the search position with the threshold value T to determine the calculation method.
S8, S9: The estimation method branching unit 23 sets the similarity of three points centered on the search position to the parabola fitting calculation unit 25 according to the determination result of the calculation method, and the similarity of five points centered on the search position. Is output to the high-order polynomial estimation calculation unit 24.

S10, S11 and S12, 13 are performed in parallel.
S10: The parabolic fitting calculation unit 25 calculates the sub-pixel parallax in parallel with the high-order polynomial estimation calculation unit 24.
S11: The parabolic fitting calculation unit 25 stores the sub-pixel parallax in the line buffer 43 corresponding to the horizontal position of the pixel for which the parallax has been calculated.
S12: The high-order polynomial estimation calculation unit 24 calculates the sub-pixel parallax in parallel with the parabolic fitting calculation unit 25.
S13: The high-order polynomial estimation calculation unit 24 stores the sub-pixel parallax in the line buffer 43 corresponding to the horizontal position of the pixel for which the parallax has been calculated.

  Thereafter, when the subpixel parallax for one horizontal line is stored in the line buffer 43, the calculation of the subpixel parallax for the next one line is started.

  As described above, in this embodiment, for a distant subject that requires high accuracy, the subpixel parallax is calculated using a calculation method with high subpixel accuracy, and otherwise, the subpixel parallax is calculated using a calculation method with low calculation cost. Obtain the pixel parallax and perform each operation in parallel. As a result, an increase in time for obtaining the parallax of the entire image is suppressed, and a distance with a balance between processing speed and accuracy can be measured.

  In this embodiment, a calculation method called parabolic fitting which is lower in cost and accuracy than high-order polynomial estimation is adopted, but there is another calculation method called equiangular straight line fitting. Since the equiangular straight line fitting and the parabola fitting do not have clear superiority or inferiority (depending on the calculation method or the like), the equiangular straight line fitting may be employed instead of the parabolic fitting.

  In the first embodiment, a calculation method having different calculation cost and subpixel parallax accuracy is selected from the two calculation methods. However, in this embodiment, a calculation method having different calculation cost and subpixel parallax accuracy is selected from the three calculation methods. The image processing apparatus 100 will be described.

  FIG. 14 is an example of a functional block diagram of the parallax calculation unit 13 of the present embodiment. In the present embodiment, the constituent elements denoted by the same reference numerals in FIG. 3 perform the same function, and therefore, only the main constituent elements of the present embodiment may be mainly described. In this embodiment, a sixth-order polynomial estimation calculation unit 27 is provided. The high-order polynomial estimation calculation unit 24, the parabolic fitting calculation unit 25, and the sixth-order polynomial estimation calculation unit 27 operate in parallel.

  FIG. 15 is an example of a diagram illustrating switching of the calculation method of sub-pixel parallax based on integer pixel parallax and threshold values T1 and T2. Since the high-order polynomial estimation calculation unit 24 performs high-order polynomial estimation based on the fourth-order equation, in this embodiment, the sixth-order polynomial estimation calculation unit 27 is the calculation method with the highest calculation cost and accuracy.

In this embodiment, as shown in FIG. 15, two threshold values T1 and T2 are provided, and the calculation cost and the accuracy of the sub-pixel parallax can be calculated from three calculation methods according to the comparison result between the integer pixel parallax and the threshold values T1 and T2. Select a different calculation method.
Less than threshold T1... Greater than 6th-order higher-order polynomial estimation threshold T1 and less than threshold T2... Greater than 4th-order higher-order polynomial estimation threshold T2 .. parabolic fitting. You don't have to. Thereby, the calculation time can be further shortened.

  The time required to calculate the parallax in one image is determined by the calculation method having the longest calculation time, and therefore depends on how far away pixels are included in one image. However, since three operations are performed in parallel, the time required to calculate the parallax in one image may be shorter than in the first embodiment. In addition, since the subpixel accuracy required for a subject at a short distance is low, the balance between processing speed and accuracy can be maintained.

  As described above, the image processing apparatus 100 according to the present exemplary embodiment can reduce the calculation time for a subject that is close to that of the first exemplary embodiment, so that the parallax in one image can be maintained while maintaining a balance between processing speed and accuracy. There is a possibility that the time required for the calculation of can be reduced as compared with the first embodiment.

  In the first and second embodiments, the ratio of pixels in which a far subject and a close subject are imaged differs depending on the subject (how far a distant pixel is included) captured by the left camera 11 and the right camera. Is not constant. That is, the time until one parallax image is created becomes indefinite depending on the subject.

  Therefore, in this embodiment, a description will be given of the image processing apparatus 100 in which the time required for creating one parallax image is constant.

  In addition, the functional block diagram of the parallax calculation part 13 of a present Example may be the same as that of FIG. Since components having the same reference numerals in FIG. 3 perform the same function, only the main components of the present embodiment may be mainly described.

  FIG. 16 is an example of a diagram that schematically illustrates selection of a calculation method by the estimation method branching unit 23. In the present embodiment, as shown in FIG. 16, it is assumed that an area where high-precision subpixel parallax is required is known in advance. In this case, high-order polynomial estimation is used only in an area where high-precision subpixel parallax is required.

  In FIG. 16, a high-order polynomial estimation range is provided near the horizontal center of the image. For example, if the image has a width of 1280 pixels / line, the image processing apparatus 100 performs high-order polynomial estimation on the center 200 pixels and performs parabolic fitting on the remaining 1080 pixels. As a result, the calculation amount can be easily predicted, and the distance can be measured without variation in the calculation speed.

  The high-order polynomial estimation range (540 pixels to 740 pixels) is held in the estimation method branching unit 23 in advance. The estimation method branching unit 23 selects the high-order polynomial estimation calculation unit 24 when the horizontal position of the pixel of interest calculated by the similarity calculation unit 22 falls within the high-order polynomial estimation range.

  Note that the high-order polynomial estimation range is a range in which a subject that is important for the image processing apparatus 100 is highly likely to be captured. For example, when the image processing apparatus 100 is mounted on a moving body that moves forward, the high-order polynomial estimation range is in the center as shown in FIG. Therefore, if the range in which a subject that is important in the distance is imaged is known, the designer or the like can set the range as the high-order polynomial estimation range even if it is other than the center.

  Further, as described in the first embodiment, the range of the high-order polynomial estimation range is a wide range that is allowed as a creation time of one parallax image according to the calculation time of high-order polynomial estimation and the calculation time of parabolic fitting. Is set.

  Further, by determining the width of the high-order polynomial estimation range so that the high-order polynomial estimation and the parabolic fitting are completed in about the same time, the high-order polynomial estimation calculation unit 24 or the parabolic fitting is performed until the line buffer 43 is filled. The time for which the calculation unit 25 waits can be substantially eliminated.

  When there are three calculation methods as in the second embodiment, the image may be divided into three ranges and the calculation methods may be distributed.

  Further, the user may be able to set whether the high-order polynomial estimation range is fixed as in the present embodiment or switched according to the integer pixel parallax as in the first and second embodiments.

  According to the image processing apparatus of the present embodiment, by setting the high-order polynomial estimation range, the time required to create one parallax image can be made constant, and the high-order polynomial estimation calculation unit 24 or The time that the parabola fitting calculation unit 25 waits can be substantially eliminated.

  In the first embodiment, the threshold value T (threshold values T1 and T2 in the second embodiment) is constant. In this embodiment, an image processing apparatus 100 that can dynamically change the threshold value T will be described.

  By changing the threshold value T, the number of pixels to which high-order polynomial estimation is applied can be increased or decreased. Therefore, in a scene that requires high-precision subpixel parallax, the threshold value T is changed to increase the frequency of application of high-order polynomial estimation. Otherwise, the threshold value T can be changed so that the frequency of application of high-order polynomial estimation is reduced.

  FIG. 17 is an example of a functional block diagram of the parallax calculation unit 13 of the present embodiment. In the present embodiment, the constituent elements denoted by the same reference numerals in FIG. 3 perform the same function, and therefore, only the main constituent elements of the present embodiment may be mainly described. In this embodiment, the image processing apparatus 100 has a threshold value changing unit 28, and the sensor 15 is connected to the threshold value changing unit 28. The sensor 15 detects information for determining whether or not the scene changes the threshold value T.

[Scene 1 at night imaging]
Since the surroundings are dark at night, there is a possibility that sufficient accuracy cannot be obtained for the five points of similarity for high-order polynomial estimation. For this reason, when the image processing apparatus 100 creates a parallax image at night, it may be preferable to change the threshold T so that the application frequency of parabolic fitting is increased.

  FIG. 18 is an example of a diagram illustrating the change of the threshold T. 18A shows the threshold value T during the day, and FIG. 18B shows the threshold value T during the night. At night, the threshold T can be made smaller than during the day so that the parabolic fitting application frequency can be increased.

  The sensor 15 acquires the shutter speed from, for example, the left camera 11 or the right camera 12 in order to determine whether it is nighttime or daytime. Since the shutter speed becomes longer to increase the exposure time at night, it can be determined whether the shutter speed is night or daytime. Thereby, the threshold value T can be changed in a tunnel or the like even at night. In addition to the shutter speed, whether the current location is nighttime or daytime may be determined from time information and position information.

[Scene 2 When moving at high speed]
The faster the moving body moves, the farther the subject approaches the moving body in a short time. When moving at high speed, it is preferable to obtain a highly accurate sub-pixel parallax for a distant subject. For this reason, when the image processing apparatus 100 creates a parallax image during high-speed movement, it may be preferable to change the threshold T so that the application frequency of high-dimensional polynomial estimation increases.

  FIG. 19 is an example of a diagram illustrating the change of the threshold T. FIG. 19A shows the threshold value T during low-speed movement, and FIG. 19B shows the threshold value T during high-speed movement. When moving at high speed, the threshold T can be increased so that the frequency of application of high-dimensional polynomial estimation increases.

  The sensor 15 is a wheel speed sensor for detecting the vehicle speed. The threshold changing unit 28 has a map in which the threshold T is associated with the vehicle speed, and reads the threshold T from the map according to the vehicle speed detected by the sensor 15. Thereby, the threshold value T can be changed dynamically and in multiple stages according to the vehicle speed.

  As described above, the image processing apparatus 100 according to the present embodiment can change the application frequency of high-order polynomial estimation according to the scene by dynamically changing the threshold value T.

  Note that when there are three or more calculation methods as in the second embodiment, there are two or more thresholds. In this case, only the smallest threshold T1 may be changed, or all thresholds may be changed by the same amount. May be.

  In this embodiment, an image processing apparatus 100 that applies high-order polynomial estimation to a pixel range in which an object is recognized will be described.

  FIG. 20 shows an example of a hardware configuration diagram of the image processing apparatus 100. In the present embodiment, the components denoted by the same reference numerals in FIG. 2 perform the same functions, and therefore, only the main components of the present embodiment may be mainly described. In FIG. 20, the external device 14 has an object recognition processing unit 141. In this embodiment, since a known method may be adopted as the object recognition method, the object recognition will be briefly described.

  The object recognition processing unit 141 detects, for example, a pixel range in which the same degree of parallax is continuous as a recognition target candidate. Thereby, several rectangular areas are obtained. In the image as shown in FIG. 4, a preceding vehicle, an oncoming vehicle, a pile, and the like are recognition candidates.

  The object recognition processing unit 141 performs recognition processing of a car, a pedestrian, or the like that is determined as a target object on the rectangular area detected from the parallax image. For object recognition, methods using Haar-like feature quantities and methods using HOG (Histograms of Oriented Gradients) feature quantities are known. In the present embodiment, either may be used, or a combination may be used.

  The object recognition processing unit 141 outputs a rectangular area recognized as having a car, a pedestrian, or the like to the parallax calculation unit 13 as an object pixel range. The object pixel range is specified as the coordinates of the diagonal vertices of the rectangular area. FIG. 21 is a diagram schematically illustrating an example of the object pixel range. A preceding vehicle and an oncoming vehicle are surrounded as object pixel ranges 301 and 302.

  The parallax calculation unit 13 calculates sub-pixel parallax for the object pixel ranges 301 and 302 by high-order polynomial estimation. Since there is a high possibility that an object whose distance is to be accurately detected exists in the object pixel range, the distance from the object can be detected with high accuracy by obtaining a highly accurate sub-pixel parallax.

  When there are three or more calculation methods, the estimation method branching unit 23 selects the calculation method (for example, high-order polynomial estimation) with the highest accuracy.

  Note that the calculation method for regions other than the object pixel range may be switched by integer pixel parallax as in the first and second embodiments, or the calculation method may be switched depending on the location in the image as in the third embodiment.

  Therefore, according to the image processing apparatus 100 of the present embodiment, high-dimensional polynomial estimation is applied to the object pixel ranges 301 and 302 for which it is preferable that high-precision subpixel parallax is calculated by using the object recognition result. it can.

DESCRIPTION OF SYMBOLS 11 Left camera 12 Right camera 13 Parallax calculation part 14 External apparatus 22 Similarity calculation part 23 Estimation method branch part 24 Higher order polynomial estimation calculation part 25 Parabola fitting calculation part 26 Parallax image generation part 27 Sixth order polynomial estimation calculation part 100 Image processing apparatus

Japanese Patent No. 4862816

Claims (10)

An image processing apparatus that creates a parallax image including parallax information using images captured by two imaging units,
A plurality of parallax detection means for detecting parallax in units of less than one pixel with different accuracy;
One of the plurality of parallax detection means, and a switching means for detecting parallax in units of less than one pixel,
The image processing apparatus, wherein the plurality of parallax detection units operate in parallel. An integer parallax detection means for detecting parallax in units of one pixel;
The switching unit selects the parallax detection unit that detects parallax in units of less than one pixel with high accuracy as the parallax detected by the integer parallax detection unit is small, from the plurality of parallax detection units. The image processing apparatus according to claim 1. The switching unit compares a parallax in units of one pixel detected by the integer parallax detection unit with a threshold value, and determines one parallax detection unit of the plurality of parallax detection units.
The image processing apparatus according to claim 2. The switching unit determines one parallax detection unit from the plurality of parallax detection units according to the position of a pixel from which parallax is detected.
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. If the horizontal position of the pixel where the parallax is detected is near the center,
The switching unit selects the parallax detection unit having the highest parallax accuracy of less than one pixel from the plurality of parallax detection units;
The image processing apparatus according to claim 4. A first parallax detection unit that detects parallax in units of less than one pixel; and a second parallax detection unit that detects parallax in units of less than one pixel with lower accuracy than the first parallax detection unit. Have
When the moving body on which the image processing apparatus is mounted moves at night, the detection frequency changing unit increases the frequency with which the second parallax detection unit detects a parallax of less than one pixel as compared with the case of moving outside the night. Having
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A first parallax detection unit that detects parallax in units of less than one pixel; and a second parallax detection unit that detects parallax in units of less than one pixel with lower accuracy than the first parallax detection unit. Have
When the image processing apparatus is mounted on a moving object,
A detection frequency changing unit that increases a frequency at which the first parallax detection unit detects a parallax of less than one pixel as the moving speed increases;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. From the object area detection means for detecting the imaging area of the object from the parallax image, obtain the imaging area,
When the parallax detection unit detects parallax in units of less than one pixel corresponding to the imaging region, the switching unit has the parallax detection unit having the highest parallax accuracy in units of less than one pixel, Selecting from a plurality of parallax detection means,
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. An image processing system for creating a parallax image including parallax information using two images,
Two imaging means;
A plurality of parallax detection means for detecting parallax in units of less than one pixel with different accuracy;
Switching to one of the plurality of parallax detection means to detect parallax in units of less than one pixel,
The image processing system, wherein the plurality of parallax detection units operate in parallel. An image processing method for creating a parallax image including parallax information using images respectively captured by two imaging means,
The switching means selecting one of a plurality of parallax detection means for detecting parallax in units of less than one pixel with different accuracy; and
Each of the plurality of parallax detection means operating in parallel to detect parallax in units of less than one pixel;
An image processing method comprising:

Images