JP2003150940A - Stereo image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To measure the distances of a plurality of objects different in distance with good accuracy by stereo image processing. SOLUTION: Only a pixel whose value falls within a designated range in the images of a plurality of cameras is selected (13), and a corresponding point is searched by the selected pixel (19), thereby preventing measurement of the distance of a background with a weak pattern instead of the distance of an object with a strong pattern by mistake. A plurality of pixel select ranges are provided (13, 14), the ranges are searched for a plurality of corresponding points using the selected pixels (19, 20), and according to the degree of bias of a pattern of a reference image (23), the final distance may be selected from the plurality of corresponding point search results (21). According to the degree of difference in pixel value between the target pixel in the reference image and a corresponding candidate pixel in a compared image, a pair of cameras causing occlusion are detected, and the paired cameras may not be used in corresponding point search. Concerning each target pixel, when the coincidence of pattern matching is not good, it is judged that there is the possibility of wrong detection of an object outside the measurement range.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a plurality of image information obtained by photographing the same object using a plurality of image pickup devices arranged at different positions, to a target object using the principle of triangulation. The present invention relates to a stereo image processing device and method for measuring the distance of the image.

[0002]

2. Description of the Related Art As a method of measuring the distance to an object using a plurality of images obtained by photographing an object in a three-dimensional space from two or more cameras arranged at different positions,
A method of stereo image processing is known. This stereo image processing is a method of estimating a distance to an object by performing a corresponding point search by pattern matching. The process of searching for corresponding points will be briefly described below.

In this specification, a specific one of the plurality of cameras is referred to as a "reference camera", and each of the other cameras is referred to as a "comparative camera". A pair of the standard camera and each comparative camera is called a "camera pair", and there are as many camera pairs as the number of comparative cameras. The image of the reference camera is hereinafter referred to as "reference image", and the image of each comparison camera is hereinafter referred to as "comparison image".

(1) From the reference image, one pixel as a target for distance measurement is selected. A certain distance is assumed as the distance to the object in the three-dimensional space corresponding to this target pixel. Then, the position of the pixel (hereinafter referred to as “corresponding candidate pixel”) in each comparison image corresponding to the point in the three-dimensional space separated by the assumed distance is obtained.

(2) A matching window centered on the position of the target pixel (hereinafter referred to as "reference window") is set in the reference image. A matching window centered on the position of the corresponding candidate pixel in each comparison image
(Hereinafter referred to as "comparison window") is set.

(3) For each camera pair, the pixel value of each pixel in the reference window on the standard image is compared with the pixel value of each corresponding pixel in the comparison window on each comparative image, and the pixel value between pixels is compared. "Difference" is calculated. As the “degree of difference”, the absolute value of the difference between pixel values, the square of the difference between pixel values, or the like is used.

(4) The "difference" is calculated for all the pixels in the reference window, and the "difference" is summed for all the pixels in the reference window (hereinafter, window addition).
Then, the “window-added difference degree” is obtained for each camera pair. Next, the above "window added difference degree" is calculated for all camera pairs, and the "window added difference degree" is summed up for all camera pairs (hereinafter, camera addition).
Then, the “final sum of dissimilarity” is obtained.

(4-2) Instead of the process of (4), the "difference" for each pixel in the reference window is added to the camera first, and then "the camera is added to each pixel". The "difference" may be added to the window to obtain the "final total difference". By the way, when the addition process of the dissimilarity is performed by using the hardware circuit, this method requires only one window addition circuit, so that the circuit structure becomes simpler than that of the above method (4). .

(5) As the distance to the object in the three-dimensional space corresponding to the target pixel, a plurality of other distances are further assumed, and the above-mentioned processing is performed for each of these distances to obtain the difference degree. For the final sum of. Then, one assumed distance at which the “final sum of dissimilarity” is minimized is estimated as the distance to the object corresponding to the target pixel.
Here, the fact that the “final sum of dissimilarities” is the minimum means that the image pattern in the reference window of the standard image and the image pattern in the comparison window of the comparative image are most similar. There is.

The above is a typical method of "corresponding point search" in conventional stereo image processing. By performing the corresponding point search as described above for all the pixels in the reference image,
A range image corresponding to the reference image is obtained.

[0011]

The stereo image processing method according to the above-mentioned conventional technique has the following three problems (1).
There is ~ (3).

(1) Swelling of object contour in range image In the above-mentioned corresponding point search, pattern matching is performed between the reference window and the comparison window. But,
If there are multiple objects with different distances in the reference window, or if there is an occlusion (occlus
Ion) (hidden) occurs, the degree of similarity between the image pattern in the reference window and the image pattern in the comparison window becomes poor. As a result, an incorrect distance may be output. In particular, for example, when there are a distant background and a near object, the background image has small characteristics (for example, the pattern is weak), and the object image has large characteristics (for example, the pattern is strong), The background distance may be erroneously estimated to be the same distance as the object due to the strong pattern of the object. As a result, in the calculated distance image, the contour of a nearby object swells more than the actual object.

This is a serious problem in an application in which a range of an object is automatically cut out using a range image.

The strength of the pattern, which is the main cause of the above-mentioned bulge, will be captured and explained. In this specification,
A “pattern” is defined as a feature of an image due to variations in pixel values within a certain area of the image. "Strength of pattern" means
It is the size of the feature of the image, that is, the size of the variation of the pixel value. Therefore, “strong pattern” means that the variation of pixel values is large, and “weak pattern” means that the variation of pixel values is small.

(2) False detection of an object outside the measurement range Corresponding point search assumes a plurality of distances within the range of the distance in which the object is likely to exist. Due to processing time limitations, there is also a limitation on the number of assumed distances. The range from the minimum value to the maximum value of these assumed distances is called "measurement range".
The distance to an object existing outside the measurement range cannot be measured. However, according to the above-described conventional technique, the assumed distance that minimizes the final sum of the dissimilarities is output as it is as the measured value of the distance, so that an incorrect distance is measured for an object outside the measurement range. In particular, for example, when measuring the distance of a nearby object with high accuracy using a camera having a convergence angle, the measurement range is set to be extremely narrow, so it is important to prevent erroneous detection of an object outside the measurement range. It is an issue.

(3) Occlusion (Hidden) “Occlusion” means that an object is visible to a reference camera but not to some comparison cameras (for example, that object is different from another object). Hiding behind). When occlusion occurs, an incorrect distance may be output because the similarity of the image pattern between the reference window and the comparison window becomes poor. As a remedy for this problem of occlusion, a method of detecting a camera in which an occlusion has occurred and excluding the image of that camera when performing corresponding point search has been proposed (Matsuura and Sato of the University of Tsukuba). , Nakamura and Ota's paper "Multi-view stereo method with hidden detection" IEICE Transaction D-II
Vol.J80-D-II NO.6 pp.1432-1440 June 1997). In this method, the window-added dissimilarity is obtained for each camera pair, and if the window-added dissimilarity of a camera pair is larger than a predetermined value, the occlu-
John is considered to have occurred, and the “window-added difference degree” of the camera pair is excluded from the camera addition processing.

However, in order to carry out this method, it is necessary to calculate the "window added difference degree" for each of all camera pairs. That is, in the procedure of searching the corresponding points described above, the addition method of (4-2) cannot be adopted and the method of (4) must be adopted. As a result, the hardware circuit for performing the addition process requires the same number of window addition circuits as the camera pairs, which complicates the structure.

In view of the above problems, an object of the present invention is to enable the distance between a plurality of objects having different distances to be accurately measured by stereo image processing.

Another object of the present invention is to eliminate the problem of bulging of the object contour in the range image.

Another object of the present invention is to prevent erroneous detection of an object located outside the measurement distance.

Still another object of the present invention is to solve the problem of occlusion with a circuit having a simple structure.

[0022]

[Means for Solving the Problems] In the description in this column,
The numbers in parentheses exemplify the correspondence with the elements described in the accompanying drawings, but this is merely an example for explanation and is not intended to limit the technical scope of the present invention. .

According to one aspect of the present invention, there is provided a stereo image processing apparatus for performing corresponding point search using a reference image (1) from one reference camera and a comparison image (2) from at least one comparison camera. A pixel selection unit (3) for excluding, from at least one of the reference image and the comparison image, pixels having pixel values substantially outside a predetermined pixel value range and selecting the remaining pixels; For each target pixel, for each of the various assumed distances, pattern matching between the reference image and the comparison image is performed using the pixels selected by the pixel selection unit, whereby each target pixel And a corresponding point search unit (6) for measuring the distance.

The corresponding point searching unit (6) can have means for calculating the reliability of the measured distance for each target pixel based on the result of the pattern matching at various assumed distances.

According to another aspect of the present invention, the stereo image processing apparatus has a plurality of pixel selection units (13, 14) having different pixel value ranges, and the pixels selected by the plurality of pixel selection units. Using a plurality of corresponding point search units (19,
20), and for each target pixel, the final distance is determined and output based on the plurality of distances measured by the plurality of corresponding point search units and the reliability calculated for the plurality of distances. And a measurement distance determining unit (21) for performing the measurement.

According to still another aspect of the present invention, the stereo image processing apparatus further includes a pixel value determination unit (23) for determining a range of pixel values included in the reference image, and the measured distance determination unit ( 21) determines the final distance from the plurality of distances, the reliability calculated for the plurality of distances, and the pixel value range from the pixel value determination unit (23).

According to yet another aspect of the invention, a stereo image processing device for performing corresponding point search using a reference image (31) from one reference camera and a comparison image (32) from at least one comparison camera. Is a first corresponding point search unit for measuring the distance of each target pixel by performing pattern matching between the reference image and the comparison image at various assumed distances for each target pixel in the reference image. (35), a pattern strength calculation unit (36) that outputs the pattern strength of the reference image, and the pattern strength calculation unit (36)
A distance determination determination unit (37) that determines whether or not to determine the distance of each target pixel measured by the first corresponding point search unit (35) based on the pattern intensity output from Pixels of which the distance is determined by the distance determination determining unit (37) are excluded from the reference image, and pixels of the comparison image corresponding to the pixels excluded from the reference image are excluded from the reference image. And a pixel selection unit (38) for selecting the remaining pixels from the reference image and the comparison image, and various assumed distances for each target pixel in the reference image for which at least the distance is undetermined. Pattern matching between a reference image and the comparison image is performed using the pixels selected by the pixel selection unit, whereby the distance of each target pixel whose distance has not been determined yet. And a second corresponding point searching unit (41) for measuring

In accordance with yet another aspect of the present invention, stereo image processing for performing corresponding point search using a reference image (51) from one reference camera and a comparison image (52) from at least one comparison camera. The apparatus sets a reference window for each target pixel in the base image, sets a comparison window in the comparison image corresponding to the reference window at each of various assumed distances, and sets the pixel value of the pixel in the reference window. A pixel selection unit (55) for setting a pixel value range based on the above, excluding pixels having pixel values substantially outside the set pixel value range from the comparison window, and selecting the remaining pixels. Pattern matching between the reference window and the comparison window for each of the various assumed distances for each target pixel in the reference image, A corresponding point searching unit (56) for measuring the distance of each target pixel by using the selected pixel in the comparison window and the pixel in the reference window corresponding to the selected pixel. Equipped with.

A stereo image processing apparatus for performing corresponding point search using a reference image (71) from one reference camera and a comparison image (72) from at least one comparison camera according to another aspect of the present invention. A first pixel selection unit (73) that excludes, from at least one of the reference image and the comparison image, pixels that substantially have a pixel value outside the first pixel value range, and selects the remaining pixels; For each target pixel in the reference image, for each of various assumed distances, pattern matching between the reference image and the comparison image, using the pixels selected by the first pixel selection unit. The first corresponding point searching unit (79) for measuring the distance of each target pixel, and the second narrower than the first pixel value range from at least one of the reference image and the comparison image. Pixel value A second pixel selection unit (74) that excludes pixels that substantially have pixel values outside the range and selects the remaining pixels, and various assumed distances for each target pixel in the reference image. The pattern matching between the reference image and the comparison image is performed using the pixels selected by the second pixel selection unit, thereby measuring the distance of each target pixel. A point search unit (80), a pattern bias calculation unit (83) that outputs the pattern bias degree of the reference image, an output from the pattern bias calculation unit (83), and the first corresponding point search unit (79). ) And a measured distance determination unit (81) that determines a final distance based on the distance measured by the second corresponding point search unit (80).

In accordance with yet another aspect of the present invention, a stereo image processing apparatus for performing corresponding point search using a reference image (91) from one reference camera and a comparison image (92) from at least one comparison camera. For each target pixel in the reference image, for each of the various assumed distances, performs pattern matching between the reference image and the comparison image, thereby corresponding points for measuring the distance of each target pixel. A search unit (96) and a reliability output unit (98) that outputs a reliability for the measured distance are provided. Then, according to the result value of the pattern matching for the measured distance (S2), or, in the result value of the pattern matching for the result value of the pattern matching for the measured distance and other assumed distances. The reliability of the measured distance changes according to the difference between the worst value (S3) and the value (S5, S4).

According to still another aspect of the present invention, a stereo image processing apparatus that performs a corresponding point search using a reference image from one reference camera and a plurality of comparison images from a plurality of comparison cameras has various assumptions. For each of the distances, occlusion may occur among the plurality of comparison cameras based on the degree of difference between the target pixel in the reference image and the corresponding candidate pixel in each comparison image. Means for selecting and ignoring some of the comparison cameras (S11-S14), and for each of the various hypothetical distances, for each pixel in the reference window set in the reference image, the remainder excluding the ignored comparison cameras. Of each of the comparison images from the comparison cameras of the respective comparison cameras, the degree of difference with the corresponding pixel in the comparison window set for each of the comparison images is obtained, and the difference obtained for each pixel is calculated for all the remaining comparison cameras. La addition means (S15), means for adding the camera-added dissimilarity to all pixels in the reference window by window addition, and means (S16, S17) for obtaining the final dissimilarity, and various assumed distances. Means for measuring the distance of the target pixel based on the final sum of the dissimilarity degrees obtained for each.

[0032]

BEST MODE FOR CARRYING OUT THE INVENTION Various embodiments of the present invention will be described below.

1. Embodiments Addressing the Problem of Swelling of the Object Contour First, some embodiments will be described that address the problem of the object contour swelling in the range image. In these embodiments, the following two principles are applied.

(1) First Principle The problem that the contour of the object swells easily occurs under the following conditions a) to d).

A) two or more objects having different distances, for example, a near object and a distant background exist in a three-dimensional space; b) the reference window in the standard image shows the background, and the background is the distance. It is the target of the measurement; c) The comparison window in the comparison image of one or more comparison cameras shows a nearby object, but due to the occlusion of that object, the background of the target is not. And d) the dissimilarity value between the target background and the object is much greater than the dissimilarity value between the target background and the background at a different location.

In the corresponding point search under the above conditions a) to d), the minimum peak of the final sum of the dissimilarities appears when the assumed distance is equal to the actual distance (correct answer) of the target background. When the distance is equal to the actual distance of a nearby object. However, the latter minimum peak value may be smaller than the former minimum peak value (that is, the latter seems to be the correct answer). In this case, an erroneous value equal to the distance of a nearby object is output as the measured distance of the background of the target.

The above problems can be reduced by the following methods A) to B).

A) From the images of each camera, only the pixels having the pixel values included in the predetermined pixel value range are selected. Alternatively, from the image of each camera, only the pixel whose pixel value in the neighborhood area of the pixel is included in the predetermined pixel value range is selected.

B) Then, the corresponding points are searched using only the selected pixels.

(2) Second Principle The problem that the contour of the object swells is also caused by the following conditions e) to h).
It easily occurs even under.

E) two or more objects with different distances, eg near object and distant background, exist in three-dimensional space; f) both the background and the object are reflected in the reference window in the standard image, and The background is the target for distance measurement; g) the variation in the pixel value of the target background is small; and h) the difference in pixel value between the target background and nearby objects is large.

Also in the corresponding point search under the above conditions e) to h), the minimum peak of the final sum of the dissimilarities may appear in a similar form under the conditions a) to d) already explained. is there.
Even in that case, an erroneous value equal to the distance of the nearby object is output as the measured distance of the target background.

The above problems can be reduced by the following methods C) to G).

C) Set a plurality of pixel value ranges.

D) A pixel is selected from the image of each camera in the manner described in A) above, using each of the plurality of pixel value ranges. Thereby, for each camera, a plurality of images respectively composed of the selected pixels corresponding to the plurality of pixel value ranges are obtained.

E) Corresponding point search is performed using images from a plurality of cameras corresponding to the same pixel value range. Corresponding point searches are performed a plurality of times for a plurality of pixel value ranges.

F) The degree of deviation of the pattern (pixel value variation) in the reference window of the standard image is detected.

G) Based on the detected degree of pattern deviation, the distance to be finally output is determined from the results of the above-described corresponding point search.

[1] First Embodiment FIG. 1 shows the configuration of this embodiment. This embodiment is
By applying the above-mentioned principle (1), the problem of bulging of the contour of an object is improved.

The problem that the object contour is bulging is typically caused by a weak background pattern (small variation in pixel values).
This may occur when the pattern of the object is strong (the variation in pixel values is large). Alternatively, even if the background and the pattern of the object are both weak, this problem may occur when there is a large difference in the pixel values of each other.

FIG. 1 shows the configuration of the main part of this embodiment.

Although not shown in FIG. 1, as in the conventional case, one reference camera and one or more comparison cameras are arranged at different positions according to the principle of stereo image processing, and the same three-dimensional image is obtained from each position. Take a picture of the space. The number of comparison cameras may be one, but in order to improve the measurement accuracy,
It is desirable to have more than one. For example, nine cameras are arranged in a 3 × 3 plane matrix, the central camera is the reference camera, and the eight cameras around it are comparison cameras. These nine cameras shoot. In this embodiment, it is assumed that there are a plurality of comparison cameras. Such a camera arrangement is commonly adopted in other embodiments described later.

As shown in FIG. 1, the reference image input unit 1 is
A reference image is input from the reference camera and the reference image is output to the pixel selection unit 3. The comparison image input unit 2 inputs each comparison image from a plurality of comparison cameras and outputs the comparison images to the pixel selection unit 3.

The pixel selection section 3 stores one or more preset pixel value ranges. The setting of the pixel value range can be changed by the user. The pixel selection unit 3 performs pixel selection processing on the input image using the stored pixel value range. In the pixel selection process, the mask information is added (that is, excluded) only to the pixels having the pixel value outside the applicable pixel value range among the pixels in the image, and the pixels within the pixel value range are included. Mask information is not added (that is, selected) to pixels having a value.

The above-mentioned pixel selection processing can be performed on all of the reference image and the plurality of comparison images, can be performed only on the reference image, or can be performed only on the comparison image. You can also do it. The same pixel value range can be applied to all images, or the pixel value range applied to each image can be changed in consideration of variations in characteristics between images. You can also

The pixel value range applied to one image may be one continuous range or may be divided into two or more sub-ranges. For example, when the pixel value of the image is luminance, a pixel selection process is performed by using one appropriate continuous luminance range to exclude an area having too small or too large luminance and an area having appropriate luminance. You can only choose. Further, for example, when the pixel value of the image is a brightness differential value that represents the spatial change rate of brightness (pattern intensity), two appropriate brightness differential value sub-values set on the plus side and the minus side, respectively. By performing pixel selection processing using a range (meaning an area where the pattern strength is appropriate), the absolute value of the brightness differential value is too small or too large (area where the pattern is too weak or too strong) is excluded. , It is possible to select only the region where the pattern strength is appropriate.

The decision as to whether or not to select a pixel may be made not only based on the pixel value of the pixel itself, but also based on the image value of the area near the pixel including the pixel. For example, if you set a window of an appropriate size centered on that pixel, and the sum of all pixel values in that window, or the average value of the pixel values in that window is outside the set pixel value range. If so, that pixel can be excluded. Alternatively, if all the pixel values in the window are within the set pixel value range, the pixel at the center of the window is selected, but if not, the pixel at the center is excluded. Good. Regardless of which selection method is used, in this specification, a pixel that is to be excluded (mask information is attached) by the set pixel value range is referred to as “a pixel value outside the set pixel value range. It is referred to as a “pixel that substantially has”.

The reference image output from the image selection unit 3 is stored in the reference image storage unit 4. The comparison image output from the image selection unit 3 is stored in the comparison image storage unit 5. The corresponding point search unit 6 reads the reference image in the reference image storage unit 4 and the comparison image in the comparison image storage unit 5, and performs the corresponding point search as follows.

In the corresponding point search, one target pixel is selected from the standard image, and the reference window is set with the target pixel as the center. Next, assume the distance to the object corresponding to the target pixel. Then, in each comparison image, a corresponding candidate pixel corresponding to a point in the three-dimensional space corresponding to the assumed distance is obtained, and a comparison window centered on the corresponding candidate pixel is set. Next, the pixels without mask information (that is, selected) in the reference window and the pixels without mask information (that is, selected) in each comparison window that positionally corresponds to the pixel Find the difference between The difference is
For example, the absolute value of the difference in pixel value between corresponding pixels can be used. Next, perform window addition of these dissimilarities and camera addition (whichever may be done first) (camera addition is not necessary when there is one camera pair) to obtain the final sum of dissimilarities. Ask.

In this way, the degree of difference is calculated for both the reference image and the comparison image using only the pixels selected by the predetermined pixel value range. Therefore, when the pixel of the reference image and at least one of the corresponding pixels in each comparison image are not selected, the degree of difference between the pixels of the pair is not calculated. But instead of this method,
The following method may be adopted. That is, when the pixel in the reference image is selected but the corresponding pixel in the comparison image is not selected, the pixel value of the pixel in the comparison image is set to the middle of the pixel value range used in the pixel selection. Replace with the value. For example, when the pixel value range is 0 to 100 and the pixel value of the pixel in the comparison image is 150,
Replace the pixel value with 50. After that, the dissimilarity between the pixels of the pair is calculated and also included in the final dissimilarity sum.

The corresponding point searching unit 6 variously changes the assumed distance within the measurement range, performs the above process for each assumed distance, and obtains the final difference. Then, the corresponding point searching unit 6 selects an assumed distance that minimizes the final degree of difference,
It is output to the distance image output unit 7 as the measured distance for the target pixel.

The corresponding point searching unit 6 sequentially selects all the pixels in the reference image as target pixels and performs the above corresponding point search. As a result, the measured distance is output for all pixels. The distance image output unit 7 outputs the distance image thus completed.

In the present embodiment, if the pixel value range used for pixel selection is narrow, the number of pixels to be excluded is large, and sufficient pattern matching cannot be performed when searching for corresponding points. It may output the wrong measurement distance. Therefore, the distance image output unit 7 may obtain the reliability indicating the reliability of the corresponding point search as described below, and output only the measured distance of the pixel with high reliability.

That is, for example, as shown in FIG. 2, it is assumed that the n-th assumed distance Z (n) among a plurality of assumed distances is output as the measured distance. In that case, SSAD (n) is the final sum of the dissimilarities for the measured distance Z (n). The assumed distances Z (n-1) and Z (n + 1) before and after the measured distance Z (n)
The final sum of dissimilarities for SSAD (n-1) and SSAD (n +
1) And the absolute value of the difference between SSAD (n-1) and SSAD (n),
And the sum of the absolute values of the differences between SSAD (n + 1) and SSAD (n) is taken as the reliability RE. That is, RE = | SSAD (n-1) -SSAD (n) | + | SSAD (n + 1) -SSAD (n) |. It can be considered that the larger the value of RE, the higher the reliability of pattern matching, and therefore the measured distance
It can be considered that Z (n) has high reliability.

Alternatively, as another method, for example, as shown in FIG. 3, the n-th assumed distance Z (n) among a plurality of assumed distances is output as the measured distance, and the m-th assumed distance Z (m )
Suppose that the second smallest minimum peak appeared in the final total sum of dissimilarities. In that case, the final sum of the dissimilarity at the measured distance Z (n), SSDA (n), and the value of the second minimum peak, SSDA
Let the absolute value of the difference of (m) be the measurement distance Z (n) reliability RE. That is, RE = | SSAD (n) −SSAD (m) |. It can be considered that the larger the value of RE, the higher the reliability of pattern matching, and therefore the measured distance
It can be considered that Z (n) has high reliability.

As another method, the flatness of the range image can be obtained and used as the reliability.

That is, when the reliability of the pattern matching is low and an incorrect range image is output, for example,
The distance image may not be planar even though a planar object is being measured, and the distance between adjacent pixels may be significantly different. Therefore, Sobel processing (first-order differentiation processing) is performed on the distance image, and the result is used as the flatness of the distance image. Pixels with low flatness (large distance change) have reliability RE.
Can be low. Alternatively, it is possible to set a window of a predetermined size around each pixel in the distance image and use the reciprocal of the difference between the maximum value and the minimum value of the distance in the window as the flatness of the pixel. Furthermore, when the measured distance of only one pixel is significantly different from the measured distances of pixels in the vicinity of the periphery, the reliability RE of the pixel can be output low. Further, even if the flatness of a certain region is high, if the region is very narrow, the reliability RE of that region may be lowered.

According to the present embodiment, pixels having substantially pixel values outside the predetermined pixel value range are excluded from the reference image or comparison image, and the corresponding points are searched using the remaining pixels. This reduces variations in pixel values of pixels used in the corresponding point search. That is, only the weak pattern portions in the reference image and the comparison image are used in the corresponding point search. Therefore, the frequency of the condition "a background pattern in the reference window is weak and the object pattern is strong" that causes the bulge of the object contour to decrease is reduced, and as a result, the background distance is erroneously determined to be close to the object distance. It becomes difficult to measure. In addition, when determining whether to select a pixel using the image value in the vicinity of each pixel, even if a pixel value detection error occurs for only one pixel, the error Since the effect of can be removed, stable results can be obtained.

[2] Second Embodiment FIG. 4 shows the configuration of this embodiment. This embodiment is
It applies the above-mentioned principle (1).

In this embodiment, a plurality of pixel selection units 13,
14 is provided. Two pixel selection units 13 and 14 are illustrated, but more pixel selection units may be provided.
Different pixel value ranges are set in the pixel selection units 13 and 14, respectively. The pixel value ranges may partially overlap each other. Each pixel selection unit 13, 14
Performs pixel selection processing similar to that described in the first embodiment on the input image using the respective pixel value ranges. The reference image and the comparison image output from the first pixel selection unit 13 are stored in the reference image storage unit 15 and the comparison image storage unit 16. The reference image and the comparison image output from the second pixel selection unit 14 are stored in the reference image storage unit 17 and the comparison image storage unit 18.

A plurality of corresponding point searching units 19 and 20 are provided corresponding to the plurality of pixel selecting units 13 and 14, respectively. The corresponding point searching units 19 and 20 use the sets of the reference image and the comparison image output from the corresponding pixel selecting units 13 and 14, respectively, in the same manner as described in the first embodiment. The search is performed, and the measurement distance for each pixel obtained as a result is output to the measurement distance selection unit 21. Each corresponding point search unit 1
Also, 9 and 20 calculate the reliability of the measured distance for each pixel by the same method as described in the first embodiment, and also output the reliability to the measured distance selection unit 21.

The measurement distance selection unit 21 compares the reliability of the measurement distance for each pixel, which is input from the corresponding point search units 19 and 20, for the same pixels, and determines the measurement distance with the highest reliability. It is selected as the final measurement distance of the pixel and output to the distance image output unit 22.

According to this embodiment, the result with the highest reliability is selected for each pixel from the results of the corresponding point search performed using the different pixel value ranges, and the final range image is selected. Can be generated, the proportion of highly reliable pixels included in the output range image is further increased.

[3] Third Embodiment FIG. 5 shows the configuration of this embodiment. This embodiment is
It applies the above-mentioned principle (1).

This embodiment includes a pixel value range determination unit 23 in addition to the configuration of the second embodiment described above. The pixel value range determination unit 23 performs the following processing.

The pixel value range determination unit 23 inputs the reference image from the reference image input unit 11, and the pixel value of each pixel in the reference image is set to the plurality of pixel selection units 13 and 14. Which of the ranges is included is determined, and the result is output to the measured distance selection unit 21. Note that this determination is made, for example, on a representative value (for example, an average value) of pixel values in a window of a predetermined size centered on each pixel, instead of making a determination on the pixel value of each pixel itself. May be.

The plurality of pixel value ranges may partially overlap each other, in which case the result of the pixel value range determination for one pixel may indicate a plurality of ranges.
Further, the result of the pixel value range determination may not point to any range.

The measured distance selection unit 21 inputs a plurality of distance images and reliability of the measured distance for each pixel in each distance image from the corresponding point search units 19 and 20, and further from the pixel value range determination unit 23. The judgment result for each pixel is also input. Then, the measurement distance selection unit 21 selects which of a plurality of measurement distances is to be adopted for each pixel in consideration of not only the reliability but also the determination result, and the selected measurement distance is used as a distance image of the pixel. Output as. As a method of considering the judgment result,
For example, for the same pixel, if the values of reliability of a plurality of measurement distances are nearly equal to each other, or if the reliability of the plurality of measurement distances is all fairly low, the pixel indicated by the determination result in the plurality of measurement distances There is a method such as selecting the measurement distance in the value range.

According to this embodiment, by utilizing the above determination result, the ratio of highly reliable pixels included in the obtained range image is further increased.

[4] Fourth Embodiment FIG. 6 shows the configuration of this embodiment. This embodiment is
It applies the above-mentioned principle (1).

In this embodiment, the pattern strength calculation unit 36
Calculates the pattern intensity for each pixel of the reference image. The corresponding point searching unit 35 performs the first corresponding point search in the same manner as the conventional method, and outputs the measured distance for each pixel of the reference image. The corresponding point searching unit 35 also outputs the reliability of the measured distance for each pixel by the method described in the first embodiment. Then, the distance determination unit 37 calculates only the measured distances of the pixels determined to have high reliability and strong pattern strength from the measured distances for each pixel from the corresponding point search unit 35.
The distance is determined as the pixel distance, and the determined distance is output to the distance image output unit 42.

Furthermore, the corresponding point searching unit 41 performs the second corresponding point search without using the pixel value of the pixel whose distance is fixed in the above processing, and the measured distance is calculated for the pixel whose distance is not fixed yet. Obtained and output it to the distance image output unit 42.

The sizes of the matching windows (reference window and comparison window) used in the first and second corresponding point searches may be the same or different. In particular, the window size of the second time can be made larger than that of the first time.

In the pattern strength calculation unit 36, an image obtained by differentiating a luminance image by a method such as Sobel or Log can be used as the pattern strength. When the input from the reference image input unit 31 is a luminance image, the pattern strength can be obtained by differentiating the luminance image as described above. On the other hand, when the input from the reference image input unit 31 is the image obtained by differentiating the luminance image, the input pixel value can be used as it is as the pattern intensity. However,
In the case of an image, such as a Log image, in which the pixel value is zero when there is no pattern and the pixel value increases in the plus or minus direction as the pattern becomes stronger, the absolute value of the pixel value is used as the pattern strength. Alternatively, a window of a predetermined size centered on each pixel may be set, and the result or average value of the pattern intensities of all pixels in the window may be used as the pattern intensity of the pixel.

For each pixel in the reference image, the distance determining unit 37 has the reliability of the measured distance obtained by the first corresponding point search of a predetermined threshold value or more, and the pattern strength of a predetermined threshold value or more. Only in that case, the measured distance is fixed as the distance of the pixel, and the fixed distance and the coordinates of the pixel are set to the distance image output unit 42 and the pixel selection unit 38.
Output to.

The pixel selection unit 38 inputs the reference image from the reference image storage unit 33 and adds mask information to the pixels in the reference image having the same coordinates as the coordinates input from the distance fixing unit 37. Further, the pixel selection unit 38 inputs each comparison image from the comparison image storage unit 34, obtains the pixel in each comparison image corresponding to the coordinates and the determined distance notified from the distance determination unit 37,
Mask information is added to the pixel. The mask information is
It may be added only to the corresponding pixel, or a window of a predetermined size centered on that pixel may be set and added to all pixels in that window.

The reference image to which the mask information is added is stored in the reference image storage unit 39, and each comparison image to which the mask information is added is stored in the comparison image storage unit 40.

The corresponding point searching unit 41 includes a reference image storage unit 39.
And the second corresponding point search is performed using the image stored in the comparison image storage unit 40. At this time, the corresponding point searching unit 41, as described in the first embodiment, when the mask information is added to at least one of the pair of pixels that are positionally corresponding to the reference image and each comparison image, The dissimilarity between the pixels of the pair is ignored. Alternatively, the corresponding point searching unit 41, as described in the first embodiment, when the mask information is added only to the pixel of the comparative image among the pixels of the pair, the pixel of the comparative image is added. After replacing the value of with a value near a middle value of a predetermined pixel value range (for example, a distribution range of all pixel values of the reference image or each comparison image),
The dissimilarity between the pixels in the pair is calculated and used in the final dissimilarity sum calculation.

The distance image output unit 42 outputs the determined distances for some pixels in the reference image input from the distance determination unit 37 and the measured distances for the remaining pixels input from the corresponding point search unit 41. Integrate to generate and output the final range image.

The sizes of the matching windows (reference window and comparison window) used in the first and second corresponding point searches may be the same or different. In particular, in the second corresponding point search, the reliability of the corresponding point search is lower than that in the first corresponding point search because the image having a weaker pattern is used. To compensate for this low reliability, the second matching window size can be made larger than the first matching window size.

According to this embodiment, first, in the first corresponding point search, the distance is obtained with respect to the pixel having a strong pattern (where the problem of the bulging of the object contour is not likely to occur originally), and then the distance is calculated. By masking the pixels with a strong pattern and then finding the distances with respect to the pixels with a weak pattern in the second corresponding point search, the adverse effect from the pixels with a strong pattern is removed, and the bulging of the object contour is less likely to occur.

[5] Fifth Embodiment FIG. 7 shows the configuration of this embodiment. This embodiment is
It applies the above-mentioned principle (1).

In this embodiment, the pixel selection unit 55 and the corresponding point search unit 56 perform the following processing for each pixel in the reference image. The following processing is repeatedly executed for all the pixels of the reference image.

In the pixel selection section 55, the pixel value range is set in advance. When the corresponding point search is performed for a certain target pixel in the reference image, the pixel selection unit 55 adds mask information to all pixels having a pixel value outside the predetermined pixel value range.

Next, the pixel selection section 55 displays the reference window.
Among the pixel values of the pixels in which the mask information is not added
Maximum value G at_1maxAnd the minimum G_{1 min}Ask for. Alternatively, the picture
The element selection unit 55 calculates the pixel values of all the pixels in the reference window.
And the maximum value G_1maxAnd the minimum G _{1 min}May be asked.

Further, the pixel selection unit 55 selects the maximum value G _1max among the pixels in each comparison window in each comparison image.
And mask information is added to all the pixels having a pixel value that is not between the minimum value G _1min and the minimum value G _1min .

The corresponding point search unit 56 uses the reference window and each comparison window output from the pixel selection unit 55 to perform a corresponding point search by the same method as described in the first embodiment. That is, in the corresponding point search, when the mask information is added to at least one of the pair of pixels that correspond positionally in the reference window and each comparison window, the degree of difference between the pixels of the pair is ignored. Alternatively, when the mask information is added only to the pixel of the comparison image of the pixels of the pair, the value of the pixel of the comparison image is replaced with a value near the intermediate value of the pixel value range. Then, the dissimilarity between the pixels in the pair is calculated and used in the calculation of the final sum of the dissimilarities.

According to this embodiment, when performing the corresponding point search, the pixel value range (the maximum value G _1max and the minimum value G _1min ) is automatically set for each reference window of the standard image, and the pixel value range is set. By selecting pixels in each comparison window by using, a plurality of corresponding point searching units can be used as in the second embodiment shown in FIG. 4 by using only one corresponding point searching unit 56. A range image is obtained with almost the same reliability as when there was.

Furthermore, in the second embodiment shown in FIG. 4, it is necessary to preset the pixel value range considered to be optimum. Further, in the second embodiment, when the pattern of the image changes greatly when continuously performing the corresponding point search,
It is necessary to increase the number of pixel value ranges to be set.
On the other hand, in this embodiment, the optimum pixel value range is automatically set in each corresponding point search.

[6] Sixth Embodiment FIG. 8 shows the configuration of this embodiment. This embodiment is
The principle (2) described above is applied.

According to the above-mentioned principle (2), the condition that the swelling of the object contour occurs is that "the target pixel is on a background far away in the reference window, the variation of the pixel value of the background is small, and the background is small. There is a large difference in the pixel values of the nearby objects. " This condition is satisfied, for example, when the pattern of the background is weak and the pattern of the object is strong, or when the patterns of the background and the object are both weak but there is a large difference in pixel value between them.

In this embodiment, the pattern deviation calculator 83 for obtaining the degree of pattern deviation in the reference window in the standard image.
Equipped with. If the pattern in the reference window is biased,
The swelling of the contour of the object may occur due to the satisfaction of the above conditions.

A pixel value range is set in each of the two pixel selection units 73 and 74. Here, the second pixel selection unit 7
The width of the pixel value range set to 4 is determined by the first pixel selection unit 7
Narrower than that of 3. Therefore, the second pixel selection unit 7
The pixels selected in 4 are narrowed down to only images with a weaker pattern than the pixels selected in the first pixel selection unit 73. Note that the number of pixel selection units is not limited to two, and three or more pixel selection units each having a different range of pixel value may be provided.

By performing the corresponding point search by using only the pixels having the weak pattern selected by the second pixel selecting section 74, the number of pixels for which the erroneous distance is measured can be reduced.

Therefore, the measurement distance selection unit 81 inputs two measurement distances for each pixel in the reference window from the two corresponding point search units 79 and 80, and the pattern bias calculation unit 83 inputs the measurement distance in the reference window. The degree of pattern bias for each pixel is input, and for pixels determined to have a pattern bias, a measurement distance from the second corresponding point search unit 80 that uses only weak pixels of the pattern is selected, and otherwise. For pixels determined to have no pattern bias, the measurement distance from the first corresponding point searching unit 79 is selected, and the selected measurement distance is output to the distance image output unit 82.

Alternatively, the following processing may be performed instead of the above processing. That is, in the pattern deviation calculation unit 83, the direction of the pattern deviation in each pixel (the direction in which the pattern changes from the weak one to the strong one. In the pattern deviation calculation example described later, the i component of the degree of pattern deviation, j The direction of the vector when the component is a vector) is calculated. In the measurement distance selection unit 81, the first
The direction of distance change at each pixel in the distance image from the corresponding point searching unit 79 (direction in which the distance at each pixel changes from distant to near) is obtained. Then, in the pixel determined to have the pattern bias, if the pattern bias direction and the distance change direction match (or if the difference between the two directions is smaller than a predetermined value), the second corresponding point search is performed. The measurement distance from the unit 80 is selected and output to the distance image output unit 82. In other cases, the measured distance from the first corresponding point searching unit 79 is selected and output to the distance image output unit 82.

In the latter process, if the pattern bias direction and the distance change direction match, or if the difference between the two directions is smaller than a predetermined value and is small, the intensity of the pattern is different and the distance is also different in the reference window. It means that there is a high probability that there are two or more objects. In such a case, bulging of the contour of the object may occur, so the second
The measurement distance from the corresponding point search unit 80 is selected. In cases other than the above case, both pattern strength and distance are different.
Since it is unlikely that there is one or more objects, and therefore the possibility that bulging of the object contour will occur, the measurement distance from the first corresponding point searching unit 79 can be selected.

The pattern bias calculation unit 83 obtains the pattern bias degree in the following manner, for example.

When the reference image input from the reference image input unit 71 is a luminance image, the pattern bias calculation unit 83 applies a differentiating process to the luminance image, for example, to obtain each image like a Sobel image or a Log image. It is converted into a pattern intensity image in which the pixel value of the pixel represents the pattern intensity at that pixel. Next, the reference image input unit 71 uses the pattern intensity image shown in FIG.
As shown in, the pixel P (i, j) of the coordinate (i, j) has a predetermined size (2A + 1) pixels × (2B + 1) pixels (A = 2, B = in the example of FIG. 9). (2 pixels of 5 pixels x 5 pixels) is set, the barycentric position of the pixel value in this window is determined, and the difference between the coordinates of the barycentric position and the coordinates (i, j) of the pixel P (i, j) And the coordinate difference is defined as the degree of pattern deviation at the pixel P (i, j).

On the other hand, when the image input from the reference image input unit 71 is an image obtained by differentiating a luminance image such as a Sobel image or a Log image, the pattern bias calculation unit 83 uses the input reference image as it is. The barycentric position is obtained by the method described above using the pattern strength image, and the coordinate difference from the barycentric position is taken as the pattern deviation degree at the pixel P (i, j).

More specifically, the pattern deviation calculating section 83 obtains the i component and the j component of the pattern deviation degree as follows.

First, in the window shown in FIG. 9 set in the pattern intensity image, i of the coordinate (k, l) of each pixel P (k, l) is calculated.
Component and coordinates (i, j) of the center pixel P (i, j) of the window
The difference m from the i component of the pixel is calculated, and the product of the difference m and the pixel value V (k, l) of the pixel P (k, l) is calculated. The above product is obtained in the same manner for all the pixels in the window. Then, the products of all the pixels in the window are added, and the sum is divided by the total number of pixels in the window. The quotient of this division is taken as the i component of the degree of pattern deviation at the central pixel P (i, j). That is,

[0113]

[Equation 1] Is.

Further, within the window shown in FIG. 9 set in the pattern intensity image, j of the coordinates (k, l) of each pixel P (k, l) is j.
Component and coordinates (i, j) of the center pixel P (i, j) of the window
The difference n from the j component of the pixel is calculated, and the product of the difference n and the pixel value V (k, l) of the pixel P (k, l) is calculated. The above product is obtained in the same manner for all the pixels in the window. Then, the products are added to all the pixels in the window, and the sum is divided by the total number of pixels in the window. The quotient of this division is taken to be j, which is the degree of pattern deviation at the central pixel P (i, j).
As an ingredient. That is,

[0115]

[Equation 2] Is.

The degree of deviation of the pattern in each pixel obtained as described above is small when the deviation of the pattern in the neighboring area centering on the pixel is small, and becomes large as the deviation of the pattern is large.

The pattern bias calculator 83 calculates at least either the absolute value of the i component or the absolute value of the j component of the degree of pattern bias, or the value obtained by squaring the i component and the j component.
When the sum of the multiplied values is equal to or larger than the predetermined value, it is determined that the pixel has a pattern bias.

According to the present embodiment, when the reference window has a biased pattern, the pattern in the reference window and each comparison window is weak (the variation in pixel value is small).
By performing the corresponding point search using only pixels, the number of pixels outputting an incorrect distance can be reduced.

2. Embodiment [7] Seventh Embodiment Addressing Problem of False Detection of Object Outside Measurement Range FIG. 10 shows the configuration of this embodiment.

This embodiment comprises a distance image output unit 97 for outputting a distance image, and a reliability output unit 98 for outputting the reliability of the measured distance of each pixel in the distance image. In addition, a pattern intensity calculation unit 93 is provided, in which the pattern intensity at each pixel in the reference image input from the reference image input unit 91 (for example, a large variation in pixel value in a neighboring region centered on the pixel). Is calculated. The calculated pattern intensity is input to the corresponding point searching unit 96.

The corresponding point searching unit 96 performs corresponding point searching for each target pixel in the reference image by the method already described, and the resulting measured distance is calculated by the distance image output unit 9
Output to 7. Further, the corresponding point searching unit 96 obtains the reliability for the measurement distance of each target pixel as follows and outputs it to the reliability output unit 98.

FIG. 11 shows the flow of processing for obtaining the reliability.

The corresponding point searching unit 96 firstly executes step S1.
Then, for each target pixel, it is determined whether or not the distance can be normally measured depending on whether or not the pattern strength is larger than a certain threshold value A. For target pixels whose pattern strength is less than or equal to the threshold value A, since the pattern may be too weak to perform normal distance measurement, the process proceeds to step S4,
A low value of the reliability of the measured distance, for example, zero is output to the reliability output unit 98.

On the other hand, for a target pixel whose pattern intensity is larger than the threshold value, the process proceeds to steps S2 and S3. In step S2, the minimum value SSAD of the final total dissimilarity SSAD
It is detected that min is larger than a certain threshold value B, and in step S3, it is detected that the difference between the maximum value SSADmax and the minimum value SSADmin of the final sum SSAD of the dissimilarities is smaller than a certain threshold value C, and either of them is detected. If the case is detected, it means that an object outside the measurement range may have been erroneously detected, so the process proceeds to step S4, and the reliability of the measurement distance is low, for example, zero. , To the reliability output unit 98.

If neither of the above cases applies, it is determined that normal distance measurement has been performed, and the process proceeds to step S5.
Then, the reliability of the measured distance is calculated by the method described in the first embodiment, and is output to the reliability output unit 98.

12 and 13 show steps S2 and S3 above.
Shows an example in which it is determined that there is a possibility of false detection of an object outside the measurement range.

As shown in FIG. 12, when the minimum value SSADmin of the final sum of the dissimilarities is larger than the predetermined threshold value B, an object outside the measurement range may be erroneously detected. Therefore, the measurement distance Z
A zero value is output as the reliability of.

Further, as shown in FIG. 13, the difference between the maximum value SSADmax and the minimum value SSADmin of the final sum of the dissimilarity (SSAD ₃₀ −SS
Even if AD ₃₃ ) is smaller than the predetermined threshold value C, there is a possibility of false detection of an object outside the measurement range. Therefore, a zero value is output as the reliability of the measurement distance Z.

According to this embodiment, whether or not there is a possibility of erroneous detection of an object outside the measurement range appears for each pixel in the output reliability, so that when using the output distance image. It is possible to deal with this problem by excluding the measurement distance that may cause the false detection.

3. Embodiment for Addressing Occlusion Problem [1] Eighth Embodiment When occlusion occurs in a camera pair,
For the image part where the occlusion occurs,
It is considered that the degree of difference between the pixels of the pair that correspond in position to the reference window and the comparison window has a considerably large value. In view of this, in the present embodiment, first, the degree of difference between the target pixel and the corresponding candidate pixel is obtained for each camera pair, and if the degree of difference is considerably large, the occlusion occurs in the camera pair. Assuming that the camera pair has occurred, the camera pair is ignored, and only the dissimilarity of the remaining camera pairs is used. Window addition is performed to obtain the final total difference.

FIG. 14 shows the flow of processing for calculating the final sum of dissimilarities in the corresponding point search in this embodiment.
Note that the processing other than the corresponding point search may be the same as that of the conventional technique, or may be the same as that of any of the above-described first to seventh embodiments. In any case, in the corresponding point search, the final sum of dissimilarities is calculated by the procedure shown in FIG.

As shown in FIG. 14, steps S11 and S12
, The target pixel (i, j) and the assumed distance Zn are selected. Next, in step S13, for each camera pair k, the pixel value of the corresponding point candidate pixel in the comparison image corresponding to the selected target pixel (i, j) and the assumed distance Zn and the target pixel (i, j The difference ADk (i, j, Zn) between the pixel value of j) and the pixel value of j) is calculated. Next, in step S14, the camera pair that has the largest difference among the difference ADk (i, j, Zn) of all camera pairs is selected. The chosen camera is considered there to be occluded and is ignored. Next, in step S15, for each of the remaining camera pairs excluding the ignored camera pair, the degree of difference from the corresponding pixel in the comparison window is calculated for each pixel in the reference window, and the camera addition is performed. Next, step S16
Then, the "camera-added difference degree" for each pixel in the reference window is window-added. The result of the window addition is the target pixel (i, j) in step S17.
"Final sum of dissimilarities" at the assumed distance Zn of
It is output as (i, j, Zn).

In step S14, the difference ADk
Instead of ignoring the camera pair with the largest (i, j, Zn), ignore the given number of camera pairs whose dissimilarity ADk (i, j, Zn) is in the upper predetermined order. Good
Alternatively, a camera pair whose dissimilarity ADk (i, j, Zn) is equal to or higher than a predetermined threshold may be ignored. The number of camera pairs to be ignored, the threshold value, and the like may be determined in consideration of the type of object to be measured and the arrangement of cameras to be used. For example, when 9 cameras are arranged in a 3 × 3 matrix, there are 8 camera pairs, but among them, 3 camera pairs are ignored, and the remaining 5 camera pairs are used to calculate the difference degree. Calculate the final sum, and so on.

In the prior art, the occlusion is detected by using the window-added difference, whereas in the present embodiment, the occlusion is detected based on the difference between the target pixel and the corresponding candidate pixel. For detection, camera pairs that may have occluded can be removed before performing the window addition. Therefore, in the present embodiment, when calculating the final sum of dissimilarities,
After the camera addition of the dissimilarity for each pixel is performed, the window addition of the camera-added dissimilarity can be performed. As a result, even if the number of cameras is large, the window addition process needs to be performed only once, so that this embodiment can be realized by a simple circuit having one window addition circuit.

Further, according to this embodiment, since the problem of occlusion is reduced, the problem of bulging of the object contour caused by occlusion is also reduced.

Although the embodiment of the present invention has been described above, this is merely an example for explaining the present invention, and the scope of the present invention is not limited to this embodiment. Therefore, the present invention can be implemented in various other modes without departing from the gist thereof. The present invention can be implemented using one or more programmed computers, can be implemented using dedicated hardware circuits, or can be programmed computers and dedicated hardware circuits. It is also possible to use a combination of

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a first embodiment of the present invention.

FIG. 2 is a diagram illustrating an example of reliability of measured distance.

FIG. 3 is a diagram illustrating another example of the reliability of the measured distance.

FIG. 4 is a block diagram showing a configuration of a second exemplary embodiment of the present invention.

FIG. 5 is a block diagram showing a configuration of a third exemplary embodiment of the present invention.

FIG. 6 is a block diagram showing a configuration of a fourth exemplary embodiment of the present invention.

FIG. 7 is a block diagram showing the configuration of a fifth embodiment of the present invention.

FIG. 8 is a block diagram showing a configuration of a sixth exemplary embodiment of the present invention.

FIG. 9 is a diagram showing a window set when calculating the degree of deviation of a pattern.

FIG. 10 is a block diagram showing a configuration of a seventh exemplary embodiment of the present invention.

FIG. 11 is a flowchart showing a process of calculating reliability in the embodiment.

FIG. 12 is a diagram showing an example in which an object outside the measurement range may be erroneously detected.

FIG. 13 is a diagram showing another example in which an object outside the measurement range may be erroneously detected.

FIG. 14 is a flowchart showing a process of calculating a final sum of dissimilarities in the eighth embodiment of the present invention.

[Explanation of symbols]

1, 11, 31, 51, 71, 91 Reference image input section 2, 12, 32, 52, 72, 92 Comparative image input section 3, 13, 14, 38, 55, 73, 74 Pixel selection section 6, 19, 20, 35, 51, 56, 79, 80, 96
Corresponding point search unit 7, 22, 52, 57, 82, 97 Distance image output unit 21, 61 Measurement distance selection unit 37 Distance determination unit 36, 93 Pattern strength calculation unit 83 Pattern bias calculation unit

─────────────────────────────────────────────────── ─── Continuation of front page (51) Int.Cl. ⁷ identification code FI theme code (reference) H04N 7/18 H04N 7/18 C (72) Inventor Tetsuya Shinbo 1200 Manda, Hiratsuka-shi, Kanagawa Product made by Komatsu Co., Ltd. Tokoro Institute in the F-term (reference) 2F065 AA04 AA06 BB05 DD03 DD04 FF05 JJ03 JJ05 JJ26 KK01 QQ07 QQ24 QQ25 QQ32 QQ36 QQ37 QQ39 UU05 5B057 BA13 CA08 CA12 CA13 CA16 CB12 CB13 CB16 DC02 DC22 DC33 5C054 AA01 CA04 CC02 EH07 FC15 FD02 GA00 HA05 5L096 AA06 AA09 CA05 CA24 DA02 FA66 GA02 GA17 HA07 JA18

Claims (14)

[Claims] 1. A reference image (1) from one reference camera.
In a stereo image processing device which performs a corresponding point search by using a comparison image (2) from at least one comparison camera, a pixel value outside a predetermined pixel value range is detected from at least one of the reference image and the comparison image. Excluding the pixels that you have,
A pixel selection unit (3) that selects the remaining pixels, and pattern matching between the reference image and the comparison image for each of the various assumed distances for each target pixel in the reference image. A stereo image processing apparatus comprising: a corresponding point search unit (6) for measuring the distance of each target pixel by using the pixel selected by the selection unit. 2. A plurality of pixel selection units (13, 14) having different pixel value ranges, and pixels respectively selected by the plurality of pixel selection units are used to measure the distance of each target pixel. A plurality of corresponding point search units (19, 20) for calculating the reliability of the measured distance, a plurality of distances measured by the plurality of corresponding point search units for each target pixel, and a plurality of the distances. The measurement distance determination unit (21) for determining and outputting a final distance based on the reliability calculated for
The stereo image processing device described. 3. A pixel value determination unit (23) for determining a range of pixel values included in the reference image, wherein the measured distance determination unit (21) determines the plurality of distances and the plurality of distances. The stereo image processing device according to claim 2, wherein the final distance is determined from the calculated reliability and the pixel value range from the pixel value determination unit (23). 4. A reference image (3 from one reference camera).
1) and comparison images from at least one comparison camera (3
2) In a stereo image processing device that searches for corresponding points using and, by performing pattern matching between the reference image and the comparison image at various assumed distances for each target pixel in the reference image. , A first corresponding point searching unit (35) for measuring the distance of each target pixel, and a pattern intensity calculating unit (3) for outputting the pattern intensity of the reference image.
6) and whether or not to fix the distance of each target pixel measured by the first corresponding point searching unit (35) based on the pattern strength output from the pattern strength calculating unit (36). A distance determination determination unit (37) that determines whether or not the pixel whose distance has been determined by the distance determination determination unit (37) is excluded from the reference image, and from the comparison image,
A pixel selecting unit (38) for excluding pixels corresponding to the pixels excluded from the reference image at the determined distance and selecting the remaining pixels from the reference image and the comparison image; Pattern matching between the reference image and the comparison image is performed using the pixels selected by the pixel selection unit for each of the various assumed distances for each target pixel whose distance is undetermined. The second corresponding point searching unit (41) for measuring the distance of each target pixel whose distance is undetermined, according to the stereo image processing apparatus. 5. A reference image from one reference camera (5
1) and comparison images from at least one comparison camera (5
2) In a stereo image processing apparatus that performs a corresponding point search using and, a reference window is set for each target pixel in a standard image, and the reference window is set at each of various assumed distances in a comparison image. A comparison window is set, a pixel value range is set based on pixel values of pixels in the reference window, and pixels having substantially pixel values outside the set pixel value range are excluded from the comparison window. , A pixel selection unit (55) that selects the remaining pixels, and pattern matching between the reference window and the comparison window for each of the various assumed distances for each target pixel in the reference image, Using the selected pixel in the comparison window and the pixel in the reference window corresponding to the selected pixel, thereby , The stereo image processing apparatus including a corresponding point searching unit (56) for measuring the distance of each target pixel. 6. A reference image (7) from one reference camera.
1) and comparison images from at least one comparison camera (7
2) In a stereo image processing apparatus that performs a corresponding point search using and, excludes pixels having pixel values outside the first pixel value range from at least one of the reference image and the comparison image,
A first pixel selection unit (73) for selecting the remaining pixels, and pattern matching between the reference image and the comparison image for each of various target distances for each target pixel in the reference image. , A first corresponding point search unit (7) for measuring the distance of each target pixel by using the pixels selected by the first pixel selection unit.
9) and from at least one of the reference image and the comparison image,
A second pixel selection unit (74) for excluding pixels substantially having a pixel value outside the second pixel value range narrower than the first pixel value range and selecting the remaining pixels; Pattern matching between the reference image and the comparison image for each of the various assumed distances for each target pixel in the image using the pixels selected by the second pixel selection unit; As a result, a second corresponding point search unit (8) for measuring the distance of each target pixel
0), the pattern bias calculation unit (83) that outputs the pattern bias degree of the reference image, the output from the pattern bias calculation unit (83), and the first corresponding point search unit (79) A stereo image processing apparatus including a measured distance determining unit (81) that determines a final distance based on the measured distance and the distance measured by the second corresponding point searching unit (80). 7. A reference image (9) from one reference camera.
1) and comparison images from at least one comparison camera (9
2) In a stereo image processing device that performs a corresponding point search using and, for each target pixel in the reference image, for each of various assumed distances, pattern matching between the reference image and the comparison image. And a reliability output unit (98) for outputting a reliability for the measured distance, and the measured distance is measured by the corresponding point searching unit (96) for measuring the distance of each target pixel. Depending on the result value of the pattern matching for (S2), or between the result value of the pattern matching for the measured distance and the poorest value among the result values of the pattern matching for other assumed distances. A stereo image processing apparatus configured to change the reliability value for the measured distance (S5, S4) according to the difference (S3). 8. A stereo image processing apparatus which performs a corresponding point search using a reference image from one reference camera and a plurality of comparison images from a plurality of comparison cameras, wherein the reference is obtained for each of various assumed distances. Based on the degree of dissimilarity between the target pixel in the image and the corresponding candidate pixel in each comparison image, some comparison cameras in which occlusion may occur are selected from among the plurality of comparison cameras. Means for selecting and ignoring (S11 to S14), and for each of the various assumed distances, for each pixel in the reference window set in the standard image, comparison images from the remaining comparison cameras excluding the ignored comparison cameras. Means for obtaining the degree of difference between each pixel and the corresponding pixel in the comparison window, and adding the obtained degree of difference for each pixel to all the remaining comparison cameras (S15
And a means for window-adding the camera-added dissimilarity for all pixels in the reference window to obtain a final sum of dissimilarities, and the dissimilarity obtained for each of various assumed distances. And a means for measuring the distance of the target pixel based on the final sum of. 9. Reference image (1) from one reference camera.
And a comparison image (2) from at least one comparison camera, a stereo image processing method for searching for corresponding points, wherein pixel values outside a predetermined pixel value range are extracted from at least one of the reference image and the comparison image. Excluding the pixels that you have,
A pixel selection step (3) of selecting the remaining pixels, and pattern matching between the reference image and the comparison image for each of various target distances for each target pixel in the reference image, A stereo image processing method, comprising: using the pixels selected in the selection step, thereby measuring a distance between each target pixel and a corresponding point searching step (6). 10. A reference image (3 from one reference camera).
1) and comparison images from at least one comparison camera (3
2) In the stereo image processing method for searching corresponding points using and, by performing pattern matching between the reference image and the comparison image at various assumed distances for each target pixel in the reference image. , A first corresponding point search step (35) for measuring the distance of each target pixel, a pattern intensity calculation step (36) for outputting the pattern intensity of the reference image, and a pattern intensity calculation step (36) for output. Based on the pattern strength, the first corresponding point searching step (3
Regarding the distance of each target pixel measured in 5), a distance confirmation determination step (37) of determining whether or not to confirm the distance, and a distance determination determination step (37) from the reference image.
The pixels whose distances have been determined by are excluded from the comparison image, the pixels corresponding to the pixels excluded from the reference image at the determined distance are excluded, and the remaining pixels are removed from the reference image and the comparison image. A pixel selection step of selecting (38), and pattern matching between the reference image and the comparison image for each of various assumed distances for each target pixel of which at least distance is undetermined in the reference image, Using the pixels selected in the pixel selection step,
Thereby, a second corresponding point searching step (41) of measuring the distance of each target pixel whose distance is undetermined, is provided. 11. A reference image from one reference camera (5
1) and comparison images from at least one comparison camera (5
2) In a stereo image processing method for performing a corresponding point search using and, a reference window is set for each target pixel in a standard image, and a pixel value range is set based on pixel values of pixels in the reference window. Pixel value range setting step (5
5) and setting a comparison window corresponding to the reference window at each of various assumed distances in the comparison image, and pixels having substantially pixel values outside the set pixel value range from the comparison window. And selecting the remaining pixels, a pattern between the reference window and the comparison window for each of the various assumed distances for each target pixel in the reference image. Matching is performed using the selected pixel in the comparison window and the pixel in the reference window corresponding to the selected pixel, thereby finding a corresponding point for measuring the distance of each target pixel. (56) A stereo image processing method comprising: 12. A reference image (9) from one reference camera.
1) and comparison images from at least one comparison camera (9
2) A stereo image processing method for performing corresponding point search using and, for each target pixel in the reference image, for each of various assumed distances, pattern matching between the reference image and the comparison image. Corresponding point search step (96) for measuring the distance of each target pixel.
And a reliability output step (98) for outputting a reliability for the measured distance, the reliability output step (98) depending on a result value of the pattern matching for the measured distance ( S
2), or depending on the difference between the pattern matching result value for the measured distance and the poorest value among the pattern matching result values for other assumed distances (S3), the measured A stereo image processing method having steps (S5, S4) of changing a reliability value for a distance. 13. A stereo image processing method for searching corresponding points using a reference image from one reference camera and a plurality of comparison images from a plurality of comparison cameras, wherein the reference is obtained for each of various assumed distances. Based on the degree of dissimilarity between the target pixel in the image and the corresponding candidate pixel in each comparison image, some comparison cameras in which occlusion may occur are selected from among the plurality of comparison cameras. Steps (S11 to S14) of selecting and ignoring, and for each of the various assumed distances, for each pixel in the reference window set in the reference image, comparison images from the remaining comparison cameras except the ignored comparison camera. The difference between each pixel and the corresponding pixel in the comparison window is calculated, and the difference calculated for each pixel is added to all remaining comparison cameras by camera addition. -Up (S1
5), the step of adding the camera-added dissimilarity to all pixels in the reference window to obtain the final sum of the dissimilarity (S16, S17), and Measuring the distance of the target pixel based on the final sum of the dissimilarities. 14. A stereo image processing method for causing a computer to perform a corresponding point search using a reference image (1) from one reference camera and a comparison image (2) from at least one comparison camera. In the computer program of, the stereo image processing method, from at least one of the reference image and the comparison image, excludes pixels having substantially a pixel value outside a predetermined pixel value range,
A pixel selection step (3) of selecting the remaining pixels, and pattern matching between the reference image and the comparison image for each of various target distances for each target pixel in the reference image, A computer program for a stereo image processing method, comprising: using the pixels selected in the selection step, thereby finding the distance of each target pixel (6).