JPH0843055A - Method and apparatus for recognizing shape of three dimensional object - Google Patents

Abstract

PURPOSE:To accurately measure three dimensional shape of an object. CONSTITUTION:Two cameras 10 and 20 are used. A distance DS to a subject is obtained from the picked-up stereo images at a distance-measuring part 70 according to a stereo measuring method. Meanwhile, distance data DR and DL are obtained at front and rear stereo distance-measuring parts 60 and 80 according to a front and rear stereo measuring method while the distance to the subject is changed. A total processing part 90 processes each data DS, DR, DL through a weighted mean treatment in accordance with the respective reliability, thereby to obtain an ultimate distance to the subject. Accordingly, a three- dimensional shape of the subject can be highly accurately and stably measured with the measuring results according to two methods compensated each other.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention
Non-contact measurement and recognition of obstacles, terrain, industrial products, etc. 3
The present invention relates to a three-dimensional object shape recognition method and apparatus.

[0002]

2. Description of the Related Art Conventionally, as a three-dimensional object shape recognition method and apparatus using image data, a spatial method obtained from a correspondence relationship between respective points of the target object in a plurality of image data obtained by imaging the target object from different viewpoints. A method of recognizing and measuring by using position information according to the principle of triangulation, a so-called multi-view imaging method is widely used. In this multi-viewpoint imaging method, there is also a method of quasi-realizing the multi-viewpoint imaging method by using time-series image data taken by changing the viewpoint with a single eye.

Here, of the multi-viewpoint imaging method (or multi-viewpoint distance measuring method), the distance measuring principle of a normal stereo image pickup method in which two image pickup devices (hereinafter, referred to as cameras) are horizontally arranged and 1 The distance measurement principle of the front-back (parallax) stereo distance-measuring method using image data obtained by moving the two cameras back and forth in the optical axis direction will be described with reference to FIGS. 10 and 11, respectively.
The former corresponds to the basic principle of the horizontal / vertical in-plane parallax type multi-viewpoint imaging method described in [Claims], and the latter corresponds to the front-back parallax type multi-viewpoint imaging method described in [Claims]. It corresponds to the basic principle.

Further, the "horizontal / vertical in-plane parallax type multi-viewpoint imaging method" used here is an image picked up from different viewpoints parallel to an object, such as human eyes or a stereo imaging device. Then, the object is recognized according to the triangulation rule. Hereinafter, for the sake of simplicity, simply referred to as “stereo imaging method” or “stereo distance measuring method”. On the other hand, the "front-back parallax multi-viewpoint imaging method" uses images captured from different viewpoints in the front and back along the optical axis of the camera without changing the optical axis of the camera. Hereinafter, this method will be referred to as “front and rear stereo imaging method” or “front and rear stereo distance measuring method” for simplicity.

FIG. 10 is a diagram for explaining the basic principle of the stereo distance measuring method and the principle of triangulation. Symbols D, f, B, h _L , and h _R in the figure represent the subject distance, the focal length, the base line length, the imaging position of the object (target object) in the left imaging system, and the imaging position of the object in the right imaging system, respectively. Shows. From the figure, it is obvious that the following geometric relationship holds. That is, (h _R −h _L ): f = B: (D−f) (1) If the equation (1) is solved for the subject distance D, D = {(B · f) / (H _R −h _L )} + f …………………… (2) In the equation (2), B and f are constants and known,
To obtain the subject distance D, the parallax (h _R
-H _L ) needs to be detected. For this reason, a matching method and a gradient method (cf.
0-46878) and the like.

Next, the principle of distance measurement of the front-back (parallax) stereo distance measuring method will be described with reference to FIG. In FIG. 11, H,
Of D, D ', h, h', and f, H is the height from the point of interest of the subject to the optical axis, and D, D'is the distance between the subject and the imaging surface, when the camera is closer to the subject. Is D, and the distance when it is far is D '. Further, when D and D ', the image forming points on the image pickup surface are h and h', and the focal length is f. From the geometrical relationship of FIG. 11,

H: (D−f) = h: f ……………… (3) H: (D′−f) = h ′: f ……………… (4) It holds. From the equation (3), H · f = h · (D−f) ……………… (5) From the equation (4), H · f = h ′ · (D′−f) …… …………… (6) Since (5) = (6) here,

H · (D−f) = h ′ · (D′−f) (7) Now, it is assumed that the movement amount (difference between front and rear) (D'-D) of the camera is known. When this front-back difference (D'-D) is set to B and equation (7) is solved for D,

H · (D−f) = h ′ · (D′−f) = h ′ · {(B + D) −f} Therefore, D = {(B · h ′) / (h−h ′)} + F ………………………… (8)

To obtain Therefore, if the amount of movement (h-h ') of the point of interest on the imaging surface is obtained from the equation (8), the distance D to the subject can be calculated.

The distance measuring principles of the stereo distance measuring method for left and right parallax and the stereo distance measuring method for front and rear parallax have been described above with reference to FIGS. 10 and 11. From equations (2) and (8),
It can be seen that in both cases, the problem of associating each point between the left and right images or the front and rear images results.

[0012]

However, in the above conventional example, since the corresponding points are searched using the image data,
There is a drawback that sufficient accuracy cannot be obtained when there are not enough features in the image or conversely when there are many high frequency components such as a fine periodic pattern. In addition, since images obtained from different viewpoints are used, on the one hand there is always a hidden area, but on the other hand there is always a hidden area, and there is no corresponding point in this area, so it is theoretically impossible to measure the distance. There are drawbacks. As described above, the accuracy of the corresponding point search using the image data is directly influenced by the characteristics of the peripheral image data centering on the target point, so that there is a problem that the measurement result is uncertain and unstable.

In addition to the above-mentioned problems, the front-back (parallax) stereo distance measuring method has a problem that the accuracy is deteriorated due to the difference in the pattern magnification between the images to be associated with each other. Furthermore, the magnification increases as the front and rear viewpoint positions increase, the common field of view decreases between the images input by the left and right or the upper and lower cameras, the range that can be measured decreases, and the resolution decreases. There is a problem such as doing.

The present invention has been made in view of the above problems, and an object of the inventions of claims 1 and 8 is to increase the accuracy, reliability, and stability of the operation of measuring the three-dimensional shape of an object, thereby increasing the three-dimensional shape. It is to improve the reliability of the object shape recognition method and apparatus.

An object of the inventions of claims 2 and 9 is to improve the accuracy of the front-rear stereo imaging method.

It is an object of the inventions of claims 3 and 10 to utilize the object distance obtained by the horizontal / vertical in-plane parallax type multi-viewpoint imaging method to perform scaling with the front-back parallax type multi-viewpoint imaging method. Another object is to improve the distance measurement accuracy by the front-back parallax multi-viewpoint imaging method by correcting the magnification between the images.

It is an object of the inventions of claims 4 and 11 to associate the scaled images used in the front-back parallax type multi-viewpoint imaging method with a Fourier-Mell that is strong against the scaling of the pattern.
The in-conversion method is used to improve the ranging accuracy by the front-back parallax multi-viewpoint imaging method.

It is an object of the present invention to provide a filter for separating a spatial gradient component in an oblique direction and a spatial gradient component in a horizontal / vertical direction of an input image, each component being translated in the image. Supplying to a horizontal / vertical in-plane parallax type multi-viewpoint imaging unit whose main function is to detect and a front-to-back parallax type multi-viewpoint imaging unit whose main function is to detect radial movement of an image. Therefore, the distance measurement accuracy of both methods is improved.

It is an object of the sixth and thirteenth aspects of the present invention to provide a highly flexible subject that can flexibly meet various required precisions by successively increasing the ranging accuracy of each of the front-rear stereo imaging method and the stereo ranging method. To realize the three-dimensional shape recognition method and apparatus.

The object of claims 7 and 14 is to arrange a plurality of images arranged horizontally or vertically even when the amount of forward and backward movement is large compared to the unit time (image sample time) and the effective image area is narrowed. It is intended to direct the plurality of cameras to a subject so that a common subject can be imaged by the camera, so that it can be applied to, for example, an autonomous vehicle that travels at high speed.

[0021]

In order to achieve the above object, the inventions of claims 1 and 8 are characterized in that a stereo distance measuring method and a front and rear stereo distance measuring method are combined.

The inventions of claims 2, 3, 9 and 10 are characterized in that the magnification of the image is corrected.

The inventions of claims 4 and 11 are characterized in that the Fourier-Melan transform method is used for association in the front-back stereo distance measuring method.

According to the fifth and the twelfth aspects of the present invention, the image horizontal / horizontal
The present invention is characterized by having a filter for separating a vertical spatial gradient component and a diagonal spatial gradient component.

The invention of claims 6 and 13 is characterized in that the distance measurement results are integrated in the time direction.

The inventions of claims 7 and 14 are characterized in that a plurality of cameras are oriented so as to capture the same subject.

[0027]

According to the inventions of claims 1 and 8, the above-mentioned two different distance measuring methods are combined, and the information on the object distance measured based on each method is weighted averaged according to each reliability. , The accuracy, reliability and stability of the distance measuring operation are increased, and the reliability is improved.

According to the inventions of claims 2, 3, 9, and 10,
Since the operation is performed so as to correct the magnification of the image according to the front and rear stereo distance measuring method and the distance measuring information measured based on the stereo distance measuring method, the accuracy of the front and rear stereo distance measuring method and the stereo distance measuring method is improved.

According to the inventions of claims 4 and 11, the Fourier-Melan transform is described in "Applied Optics, 1976, Vo.
1.15 "(D.Casasen et al.)" Position, rotation a
nd scale invariant optical correlation "and JP-A-6
As described in detail in Japanese Patent Publication No. 28476, since it is possible to make a strong correspondence to the rotation and the magnification change of the image pattern, the accuracy of the front and rear stereo distance measuring method is improved. (Note that the Fourier-Melan transform procedure will be described in a third embodiment described later.)

According to the fifth and the twelfth aspects of the invention, the horizontal / vertical components extracted by the filter are sent to the distance measuring means by the stereo distance measuring method, and the extracted oblique components are sent to the distance measuring means by the front and rear stereo distance measuring method. Each is entered with priority. In the stereo distance measuring method, since the viewpoints are changed in parallel, it is common to search corresponding points in parallel directions. Therefore,
Horizontal or vertical spatial gradient information is required. On the other hand, in the front-back stereo distance measuring method, since the viewpoint is changed back and forth along the optical axis of the camera, the corresponding points are searched radially. Therefore, the gradient information in the diagonal direction is effective. Since the above-mentioned filter outputs the gradient component which each one is good at to each distance measuring means, the distance measuring accuracy is improved in both the stereo distance measuring and the front and rear stereo distance measuring.

According to the sixth and thirteenth aspects of the present invention, the distance information measured in the past and the latest distance information are weight-averaged based on their respective reliability, so that they suddenly occur at a certain time. The stability is increased with respect to the added noise, and the accuracy of distance measurement is improved by combining the information in the time direction. Further, since the distance measurement accuracy is gradually increased, the distance measurement with a high degree of freedom capable of flexibly meeting various required specifications can be performed.

According to the seventh and fourteenth aspects of the present invention, the trapezoidal distortion of the image generated when the camera is directed to the subject so that the same subject can always be input within the field of view according to the forward and backward moving speed of the camera. Since the correction operation is performed, the resolution and the distance measurement accuracy of the stereo distance measuring method when the moving speed is high are improved.

[0033]

【Example】

(First Embodiment) FIG. 1 is a drawing best showing the features of the present invention, and is an example in which the three-dimensional object shape recognition device of the present invention is mounted on an autonomous vehicle. In the figure, 10 and 20 are left and right cameras as an image pickup unit, 30 is a moving amount detection unit of a self-propelled vehicle, 40 and 50 are image memories such as a field memory or a frame memory, and 60 and 80 are the front and rear stereos described above. A distance measuring unit, 70 is the above-described stereo distance measuring unit, 90 is a distance measuring result integration processing unit, and 100 is a final distance measuring result output terminal.

The operation of FIG. 1 will be described in detail below. The two cameras 10 and 20 arranged horizontally or vertically are fixed so that their optical axes are parallel to each other, and the distance between them is
The so-called baseline length is also fixed. In the distance measuring units 60 and 80, images at different times are associated with each other via the image memories 40 and 50, respectively, and the movement amounts before and after the camera obtained by the movement amount detection unit 30 are used to obtain the equation (8). Distance measurement is performed according to the triangulation rule. The method of associating is not limited to the above-mentioned matching method and gradient method, but may be any method using image data, such as a cross-correlation method and a Fourier transform method.

On the other hand, the distance measuring unit 70 associates the image pairs at the same time when the object is imaged from different viewpoints, and calculates the object distance based on the triangulation law of the equation (2). The association method here is not limited to the specific method as described above.

In the integrated processing unit 90, the distance measuring units 60, 70,
The distance data D _R , D _S , and D _L obtained at 80 are weighted averaged based on their respective reliability. Taking the matching method as an example, the reliability of the distance data can utilize the correlation coefficient value of the matching, the sharpness of the peak of the correlation coefficient, and the like. Now, the distance information obtained from the correspondence between the time series images during the forward movement of the camera 10 is D _R , the distance information obtained from the correspondence between the time series images during the forward movement of the camera 20 is D _L , Then, the distance information obtained from the correspondence between the cameras 10 and 20 is D _S, and the reliability of each is k.
If expressed as _R , k _L , k _S , the distance D of the final result is, for example,

D = [{k _R / (k _R + k _L + k _S )} · D _R ] + [{k _L / (k _R + k _L + k _S )} · D _L ] + [{k _S / (k _R + k _L + k _S )} ・ D _S ] ………… (9)

When the two cameras 10 and 20 are arranged horizontally, the distance measuring accuracy along the horizontal edge is generally lower than the distance measuring accuracy along the vertical edge.
The opposite is true when the two cameras 10 and 20 are arranged vertically. That is, in the stereo distance measuring method, the direction in which the two cameras are arranged determines the direction in which the measurement can be performed with high accuracy. Therefore, the weight average of the D _R , D _L and D _S may be performed according to the direction of the edge by utilizing the directionality of the distance measuring accuracy.

As described above, in the first embodiment of the present invention,
By combining two distance measurement methods, the operation of remote sensing (non-contact measurement) by image processing, which was originally uncertain and unstable, is made reliable and stable, and the characteristics of input image data are changed. A highly robust and highly reliable three-dimensional shape measuring method and apparatus can be provided. Two or more cameras 10 and 20 may be used.

(Second Embodiment) FIG. 2 shows a second embodiment. Also in the second embodiment, as in the first embodiment, the three-dimensional object shape recognition device according to the present invention is mounted on an autonomous vehicle,
It recognizes obstacles and roads. The difference from the first embodiment is that magnification correction units 45 and 55 are newly added, and the other parts are configured the same as in FIG.

In the magnification correction units 45 and 55, the distance information D _S obtained by the stereo distance measuring unit 70 and the image memories 40 and 50 obtained by the movement amount detection unit 30 hold the image data in the self-propelled vehicle within the time period. The magnification of the image pattern is corrected using the movement amount B of. The specific contents of the magnification correction will be described below. The magnification ratio r of the image pattern can be obtained by solving the above equations (3) and (4) and solving for h / h '.

R = h / h ′ = (D′−f) / (D−f) (10) Then, using D′−D = B, r = h / h ′ = 1 + {B / (D−f)} …………………… (11) This outline is shown in FIG. According to this FIG.
It can be seen that the relationship between the magnification ratio r and the subject distance D is represented by a hyperbola whose asymptote is D = f, r = 1. this is,
It means that the farther the object is, the smaller the magnification is, and the closer the object is, the larger the magnification is. Therefore, it is necessary to adaptively correct the magnification of the image pattern according to the size of the subject distance.

In the present invention, the stereo distance measuring unit 70
More subject distance D_SAnd the movement detection unit 30
Since the amount of forward (or backward) B is obtained, these
Value D _S, B to adaptively correct the image pattern magnification
It is feasible.

Specifically, the size of the template (= block) used for matching is adaptively adjusted according to the subject distance using the equation (11), for example, 5 × 5 pel is 7 × 7.
After expanding to pel, the calculation of association is started. Alternatively, conversely, the enlarged image may be reduced or thinned according to the equation (11), and the correspondence may be performed after matching the image size before the enlargement.

In the second embodiment described above, the movement amount B,
Geometrically adjusts the difference in image magnification according to the subject distance D
Since it is corrected dynamically, it is compatible with front and rear stereo distance measurement
The effect of being able to significantly improve the accuracy of injury
is there. In this embodiment, the scaling factor is large especially according to the subject distance.
It is effective for near-distance objects that change rapidly. Short distance
When measuring the distance of an object, the size of the parallax
Therefore, occlusion problems tend to occur, but before that
Comprehensive reliability to increase the reliability of post-stereo ranging
It also has the unique effect of increasing Therefore, in this embodiment
In the integrated processing unit 90 in the
Measurement result D by the stereo distance measuring method_R, D _LAnd stereo measurement
Distance measurement result D_SYou can also weight average
It

(Third Embodiment) FIG. 4 shows a third embodiment of the present invention. In FIG. 4, the difference from the first embodiment is that the front and rear stereo distance measuring units 60, 80 are provided with Freeie-Melan converting units 260, 270, 280, 290. Next, the operation of the Friede-Melan converter is shown in FIG.
Will be explained.

The image data input in step S10 is first Fourier transformed in step S20. The amplitude (or power) after Fourier transform of the image pattern is
It does not change even if the pattern moves in parallel, and when the pattern is enlarged, for example, multiplied by a, it is reduced by 1 / a. Also,
With respect to the rotation θ of the pattern, the amplitude also rotates by the same θ. Therefore, in the next step S30, the amplitude image is expressed by performing coordinate conversion by taking the deflection angle on the vertical axis and the radius vector of the logarithmic scale on the horizontal axis.

After the above coordinate conversion, when the original pattern is rotated, it is translated in the vertical axis direction, and when the original pattern is scaled, it is translated in the horizontal axis direction. The amplitude after the Fourier transform again becomes invariable with respect to the rotation / magnification of the original pattern. If this pattern is used to make correspondence between the front and rear stereo distance measuring units in the next step S50, it is possible to realize distance measurement that is strong against rotation and variable magnification of the pattern.

As described above, according to the present embodiment, there is an effect that the accuracy of the front-back stereo distance measuring method is improved by using the Fourier-Melan transform for associating the scaled patterns. In particular, in the present embodiment, since the processing is completely independent of the stereo distance measuring unit 70, it is possible to prevent both the stereo distance measuring method and the front and rear stereo distance measuring methods from collapsing, and reliability,
It has the effect of improving safety.

(Fourth Embodiment) FIG. 6 shows a fourth embodiment of the present invention. In FIG. 6, reference numeral 30 is a movement amount detection unit of the cameras 10 and 20 in the optical axis direction, and 430 and 440 are image data. Cross filter for blocking diagonal spatial gradient components, 4
50 and 460 are subtractors 470, 480, and 4 that calculate the difference between the image input through and the image after filtering.
90 and 500 are image memories.

A particularly big difference between this embodiment and the first embodiment is that filters 430 and 440 are newly provided to the front and rear stereo distance measuring units 60 and 80 and the stereo distance measuring unit 70 respectively in the diagonal direction of the image. This is a point where a gradient component and a horizontal / vertical gradient component are input. Next, the filters 430 and 440
Explain the work of.

In the stereo distance measuring method, two cameras 10,
Since 20 are arranged so as to have the same subject distance and different viewpoints, the result of associating the images captured from the two cameras 10 and 20 with each other is almost the same although the length differs depending on the subject distance. It can be expressed as an optical flow lined up in the same direction. For example, when two cameras are arranged horizontally, the optical flows showing the correspondence between the pixels of the left and right images are almost horizontal, and similarly, the two cameras are arranged vertically. When arranged, the optical flows showing the correspondence between the pixels of the upper and lower images are substantially aligned in the vertical direction.

The optical flow representing the correspondence between the pixels of the front and rear images input and output in the direction of the optical axis of one camera is distributed radially around the center of the image.

Next, the aperture problem, which is a problem at the time of matching and motion detection by image processing, will be described. 7A to 7C are patterns having a spatial gradient only in the horizontal, vertical, and diagonal (45 °) directions, respectively, and show the state when the pattern is moved by V in the diagonal direction 45 ° to the right. There is. In the case of FIG. 7A, V _x can be detected correctly, but V _y cannot be detected. On the contrary, in the case of FIG. 7B, V _y can be detected correctly, but V _x cannot be detected. On the other hand, in the case of FIG. 7C, V _x and V _y can be detected correctly.

As described above, the correlation by image processing or the motion detection uses the spatial gradient of the image as an absolutely indispensable clue. Therefore, the stereo distance measuring unit 70 is preferentially supplied with a horizontal or vertical spatial gradient, and the front and rear stereo distance measuring units 60, 80 are preferentially supplied with an oblique gradient component, so that both hands This has the effect of improving the distance measurement accuracy of the method.

In FIG. 6, the front and rear stereo distance measuring units 60,
Although only the oblique component is sent to 80 and only the horizontal / vertical component is sent to the stereo distance measuring unit 70, in practice, the transition range of the band limitation of the cross filters 430 and 440 is moderated. , Image data can be supplied to each method so that both gradients are mixed appropriately. Horizontal / vertical gradients are preferable for stereo distance measurement, and isotropic gradient distributions are preferable for front and rear stereo distance measurement. The method of designing the cross filter is described in detail in Japanese Unexamined Patent Publication No. 4-10873, “Motion vector detecting device”.

(Fifth Embodiment) Next, a fifth embodiment will be described. In each of the first to fourth embodiments, the distance data D _S obtained from the association of the stereo images at a certain time and the distance data D _R and D _L obtained from the association of the time-series image pairs with different magnifications. Explained. However, the image is input while continuously zooming the focal length from the wide side to the tele side, so that D _S , D _R , and D _L, which are individually and momentarily obtained, are set to the next time D _S ′. , D _R ′, D _L ′ can be referred to.

As an example of a method of using distance data in the time direction, here, a Kalman filter (Ka
The method of applying the lman Filter) will be described. In order to apply the Kalman filter, the reciprocal 1 / D of the object distance is used as a state variable, and the ranging system is formulated as follows.

[0059] - (characteristic of the state variable) signal process _{(1 / D t) = (} 1 / D t-1) + η t ........................ (12) · observation process (triangulation law) u _t = B · f · (1 / D _t ) + ξ _t ………… (13) However, η _t is noise added to the signal and corresponds to the time variation of the distance data 1 / D. ξ _t is the noise added during the process of searching for the corresponding points and corresponds to the measurement error. B is the baseline length, f is the focal length, and u _t is the motion vector. At this time, the distance information (1
/ D _t ) ⁺ is expressed by the following equation (14). (1 / D _t ) ⁺ = K _t · u _t + [1-B · f · K _t ] · (1 / D _t ) ⁻ …………………… (14) where K _t is Kalman gain , (1 / D _t) ^- the distance data which is currently predicted from one time before the distance ^{_{data, (1 / D t) -}} = (1 / D t-1) + ........................ ( 15) is established. Here, the + symbol indicates the output of the Kalman filter, and the − symbol indicates the predicted value for the next time.

FIG. 8 is a diagram schematically illustrating the above equations (12) to (15). In the figure, 600 is a state variable of the distance measuring system, which is a signal to be obtained and is defined as 1 / (subject distance). The process of time change of the state variable 600 is expressed by the above equation (12). A multiplier 610 calculates the amount of movement (motion vector = parallax) on the image by multiplying the state variable 600 by the baseline length B and the focal length f. This relationship is based on the triangulation law, and D
>> This is a simplified version assuming f.

Reference numeral 620 is an adder, which adds the measurement error ξ added when the motion vector is obtained. 630 is the motion vector. Reference numeral 640 is a multiplier that multiplies the motion vector by the Kalman gain. The Kalman gain will be described later.

Reference numeral 650 is an adder for weighting the distance estimated based on the currently observed motion vector (= parallax) and the distance estimated from the past. Reference numeral 660 denotes a multiplier for multiplying the distance estimated from the past by a load. 670 is distance data estimated by the Kalman filter. This distance data 670 is sequentially used for the next distance estimation.

Next, the Kalman gain will be described. The Kalman gain determines the weight of the distance estimated based on the currently measured motion vector and the distance estimated from the past. In this distance measuring system using the Kalman gain, the Kalman gain K _t is expressed by the following equation. ^{_{K t = (P t - ·}} B · f) / {(B 2 · f 2 · P t -) + R t} ........................ (16) ^{_{where, P t - = P t-}} 1 + + Q _t-1 …………………… (17) where R _t is the variance of noise ξ added during the observation process, and Q _t.
t is the variance of noise η added to the signal process, and P _t is the square of the specified error (1 / D _t −1 / D _t ⁺ ) ² .

Now, when the measurement error R _t of the corresponding point search is small and the accuracy of the currently observed motion vector is high, that is, when R _t ≈0, the Kalman gain K is calculated from the equation (16).
_t approaches 1 / (B · f), and the load (1-B · f · K _t ) on the estimated value from the past approaches zero. On the contrary, if the value of R _t becomes large, the weight is adaptively changed so that the distance estimated from the past is more important than the distance obtained based on the currently observed motion vector.

As described above, in the Kalman filter, according to the reliability of the currently observed data (here, the motion vector or the parallax), that is, the magnitude of R _t , the "state estimated based on the latest observed data" is adaptively applied. The balance (variable (here, subject distance)) and "current state variable predicted from past transition" are changed to always obtain a stable estimated value.

Regarding the initial values 1 / D ₀ and P ₀ which are problematic when applying the Kalman filter, the present invention has an advantage that the values obtained by the front-back stereo distance measuring method can be used as they are. On the contrary, it is also possible to apply the Kalman filter to the zoom distance measuring method using the measurement result of the stereo distance measuring method as an initial value. Further, the R _t of determining the Kalman gain, as representing the reliability of the observation data, for example, can use the distribution of the residuals of the matching operation, As the prior art, International
"Kalman Filter-based Algorithms for Estimating" in Journal of Computer Vision, 3, 209-238 (1989).
Depth from Image Sequences "is available.

The example in which the Kalman filter is applied to the stereo distance measuring method has been described above, but it goes without saying that it can be applied to the front and rear stereo distance measuring method in the same manner.

The fifth embodiment is a method for integrating information in the time direction of distance measurement, and the distance measurement accuracy can be gradually increased without significantly increasing the amount of calculation, and the past By using the distance measurement result of, there is an effect that instability that depends only on the currently input image is reduced.

Further, since it is possible to successively improve the accuracy of distance measurement, it is possible to grasp the outline of the subject at high speed and to measure the three-dimensional shape of the subject with high accuracy and high density over a short period of time. It is possible, and distance measurement with a high degree of freedom according to the purpose can be realized.

In this embodiment, the Kalman filter has been described as an example of the method for fusing the distance measurement data in the time direction, but another method capable of integrating information in the time direction may be used.

In the above-described first to fifth embodiments, the optical axes of the stereo image pickup section are assumed to be parallel. However, in the process of traveling, the angle of view may be so narrow that the target object may not be captured in the common field of view. In this case, both cameras may be driven inward in association with the traveling speed.

(Sixth Embodiment) FIG. 9 shows a sixth embodiment. The difference from FIG. 1 is that the two cameras 10 and 20 are facing each other inward, and that distortion correction units 700 and 710 for correcting image distortion are added immediately after the cameras 10 and 20. .

As shown in FIG. 9, an image taken by the cameras 10 and 20 from two different viewpoints (left and right) has trapezoidal distortion due to the intersection of the optical axes. Therefore, it is necessary to correct the trapezoidal distortion before sending the image data to the distance measuring units 60, 70, 80.

Here, the base line length of the two cameras 10 and 20 is B, the angle formed by the two cameras 10 and 20, that is, the vergence angle is P, and the coordinates of one reference image out of a plurality of captured input images. (X, y), where the center of the image is the origin (0,
0). Letting f be the focal length of the imaging lens and D be the subject distance, the correspondence relationship (u,
v) is represented by the following formula. u = −f · B / D (x, y) − {(x ² + f ² ) · sinP} / (f · cos P + x · sinP) …………………… (18) v = [-{f -(F · cosP + x · sinP)} · y] / (f · cosP + x · sinP) …………………… (19)

The first term of the equation (18) -f.B / D (x,
y) indicates the parallel shift amount of the image due to the viewpoints of the two cameras 10 and 20 being different, and is not related to the trapezoidal distortion. The other terms are all terms that include the convergence angle P and cause trapezoidal distortion. (18), (1
It can be seen from the equation (9) that the image is distorted more in the vicinity of the image having a larger value of (x, y).

In the distortion correction units 700 and 710, (18),
The term including the vergence angle P in the equation (19) is inversely corrected to geometrically correct the image as if it was input by the two cameras whose optical axes are parallel to each other.

In the sixth embodiment described above, the trapezoidal distortion of the object, which occurs due to the convergence angle of the zoom lens, is geometrically corrected before association, so that the stereo distance measuring method and the front-rear stereo measuring method are performed. It is possible to improve the measurement accuracy with the distance method. Further, since each of the cameras 10 and 20 gazes at the subject, it is possible to capture the subject with more pixels, and it is possible to measure the distance of the subject in more detail. That is, there is a unique effect that the resolution of distance measurement is improved.

[0078]

As described above, according to the inventions of claims 1 and 8, there is an effect of improving the reliability of recognition of a three-dimensional object shape by image processing.

Further, according to the inventions of claims 2 and 9, there is an effect that the distance measuring accuracy of the front and rear distance measuring method is improved when the magnification between images is different.

According to the third and tenth aspects, there is an effect that the distance measurement accuracy of the front-rear stereo distance measurement method is improved when the magnification between images is different.

According to the fourth and eleventh aspects of the present invention, there is an effect that a front and rear stereo distance measuring method that is resistant to image scaling and rotation can be realized.

According to the fifth and the twelfth aspects of the present invention, gradient components in various directions of the image are effectively distributed to the stereo distance measuring unit and the front and rear stereo distance measuring units, so that the distance measuring accuracy of both methods is improved. effective.

According to the sixth and thirteenth aspects of the present invention, the accuracy of the front-rear stereo distance measuring method and the stereo distance measuring method is improved to be resistant to sudden noise, and the distance can be measured according to the target accuracy and resolution. effective.

According to the seventh and the fourteenth aspects, even if the distance measuring device largely moves back and forth, a common subject can be captured by a plurality of cameras, and the image distortion generated at that time is corrected. This has the effect of improving the resolution and distance measurement accuracy of the distance method.

[Brief description of drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a block diagram showing a second embodiment of the present invention.

FIG. 3 is a characteristic diagram showing a relationship between a subject distance and an image scaling ratio.

FIG. 4 is a block diagram showing a third embodiment of the present invention.

FIG. 5 is a flowchart showing a Fourier-Melan transform.

FIG. 6 is a block diagram showing a fourth embodiment of the present invention.

FIG. 7 is a configuration diagram illustrating a problem related to aperture correction.

FIG. 8 is a configuration diagram showing a Kalman filter used in a fifth embodiment of the present invention.

FIG. 9 is a block diagram showing a sixth embodiment of the present invention.

FIG. 10 is a configuration diagram showing a conventional stereo distance measuring method.

FIG. 11 is a configuration diagram showing a conventional front-back stereo distance measuring method.

[Explanation of symbols]

10, 20 Camera 30 Moving amount detector 40, 50, 470, 480, 490, 500 Image memory 60, 80 Front and rear stereo distance measuring unit 70 Stereo distance measuring unit 90 Integrated processing unit 45, 55 Magnification correction unit 260, 270, 280 , 290 Fourier-Melan transform unit 430, 440 Cross filter 700, 710 Trapezoidal distortion correction unit

Claims (14)

[Claims] 1. A method for measuring and recognizing a three-dimensional shape of an object in a non-contact manner by a horizontal / vertical in-plane parallax type multi-viewpoint imaging method using a plurality of images input from different viewpoints at the same subject distance. And the step of recognizing an object by the front-back parallax type multi-viewpoint imaging method using a plurality of images input from the same viewpoint at different subject distances. Three
A method of recognizing a shape of a three-dimensional object, integrating the weighted recognition results, and averaging the recognition results. 2. The three-dimensional object shape recognition method according to claim 1, further comprising a step of performing magnification correction between images of different magnifications used in the front-back parallax multi-viewpoint imaging method. 3. The three-dimensional object shape recognition method according to claim 1, further comprising a step of performing magnification correction according to a subject distance obtained by the horizontal / vertical in-plane parallax type multi-viewpoint imaging method. . 4. The three-dimensional object shape recognition method according to claim 1, wherein a Fourier-Melan transform method is used for associating images of different magnifications used in the front-back parallax multi-viewpoint imaging method. 5. A filter separates a diagonal spatial gradient component of the image and a horizontal / vertical spatial gradient component of the image, and the horizontal / vertical spatial gradient component of the image is the horizontal / vertical in-plane parallax. Type multi-view imaging method
The three-dimensional object shape recognition method according to claim 1, wherein the spatial gradient component in the diagonal direction of the image is preferentially supplied to the front-back parallax multi-viewpoint imaging method. 6. The three-dimensional object shape recognition method according to claim 1, wherein the past three-dimensional object shape recognition result is used. 7. The three-dimensional object shape recognition method according to claim 1, wherein a plurality of image pickup devices are oriented so that the same subject is included in the visual field according to the forward and backward movement speeds of these image pickup devices. 8. A recognition means for recognizing an object by a horizontal / vertical in-plane parallax multi-viewpoint imaging method using a plurality of images input from different viewpoints at the same subject distance, and input from the same viewpoint at different subject distances. Recognizing means for recognizing an object by a front-back parallax multi-viewpoint imaging method using a plurality of images, shape measurement of a three-dimensional object independently or cooperatively obtained by each recognizing means, and integrated processing for weighted averaging the recognition results. And a three-dimensional object shape recognition device. 9. The three-dimensional object shape recognition device according to claim 8, further comprising a correction unit for correcting magnification between images of different magnifications used in the front-back parallax type multi-viewpoint imaging method. 10. The three-dimensional object shape according to claim 8, further comprising correction means for correcting magnification in accordance with a subject distance obtained by the horizontal / vertical in-plane parallax type multi-viewpoint imaging method. Recognition device. 11. The three-dimensional object shape recognition device according to claim 8, wherein a Fourier-Melan transform method is used for associating images of different magnifications used in the front-back parallax multi-viewpoint imaging method. 12. A filter for separating a spatial gradient component in an oblique direction of an image and a spatial gradient component in the horizontal / vertical direction of the image, wherein the spatial gradient component in the horizontal / vertical direction of the image is in the horizontal / vertical plane. It is characterized in that it is preferentially supplied to the recognition means by the internal parallax type multi-viewpoint imaging method, and the spatial gradient component in the oblique direction of the image is preferentially supplied to the recognition means by the front-back parallax type multi-viewpoint imaging method. The three-dimensional object shape recognition device according to claim 8. 13. The three-dimensional object shape recognition device according to claim 8, wherein a past three-dimensional object shape recognition result is used. 14. The three-dimensional object shape recognition device according to claim 8, wherein a plurality of image pickup devices are oriented so that the same subject is included in the field of view according to the forward and backward movement speeds of these image pickup devices.