JP3138080B2 - Automatic calibration device for vision sensor - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an apparatus for automatically executing calibration of an assembling robot or a visual sensor for inspection.

[0002]

2. Description of the Related Art In order to automate the assembly and inspection processes performed by humans, it is often necessary to use a visual device corresponding to human eyes. Such a visual device mainly captures an object as an image with a camera (imaging means) or the like, and detects a spatial position where the object is placed from a position in the image.

[0003] In order to perform positioning using the visual device, it is necessary to calibrate the visual device in advance. In other words, in order to associate the position in the image with the spatial position, the camera parameters such as the position where the camera is arranged in spatial coordinates, the posture, the focal length of the camera, and the aspect ratio must be determined. Must.

Conventionally, calibration of a visual device has been
A plurality of teaching points whose coordinate values are known in spatial coordinates are imaged by a visual device to generate a teaching point image, and calibration is performed using the spatial coordinate values of each teaching point and the image coordinate values of the teaching point. This is achieved by performing calculations based on the algorithm. In this case, it is general that the association between the spatial coordinate value of each teaching point and the image coordinate value is manually performed by a human being determined from the teaching point image. Further, in order to obtain a three-dimensional position and orientation in the spatial coordinates of the camera, not all teaching points are distributed on one plane, that is, three-dimensionally distributed teaching points are required. On the other hand, a calibration method that can be performed using a calibration work posture having teaching points distributed two-dimensionally (on a plane) has been proposed by Roger Tsusai et al. (Roger Y. Tsai, “A Ve
rsatile Camera Calibration Technique for High-Accu
racy 3D Machine Metrology Using Off-the Shelf TV C
ameras and Lenses ", IIIE J. Robotics and Automati
on, Vol. RA-3, No. 4, pp. 323-344, 1987) In this method, a calibration work with a teaching point arranged on a plane is photographed diagonally, and from the position of the teaching point on the image, Camera parameters are determined.

[0005]

It is cumbersome to manually associate the spatial coordinate values of the teaching points and the image coordinate values required for calibration with the operator, and the operator looks at the teaching point image while looking at the teaching point image. There is a possibility that an error may occur in the process of performing the association.

Further, when obtaining the three-dimensional position and orientation of the camera, in a calibration requiring three-dimensionally distributed teaching points, the operation of obtaining image coordinate values of the teaching points is complicated, and the apparatus is complicated. Cause transformation. On the other hand, if the method proposed by Tsusai et al. Is used, such a problem can be solved.However, if the camera is positioned close to the vertical with respect to the calibration work, the measurement accuracy deteriorates.
There is a problem in that high-precision measurement is limited by the arrangement of cameras.

[0007] That is, the proposal of Tsusai et al. Does not describe a method for associating a plurality of teaching points with each other on an image, and has a problem that this association cannot be performed automatically. . Furthermore, in the Tsusai way, when obtaining the camera parameters, x _w which teaching point is located, the camera parameter defining the y _w plane, the camera X, the relationship between these planes between the Y plane from the corresponding relationship Is calculated, and camera parameters for associating the image coordinates with the three-dimensional space coordinates including z _w are calculated from the result. Therefore, z _w calculated from the image coordinate values includes:
Errors during the calculation of x _w and y _w are accumulated, and the errors increase. Thus, the Tsusai method has a problem in that, when the calibration work is arranged in a state nearly perpendicular to the camera, the measurement accuracy is greatly reduced.

The present invention has been made in order to solve the above problems.
Simple and highly accurate calibration that automatically associates the spatial coordinate values of the teaching points with the image coordinate values and can execute calibration that is not restricted by the camera arrangement from only the two-dimensionally distributed teaching points. It is an object to provide a simple calibration device.

[0009]

According to the present invention, there is provided an apparatus for automatically calibrating a visual sensor, comprising a plurality of reference teaching points having a specific shape having directivity. An imaging unit for imaging a calibration work having a teaching point on a plane, and a position direction for recognizing the position and direction of the reference point in the image from the position and shape of the reference teaching point in the image obtained by the imaging unit Recognition means, coordinate-to-coordinate association means for associating spatial coordinate values of each teaching point with image coordinate values of each teaching point obtained by the imaging means based on the recognized coordinates, and a plurality of associated spaces Calibration calculation means for calculating each parameter of the imaging device from the correspondence between the sets of image coordinate values.

The above-mentioned calibration calculating means calculates the parameters of the image pickup apparatus for the correspondence between the coordinate values on the plane of the calibration work and the image coordinates using the set of the space of each teaching point and the image coordinates. A feature of the imaging apparatus is to determine the correspondence between the spatial coordinates including the coordinates in the direction perpendicular to the plane of the calibration work and the image coordinates again using the set of the space and the image coordinates of each teaching point.

[0011]

[Operation and Effect] The calibration device for a visual sensor according to the present invention has the above-described configuration. First, the entire calibration work is located at an arbitrary position within a field of view of the imaging device (camera). Position the camera in the posture. Next, the calibration work is imaged, and the position and direction of the reference teaching point are recognized from the image. For example,
If the teaching point is a perfect circle and the reference teaching point is an ellipse, the ratio of the minor axis to the major axis on the image is determined. The position of the reference point on the image is recognized from the area centroid, and the direction on the image is recognized from the major axis and minor axis directions of the ellipse. Then, the coordinate values of each teaching point on the image and the coordinates of each teaching point on the calibration work are associated with each other from the reference point and direction on the image obtained in this manner. Then, each parameter of the imaging device is calculated from the correspondence between the set of the plurality of associated spaces and image coordinate values.

As described above, since the calibration can be performed by using the calibration work including only the teaching points arranged on the plane, the apparatus and the processing procedure can be simplified. Since the position and direction of the reference point can be recognized from the shape of the reference teaching point, each teaching point on the image can be automatically associated with each teaching point on the calibration work, eliminating human errors. Thus, such high-speed processing can be performed.

In the calculation of the parameters, first, parameters relating to the correspondence between the two-dimensional coordinates (x, y) of the space and the image coordinates (X, Y) are obtained. Then, after obtaining these parameters, the other parameters are used as variables, and a numerical operation is performed again using a set of a plurality of actually obtained space and image coordinate values to obtain the values of these other parameters. Therefore, when calculating other parameters, highly accurate parameters can be calculated without accumulating errors for the parameters obtained first. In particular, in consideration of use in a factory or the like, a camera is often installed in a direction nearly perpendicular to a measurement target. This is because when a measurement target is observed from an oblique direction with a camera, the measurement target is captured as a distorted image. The present apparatus is not limited by the arrangement of the cameras, and can perform high-precision calibration even in a state close to perpendicular to the calibration work. Therefore, the applicable range is very wide, and effective calibration can be performed.

[0014]

FIG. 1 is a conceptual diagram of an apparatus according to the present invention.
The calibration work 2 is imaged by the image processing device 3. The image is sent to the arithmetic unit 4 and the reference position and direction are detected from the reference teaching point of the calibration work 2. The arithmetic unit 4 associates the coordinate values of each teaching point in the image with the spatial coordinate position from the reference position, direction, and array information of the teaching points given in advance by the data input unit 5. Finally, a calibration calculation is similarly performed in the calculation unit 4 from the set of the image of each teaching point and the spatial coordinate values, and camera parameters are obtained.
The obtained camera parameters are sent to another device by the data output unit 6 and used in subsequent processing.

FIG. 2 shows the calibration work 2. Thus, a plurality of teaching point marks are arranged on the work plane, and one of them includes a reference teaching point having directionality. In this figure, the reference teaching point has an elliptical shape as an example, and the other teaching points have a perfect circle shape. However, there is no particular limitation on the shape, except that a directional shape is required for the reference teaching point. .

FIG. 3 shows an example of a processing procedure for automatically associating the spatial coordinate values of each teaching point with the image coordinate values.

In step 1, the image is divided into regions by a luminance level binarization process or the like, and the region processing is performed so that the teaching point portion is a group of pixels having a value of 0.

Next, in step 2, a moment of inertia is calculated for each area to determine a major axis and a minor axis. An area where the ratio of the minor axis to the major axis is the smallest is defined as the area of the reference teaching point. In this example, the reference teaching point is an ellipse and a circle other than the reference point is a perfect circle. Therefore, the ratio of the minor axis to the major axis is smaller in the area of the reference teaching point than in other areas. The area center of gravity of the area of the reference teaching point is defined as the reference origin, and the major axis and the minor axis are defined as x and y axes, respectively.

In step 3, the reference origin obtained earlier and x,
From the direction of the y-axis, the image coordinate value of each teaching point is correlated with the spatial coordinate value given in advance.

That is, as shown in FIG. 4, an area is searched in the x-axis direction from the reference point, and the area found for the first time is labeled as a teaching point 1. Next, an area is searched in the x-axis direction from the teaching point 1 in the same manner, and the found area is set as the teaching point 2. If the labeling is performed up to the teaching point 3 in the same manner, an area is searched from the reference origin in the y-axis direction based on the array information given in advance, and the found area is set as the teaching point 4. Then from the teaching point 4 to x
The area is searched in the axial direction, and the teaching points 5 to 7 are labeled. By repeating such processing, labeling from 1 to 15 is performed based on the reference origin, the x and y axis directions and the arrangement information of the teaching points, and the area centroid of each labeled area is represented by the image of the teaching point. The coordinates are automatically associated with the spatial coordinate values of the teaching points that have been given in advance.

In this example, there are 16 teaching points including the reference teaching points in the calibration work. However, by changing the arrangement information of the teaching points, it is possible to correspond to an arbitrary number and arrangement.

In this example, the reference origin and direction of the calibration work and the position of each teaching point are obtained by binarizing the image, dividing the area, and the like. The present invention is not particularly limited to the processing method, because other methods such as obtaining the amount can be considered. In the present invention, since all the teaching points are located on the plane of the calibration work, all the values in the z-axis direction among the spatial coordinate values of each teaching point given in advance are given as zero.

FIG. 5 shows the structure of the camera model.
Coordinate system O _c where the camera as a reference in the space coordinate system O _w where the point P exists is present. At this time, representing a point P O _w as viewed from the coordinate values _{(x w, y w, z} w) and O _c as viewed from the coordinate values (x, y, z). Further, the point P is located on the plane perpendicular to the z _c axis at a focal distance f from the O _c image coordinate system O.
Also represented by _i , O _c P and x which is the light receiving surface of the camera
It is observed as an intersection P _u (X _u , Y _u ) with the _i , y _i plane. On the other hand, in the image processing apparatus, the point P is in the frame coordinate system O
_f within Pf (X _f, Y _f) is observed at. At this time, each coordinate value has the following relationship. The relationship between (x _w , y _w , z _w ) and (x, y, z) is

(Equation 1) Here, R and T are a rotation matrix and a translation vector from O _w to O _c , respectively, and are expressed as follows.

[0024]

(Equation 2) Next (x, y, z) and (X _u, Y _u) relationship with is expressed by the following equation.

[0025]

(Equation 3) Here, f is the focal length of the camera lens.

Then, (X _u , Y _u ) and (X _f , Y _f )
Is expressed by the following equation.

[0027]

(Equation 4) Here, C _x and C _y are the coordinate values of O _{i in} the frame coordinate system, respectively, and (C _x , C _y ) is called an image center. The d _x, d _y is the distance between the centers of light receiving elements adjacent the camera, s _x is the horizontal driving frequency of the camera F _c, when an A / D converter of the frequency of the image processing apparatus F _f, s is a value represented by _x = F _c / F _f.

Based on the above relational expression, camera parameters are obtained based on the set of a plurality of teaching point spaces and image coordinate values previously obtained.

The spatial coordinate values and the image coordinate values of a plurality of teaching points are respectively represented by ( _xwi , _ywi , _zwi ), ( _Xfi , _Yfi ) (i
= 1, 2,..., 15), and assuming that the intersection point when a perpendicular line is drawn from the point P to the z-axis of O _c is P _o , O _i P
because it is _u ‖P _o P, it is a xY _u -yX _u = 0. From Expressions 1 to 4 and _zwi = 0 (i-1, 2,..., 15)

(Equation 5) Transform the above formula

(Equation 6) The above equation 5 or more _{(x wi, y wi, 0} ) (X fi, Y fi) Solving the least squares method or the like ^{_{using, T y -1 r 1, T}} y
^{_{-1 r 2, T y -1 T}} x, T y -1 r 4, T y -1 r 5 is obtained. next

(Equation 7) From the orthonormality of the rotation matrix R,

(Equation 8) Becomes here,

(Equation 9) However, if all the rows or columns of C are zero,

(Equation 10) Here, r ′ _i and r ′ _j are elements of a non-zero C column or row.

Now that T _y ² has been obtained, T _y = √
Assuming that (T _y ² ) ≧ 0, T _y ^-1 r ₁ , T _y ^-1 r ₂ , T _y ^-1
From T _x , T _y ^-1 r ₄ , T _y ^-1 r ₅ , r ₁ , r ₂ , r ₄ ,
r ₅ , T _x T _y are obtained.

However, when T _y ≧ 0, r ₁ , r ₂ , r ₄ ,
Since r ₅ , T _x , and T _y have been obtained, the validity of the sign of T _y is checked using the space of the teaching point and the image coordinate values. That is,
From the parameters just obtained and equations (1) and (4), (x _i , y _i , z _i ), (X _ui ,) for an arbitrary set of (x _wi , y _wi , z _wi ) and (X _fi , Y _fi ) Y _ui ), x
_{If the} signs of _i and X _ui and y _i and Y _ui are different, it is determined that the assumption that T _y ≧ 0 is erroneous, and r ₁ , r ₂ , r ₄ , r ₅ , The signs of T _x and T _y are inverted. This corresponds to x _i and X _ui and y _i and Y _{ui in} FIG.
Is a determination using the fact that is always the same sign.

The calculation method of the camera parameters so far is almost the same as the method proposed by Tsusai described above. However, the present invention relates to a camera parameter calculation method that has not yet been determined, and unlike the method of Roger Tsusai, can obtain the camera parameter with high accuracy regardless of the camera arrangement.

That is, when the expressions 1 to 4 are put together,

[Equation 11] Since z _w = 0, transform

(Equation 12) However,

(Equation 13) It is.

Up to now, r ₁ , r ₂ , r ₄ , r ₅ , T _x , and T _y included in the right side of Expression 12 are known values. Therefore, the teaching point data ( _xwi , _ywi , _zwi ),
Using (X _fi , Y _fi ), r ′ ₇ , r ′ ₈ , and T ′ _z can be obtained by the least square method.

Here, from the orthonormality of the rotation matrix R, it is clear that the following equation holds.

[Equation 14] Transform the above formula

(Equation 15) Becomes However, f> 0.

Here, r ₃ ² , r ₆ ² , and r ₉ ² can be obtained by the following equation using the orthonormality of the rotation matrix R.

(Equation 16) The focal length f is determined by substituting r ₉ ² in the above equation in Equation 15. Further, using Equation 13 to f, the remaining parameters,
r ₇ , r ₈ and T _z are obtained.

Finally, r ₁ , r ₂ , r ₄ , r ₅ , r ₇ , r
The codes of r ₃ , r ₆ , and r ₉ are obtained from r ₃ ² , r ₆ ² , and r ₉ ^{2 in} Equation ₈ and Expression 16 using the orthonormality of the rotation matrix R in the same manner as described above.

As described above, all camera parameters are obtained from the space of a plurality of teaching points and a set of image coordinate values ( _xwi , _ywi , _zwi ) and ( _Xfi , _Yfi ).

[Brief description of the drawings]

FIG. 1 is a conceptual diagram of an automatic calibration device according to the present invention.

FIG. 2 is a plan view of a calibration work.

FIG. 3 is a diagram showing a processing procedure for associating a spatial coordinate value of a teaching point with an image coordinate value.

FIG. 4 is a perspective view of a calibration work for explaining labeling of a teaching point.

FIG. 5 is a diagram illustrating a camera model.

[Explanation of symbols]

 Reference Signs List 1 camera 2 calibration work 3 image processing device 4 arithmetic unit

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Hiroshi Moribe 41-41, Yokomichi, Chukumi-cho, Nagakute-cho, Aichi-gun, Aichi Prefecture Inside Toyota Central R & D Laboratories Co., Ltd. (56) References JP-A-4-181106 (JP, A) JP-A-64-2889 (JP, A) JP-A-63-61903 (JP, A) JP-A-61-277012 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G01B 11/00-11/30 102 B25J 19/02-19/06 G06T 1/00 H04N 7/18

Claims (2)

(57) [Claims] An apparatus for automatically calibrating a visual sensor, comprising: an image pickup device for picking up an image of a calibration work having a plurality of teaching points including a reference teaching point of a specific shape having directionality on a plane. Means, position and orientation recognizing means for recognizing the position and direction of the reference point in the image from the position and shape of the reference teaching point in the image obtained by the imaging means, and a space for each teaching point based on the recognized coordinates. An inter-coordinate associating means for associating the coordinate values with the image coordinate values of the respective teaching points obtained by the imaging means; and a plurality of associated spaces and respective parameters of the imaging means from the correspondence relationship of the set of image coordinate values. An automatic calibration device for a visual sensor, comprising: calibration calculation means for calculating. 2. The apparatus according to claim 1, wherein said calibration calculation means includes:
After obtaining the parameters of the imaging means for the correspondence between the coordinate values on the plane of the calibration work and the image coordinates using the set of image coordinates, the space of each teaching point and the set of the image coordinates are again used by using the set of image coordinates. An automatic calibration device for a visual sensor, wherein a parameter of an image pickup means is obtained for a correspondence between spatial coordinates including coordinates in a direction perpendicular to a plane and image coordinates.