KR20150069927A - Device, method for calibration of camera and laser range finder - Google Patents

Abstract

According to the present invention, there is provided a calibration apparatus and method for performing coordinate conversion using image information and distance information, respectively, wherein a flat member including a vertex is used, and vertexes predicted from the distance information and vertexes extracted from the image information A coordinate change is implemented using a pair of vertices. Accordingly, it is possible to overcome the limitation of the position where the laser sensor, the camera and the flat plate member are disposed.

Description

Technical Field [0001] The present invention relates to a calibration apparatus and a calibration method for a camera and a laser sensor,

The present invention relates to calibration between heterogeneous sensors for terrain and object recognition.

The range of use of a laser sensor (or a laser distance sensor) having angle and distance information is getting wider as the field of robot and autonomous moving vehicle develops. Industrial or military applications often use camera images together with the laser sensors to utilize more information. However, in order to use the laser sensor and the camera together, it is essential to calibrate the calibration to find the correspondence between the laser sensor and the camera.

In order to perform calibration between the distance measuring laser sensor and the 2D CCD camera acquiring the color (brightness) image, a method of utilizing a new shape board has been developed. For example, it is a method of automatically detecting a correspondence point by drilling a hole with a certain radius in a rectangular plate, detecting laser scan data due to a difference in distance (depth) in the hole, and hole caused by a change in brightness of the color image. However, there is a case where the laser beam that is reflected back through the hole on the plate obliquely hits the back surface of the flat plate and does not return.

A method has been proposed in which two rectangular plates are attached to an acute angle V shape and the parameters of the conversion matrix are determined in a direction that minimizes the distance of the 3D point cloud of the distance sensor projected to the edge of the two plates. But it is not based on a direct correspondence point between the data and brightness image data, and thus has a disadvantage that the process is complicated or requires human intervention.

In order to detect more accurate correspondence between laser distance data and color image data, a method of detecting the correspondence between direct distance and brightness in a color image using a visible IR (infra-red) line has been disclosed. However, there is a disadvantage that the method can be performed only when an IR sensor is provided.

A method of detecting a direct correspondence point between the laser distance data and the image data using the geometric characteristics of the board has been proposed. For example, from the ratio of the distance between the laser data obtained by using the triangular board and the length of the bottom of the triangle which is already known, it is possible to infer two points passing through two triangular segments obtained from the camera image. You can calculate the values of

However, this method has a limitation that the bottom surface of the triangular planar board must be parallel to the scanning surface of the laser distance sensor and that a single laser scan data is used for the triangular surface.

Accordingly, it is an object of the present invention to provide a 3D laser scanner and an apparatus for calibrating a 2D image camera without the restriction that the triangular board of the present invention is disposed horizontally.

According to an aspect of the present invention, there is provided a calibration apparatus including a camera that captures image information, a laser sensor that senses distance information, a correspondence relation between the image information and the distance information, Wherein the calibration module includes a flat plate member having a polygonal shape including a plurality of vertexes, and a controller for predicting the plurality of vertexes using the distance information, and calculating the predicted vertexes in the image information And a control unit for converting coordinates between the image information and the distance information in correspondence with the vertexes of the plate member included.

As an example related to the present invention, the flat plate member is formed in a triangular or rhombic shape.

In one embodiment of the present invention, the controller selects one of a plurality of planes included in the distance information to predict the vertex, the one plane being located within a predetermined distance among the plurality of planes And includes the most points corresponding to the flat plate members.

In one embodiment of the present invention, the control unit orthogonalizes the points of the one plane so as to be perpendicular to the plane, and predicts the vertices of the flat member using the orthogonal points.

As an example related to the present invention, the controller detects the equations for a plurality of oblique planes using the orthographic points, and calculates the vertices through the calculation of the plurality of equations.

As an example related to the present invention, the controller compares the length between the vertexes and the edge of the flat plate member to determine whether the difference between the lengths is within an error range.

As an example related to the present invention, the controller calculates a plurality of corresponding points using first position data of the image coordinate axis calculated in the image information and second position data of the coordinate axis calculated by the predicted vertices do.

In one embodiment of the present invention, the control unit determines whether or not the corresponding point is calculated to be equal to or greater than a predetermined reference number, and receives the distance information and the image information again when the corresponding point is less than the reference number.

As an example related to the present invention, the controller calculates a transformation matrix between the image information and the distance information using the first and second position data, and performs coordinate transformation using the transformation matrix.

According to another aspect of the present invention, there is provided a calibration method for performing coordinate conversion between a camera for photographing image information and a laser sensor for sensing distance information, A step of photographing a flat plate member, a step of predicting a plurality of vertices from the distance information of the flat plate member, a step of generating at least one corresponding pair using vertexes included in the image information of the predicted vertex and the flat plate member Generating a transform matrix using the corresponding pair, and performing coordinate transform using the transform matrix.

As an example related to the present invention, the step of predicting the plurality of vertices may include detecting one plane from the distance information. The method comprising the steps of: orthogonalizing points included in the distance information on the detected plane; and calculating a plurality of oblique planes using the orthogonal points.

In one embodiment of the present invention, the method further comprises determining whether the corresponding pair is greater than or equal to a preset reference number, and when the corresponding pair is less than the reference number, receiving the image information and distance information from the camera and the laser sensor again do.

According to the present invention having the above-described structure, it is possible to easily obtain the data for the conversion matrix of the position of the camera and the laser sensor for photographing the flat plate member and the flat plate member using the shape of the flat plate member itself.

Accordingly, it is possible to minimize the error according to the position of the flat plate member and calibrate the laser scanner and the camera more easily.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A is a conceptual diagram showing a calibration system according to an embodiment of the present invention. FIG.
1B is a configuration diagram of the calibration system of FIG. 1A;
2 is a conceptual diagram showing a calibration apparatus according to an embodiment of the present invention;
3 is a flow chart for explaining a calibration method according to the present invention.
4 is a flow chart of a method for predicting a triangle vertex point for a corresponding pair in a 3D point cloud;
5A and 5B are conceptual diagrams for explaining a method of calculating aligned points.
Figure 6 is a conceptual diagram of a method for detecting edges and top vertices of a triangle in an aligned 3D point cloud.
Fig. 7 is a conceptual diagram of three vertices of a triangle board in a 2D point cloud and a 2D image used in a corresponding pair; Fig.

Hereinafter, a calibration apparatus, a calibration system, and a calibration method of a camera and a laser sensor according to the present invention will be described in detail with reference to the drawings. In the present specification, the same or similar reference numerals are given to different embodiments in the same or similar configurations. As used herein, the singular forms "a", "an" and "the" include plural referents unless the context clearly dictates otherwise.

FIG. 1A is a conceptual diagram showing a calibration system according to an embodiment of the present invention, and FIG. 1B is a structural diagram of the calibration system in FIG. 1A. 2 is a conceptual diagram illustrating a calibration apparatus according to an embodiment of the present invention.

The calibration system according to the present invention includes a laser scanner 11, a camera 12, a control module S and a flat plate member 13. The laser scanner 11 comprises a plurality of apparatuses, and the plurality of apparatuses are arranged with respect to one axis. The positions of the laser scanner 11 and the camera 12 arranged are fixed at predetermined positions as shown in the figure. The laser scanner 11 and the camera 12 are fixed so that data by the laser scanner 11 and the camera 12 are acquired on the flat plate member 13. The laser scanner 11 and the kemalas 12 can be mounted on a mobile robot body (not shown) while maintaining a relatively fixed distance.

The laser scanner 11 may be implemented in the form of a 3D laser scanner that senses distance information of the objects to be sensed, and receives the laser reflected from the object by scanning the laser beam toward the front. However, the present invention is not limited thereto. The laser sensor 10 may be a 2D laser scanner driven to generate a 3D point cloud, a frequency modulated continuous wave (FMCW) radar capable of obtaining distance information of an object, A 3D laser scanner, and the like.

The camera 12 may take the form of a CCD (Charge Coupled-Device) camera, a stereo camera in which a plurality of cameras are fixed to a mount, and the like.

The controller module S receives data from the laser scanner 11 and the camera 12 and performs calibration. The control module S may be mounted on the mobile robot body (not shown).

The data received from the array laser scanner 11 to the control unit S is made up of a 3D point cloud 30 and the 3D point cloud 30 measures the distance from the laser light receiving sensor to the surface of the object reflected by the laser sensor Means a set of points represented by a 3D coordinate system with respect to an origin or any arbitrary origin in space. However, the distance data of the present invention is not limited to the data of the 3D laser scanner 11 but may be data of a general 3D distance sensor such as a depth camera using a light pattern size or a light depth measuring camera .

The data 20 received from the camera 12 to the control unit S is 2D image information, and a plurality of cameras may be used as shown in FIG. 1A. The 2D image information 20 may be 2D images capable of expressing the geometric information of the flat plate member 13, such as a monochrome image, rather than color image information.

The flat plate member 13 is disposed within the sensing range (measurement range) of the laser sensor 11 and within the imaging range of the camera 12. [ The flat plate member 13 is disposed at a spatial position that can be included in the 3D point cloud 30 generated by the array laser scanner 11 and the 2D image 20 generated by the camera 12, respectively. The flat plate member 13 may have a shape in which the corners are straight and the opposing hypotenuses form an acute angle. For example, triangular or rhombic.

However, when the data scanning directions of the laser sensor 11 are parallel to each other, the flat plate member 13 is limited in that two or more corners of the flat plate member 13 are arranged parallel to the scanning direction.

The 3D point cloud 30 and the 2D image 20 generated in the arrangement of one of the flat plate members are defined as a pair of data. The pair of data differ in the number of corresponding pairs detected depending on the type of plate member used at the time of photographing and the number. The corresponding pair may be equal to the sum of the number of vertices of the entire detected flat plate member.

For example, if the flat plate member 13 has a triangular shape, three corresponding pairs may be formed in the data pair. Eight corresponding pairs can be formed on the data using two rhombic planar members. For convenience of explanation, it is assumed that the flat plate member has a triangular shape.

FIG. 3 is a flow chart for explaining a calibration method according to the present invention, and FIG. 4 is a flowchart of a method of predicting a triangle vertex used in a corresponding pair in a 3D point cloud. 5A and 5B are conceptual diagrams for explaining a method of calculating aligned points.

First, the control module S receives the 3D point cloud (S10). The control module S detects a plurality of points for one flat plate (triangular board) in the received 3D point cloud 30. For example, a point for one flat plate member in a 3D point cloud 30 containing data for two flat plate members may be detected.

In addition, data for the flat plate (triangular board) area is arranged (S30). The noise generated by the laser sensor 12 can be reduced by aligning the data on the flat plate member region.

4 and 5A, a plane corresponding to the flat plate (triangular board) is detected (S21). That is, the control module S detects a plane corresponding to the received flat plate member The corresponding plane P is detected.

For example, the control module S selects any three of the points corresponding to the plate member to detect the plane P. The controller module S repeatedly calculates to detect a plane having a maximum number of points corresponding to the flat plate members within a certain distance from the plane including the three points.

The controller module S can obtain points 40 aligned by orthogonal projection so that the points 30 corresponding to the flat plate member are perpendicular to the plane on the detected plane member plane P (S22). However, if the position of the light emitting point of each point can be estimated, orthogonal projection can be shown so as to be parallel to the laser emitting portion.

5B, the controller module S is mounted on a plane P detected by a portion 30a, 30b, 30c, or 30d of 3D points corresponding to the flat member (triangular board) And calculates aligned points 40a, 40b, 40c, and 40d.

The control module S predicts the vertex of the flat member (triangular board) (S30). FIG. 6 is a conceptual diagram of a method for detecting a corner and an upper vertex of a triangle in an aligned 3D point cloud, and a method for predicting a vertex will be described in more detail with reference to FIG.

The controller module S detects a 3D linear equation corresponding to the left oblique plane of the flat member among the aligned points 40a, 40b, 40c, and 40d (S31), and detects a 3D linear equation corresponding to the right oblique plane of the flat member The 3D linear equation is detected (S32).

In order to detect the 3D linear equation, the points 41 corresponding to the left oblique plane are detected and the points 42 corresponding to the right oblique plane are detected.

For example, if the laser sensor 11 is composed of laser sensor devices that scan in the same direction parallel to each other, it is preferable that each of the aligned points 40a, 40b, 40c, The points corresponding to the left oblique plane can be detected by detecting the points corresponding to the leftmost oblique plane. In addition, it is possible to detect points corresponding to the right oblique plane by detecting the rightmost points in each sensor scan line among the aligned points.

The 3D linear equation 43 corresponding to the left oblique plane of the triangular board selects any two points from the detected points 41 corresponding to the detected left oblique plane, It is possible to calculate a 3D linear equation that maximizes the number of points 41 corresponding to the left oblique plane in the distance (S31). The 3D linear equation 44 corresponding to the right oblique plane of the triangular board can be calculated from the points 42 corresponding to the detected right oblique plane in the calculation method S31.

Next, the controller module S calculates the upper vertex of the triangular board (S33). The vertex can be calculated by calculating the intersection point of the 3D linear equation 43 corresponding to the left oblique plane of the triangular board and the 3D linear equation 44 corresponding to the right oblique plane of the triangular board.

Since the left 3D linear equation 43 and the right 3D linear equation 44 are the straight lines obtained by the subsets 41 and 42 in the points 40 existing on the same plane P respectively, ). Thus, if the two straight lines 43 and 44 are not parallel, the intersection point 51 is on the plane P.

Since the left 3D linear equation 43 includes the position information of the left corner of the triangular board and the right 3D linear equation 44 includes the position information of the left corner of the triangular board, Corresponds to the 3D position information of the upper vertex of the triangular board.

The controller module S calculates the position information of the left lower vertex 52 and the position information of the lower right vertex 53 of the triangular board (S34). For example, it is assumed that a 3D segment is a subset of the left 3D linear equation (43), and one end point is an intersection (51) of the two straight lines. At this time, the distance between the upper vertex of the triangular board and the lower vertex of the left corner is equal to the length of the left corner. Therefore, it is assumed that the length of the line segment is equal to the length of the left corner of the triangle board, and the position information of the end point 52 of the line segment corresponding to the lower left corner point is calculated. The position information of the right lower corner point 53 is also calculated by using the same calculation method, using the intersection 51 of the two straight lines and the length of the right edge of the triangular board.

The controller module S determines suitability of the position information of the vertexes 51, 52, and 53 (S35). For example, the distance between the lower left vertex 52 and the lower right vertex 53 of the detected triangular board is the length of the lower edge of the calculated triangular board. A condition for detecting the plane P and the linear equations 43 and 44 in the calculation steps S20 and S30 when an error equal to or greater than the threshold value is generated by calculating the difference in size from the bottom edge of the actual triangular board, Can be repeated until the error is less than or equal to the threshold value.

Meanwhile, the control module S receives the 2D image 20 (S20). The reckoning module S detects position data of a vertex of the flat plate member in the received 2D image (S50). For example, when the triangular board is used, 2D position data of three vertexes can be detected.

The control module S forms a corresponding pair of two vertexes (S60). Fig. 7 is a conceptual diagram of three vertices of a triangle board in a 2D image cloud and a 2D image used in a corresponding pair. Referring to Figs. 3 and 7, a method of forming a corresponding pair will be described.

When the flat plate member 13 is a triangular board, the three vertexes of the triangular board are divided into first position data (21, 22, 23) for two 2D coordinate axes calculated in the 2D image, and 3D And second position data 51, 52, 53 for three image coordinate axes.

The upper vertex of the triangle board has the position information 21 of the two coordinate axes of the 2D position data and the position information 51 of the three coordinate axes of the 3D point cloud.

The controller module S determines whether the number of the corresponding pairs formed is sufficient for a reference value capable of calculating the transformation matrix (S70).

The transformation matrix means a matrix for transforming the coordinate values of the 3D point cloud into the 2D image coordinate values. If the transformation matrix is a matrix of size 3X4, the transformation matrix can be computed when the number of corresponding pairs is 12 or more.

If the corresponding pair is not sufficient, the control module S receives the point cloud generated through the 3D parallel laser scanner again (S10) and receives the 2D camera image again (S40). That is, the upper steps S10 to S50 are performed again.

The control module S generates a transformation matrix from the corresponding pairs (S60). The controller module S may obtain an inverse transformation matrix for transforming a coordinate value of the 3D point cloud into a 2D image coordinate value and a 2D image coordinate value corresponding to an inverse matrix of the transformation matrix into a coordinate value of the 3D point cloud have. The transformation matrix includes a coordinate transformation matrix between the internal variables of the camera, external variables, and the camera and the laser sensor

Thereby enabling direct coordinate conversion between image information and distance information.

The controller module S performs calibration of 2D image data and 3D point cloud data using the transformation matrix and the inverse transformation matrix (S90).

And convert the coordinate axis of the 3D point cloud 30 to the coordinate axis of the 2D image 20 using the generated transformation matrix. In addition, it is possible to convert from the coordinate axis of the 2D image 20 to the coordinate axis of the 3D point cloud 30 using the inverse transformation matrix computed in the step S80.

The calibration device and the calibration method of the camera and the laser sensor are not limited to the configuration and the method of the embodiments described above, but the embodiments may be modified such that all or some of the embodiments are selectively combined .

Claims (12)

A camera for photographing image information;
A laser sensor for sensing distance information;
And a calibration module for calculating the correspondence between the image information and the distance information and performing calibration of the camera and the laser sensor,
Wherein the calibration module comprises:
A flat plate member made of a polygon including a plurality of vertexes; And
And a control unit for predicting the plurality of vertexes using the distance information and converting the coordinates between the image information and the distance information by associating the predicted vertices with the vertices of the flat plate member included in the image information, . The method of claim 1,
Wherein the flat plate member is in the form of a triangle or a rhomboid. 2. The apparatus of claim 1, wherein the controller selects one of a plurality of planes included in the fraud distance information to predict the vertex,
Wherein the one plane is located within a predetermined distance of the plurality of planes and includes the most points corresponding to the flat plate members. The method of claim 3,
Wherein the control unit corrects the points of the one plane to be perpendicular to the plane and predicts the vertex of the flat member using the orthogonal points. 5. The method of claim 4,
Wherein the control unit detects the equations for a plurality of oblique planes using the orthogonal projection points and calculates the vertexes through calculation of the plurality of equations. The method of claim 3,
Wherein the controller compares the length between the vertexes with the length of the edge of the flat plate member to determine whether the difference between the length of the vertexes and the length of the edge falls within an error range. The method according to claim 1,
Wherein the controller calculates the plurality of corresponding points using the first position data of the image coordinate axis calculated in the image information and the second position data of the coordinate axis calculated by the predicted vertices. 8. The apparatus of claim 7, wherein the controller determines whether the corresponding point is calculated to be equal to or greater than a predetermined reference number,
And the distance information and the image information are received again when the corresponding point is less than the reference number. 9. The apparatus according to claim 8, wherein the controller calculates a transformation matrix between the image information and the distance information using the first and second position data, and performs coordinate transformation using the transformation matrix . A calibration method for performing coordinate conversion between a camera for capturing image information and a laser sensor for sensing distance information,
Photographing a flat plate member using the camera and the laser sensor;
Estimating a plurality of vertexes from the distance information of the flat plate member;
Generating at least one corresponding pair using the vertex included in the image information of the predicted vertex and the flat plate member;
Generating a transform matrix using the corresponding pair; And
And performing a coordinate transformation using the transformation matrix. 11. The method of claim 10, wherein the step of predicting the plurality of vertexes comprises:
Detecting one plane from the distance information;
Orthogonalizing points included in the distance information on the detected plane; And
And calculating a plurality of oblique planes using the orthogonal projection points. 11. The method of claim 10,
Further comprising determining whether the corresponding pair is greater than or equal to a predetermined reference number,
And when the corresponding pair is less than the reference number, receiving the image information and the distance information from the camera and the laser sensor, respectively.

Images