KR20200084552A - Apparatus and method for calibrating camera and lidar sensor of vehicle - Google Patents

Abstract

The present invention relates to an apparatus for calibrating a camera and a LIDAR sensor for a vehicle, comprising: a camera unit which is an image sensor for photographing an image in front of the vehicle; a LIDAR sensor unit which measures a distance using light and detects an object; a feature point detection unit which detects at least one feature point designated in a rigid body image of the vehicle visible in the field of view (FOV) of the camera unit and the LIDAR sensor unit, and provides location information thereof; a feature point information storage unit for storing feature point information set as a reference during final calibration for each sensor; and a control unit which detects a sensor in which the detected feature point information of each sensor has been changed compared to the feature points respectively stored in the feature point information storage unit, and performs recalibration on the corresponding sensor of which the feature point information has been changed compared to the previous information.

Description

A device and method for calibrating a vehicle camera and a lidar sensor{APPARATUS AND METHOD FOR CALIBRATING CAMERA AND LIDAR SENSOR OF VEHICLE}

The present invention relates to a device and method for calibrating a vehicle camera and a lidar sensor, and more specifically, without using a special marker or jig for calibrating the vehicle camera and a lidar sensor, using a rigid body of the vehicle It relates to a calibration device and method for a vehicle camera and a lidar sensor, so that it can automatically perform the calibration.

In general, various sensors for supporting a driver are installed in a vehicle, and there are typically camera sensors (eg, single camera sensors, stereo camera sensors).

In particular, a stereo camera sensor has a feature that works in the same way as a person feels a sense of stereoscopic and distance when looking at an object. That is, when the left/right camera of the stereo camera looks at a specific object, such as the binocular vision of a person, a parallax occurs due to a spatial difference, and the parallax makes it possible to feel a distance to the object or a three-dimensional effect of the object.

In this way, the camera sensor mounted on the vehicle (or another sensor additionally mounted on the vehicle) needs to perform calibration and rectification. For example, calibration in the stereo camera is a process of extracting the camera parameters of the stereo camera, and rectification accurately aligns the left and right camera images using these extracted camera parameters, thereby reducing the computation amount and accuracy when calculating the stereo parallax. It is a process of heightening.

The camera calibration as described above is a process that must be performed for distortion correction of a camera and rectification of a stereo image, and conventionally, a person performs calibration using a check board (or marker).

However, in the conventional calibration method using a check board, the size of the check board is too large and space for calibration is required, and the user cannot perform the calibration alone, and the calibration cannot be automatically performed at a desired point in time. There is a problem. In other words, the calibration method of the camera mounted on the conventional vehicle can be performed for the first time when the vehicle is shipped, and if the camera sensor is distorted due to vibration shock of the vehicle, it must be received at the A/S center or factory to perform the calibration. There is a problem that the normal function of the camera sensor cannot be used until then.

Moreover, in recent vehicles, not only the camera sensor, but also other sensors (eg, Lidar sensors) for supplementing the camera sensor are being installed (attached), thereby complementing the camera sensor and the camera sensor. Calibration must be performed in synchronization with other sensors (eg, Lidar sensors) (ie, calibration should be performed using the same criteria).

As described above, in order to perform calibration in synchronization with a conventional method for a plurality of sensors (for example, a camera or a lidar), the plurality of sensors (because the calibration must be performed after being received at an A/S center or a factory). Example: There is a problem that the functions of the camera, lidar) cannot be used normally.

Therefore, there is a method to enable multiple sensors (e.g., cameras, lidars) to synchronize and automatically perform calibration at a desired point in time (i.e., multiple sensors perform calibration using the same criteria). This is a necessary situation.

Background art of the present invention is disclosed in Republic of Korea Patent Registration No. 10-1748398 (2017.06.12. registration, a calibration device and method for a camera mounted on a vehicle).

According to one aspect of the present invention, the present invention was created to solve the above problems, without using a special marker or jig for the calibration of the vehicle camera and the lidar sensor, the rigid body of the vehicle (rigid body) An object of the present invention is to provide an apparatus and method for calibrating a vehicle camera and a lidar sensor, which enable automatic calibration.

A calibration device for a vehicle camera and a lidar sensor according to an aspect of the present invention includes a camera unit that is an image sensor for capturing an image in front of a vehicle; A lidar sensor unit, which is a sensor for measuring distance and sensing an object using light; A feature point detector configured to detect at least one feature point designated by a rigid body image of a vehicle viewed according to a field of view (FOV) of the camera unit and the lidar sensor unit and position information thereof; A feature point information storage unit for storing feature point information set as a reference for the last calibration for each sensor; And a control unit that detects a sensor whose feature point information of each of the detected sensors is changed compared to a feature point stored in the feature point information storage unit, and re-calibrates the corresponding sensor where the previous feature point information is detected as changed. It is characterized by.

In the present invention, the feature point information is characterized by including the type and location information of the feature point.

In the present invention, the feature point information is characterized in that it is the information of the feature point set as a reference at the time of the last calibration.

In the present invention, when the calibration feature point information previously stored in the feature point information storage unit is changed, the feature point information correction unit to correct the new feature point information detected by the feature point detection unit.

In the present invention, when the feature point information is changed compared to the previous, it further comprises a parameter correction for calculating a correction parameter in response to the difference of the newly changed feature point information from the previously stored feature point information.

In the present invention, the correction parameter is characterized in that it is a parameter for calibration to be performed again according to the change of the feature point information in the previous calibration state.

According to another aspect of the present invention, a method of calibrating a vehicle camera and a lidar sensor includes: a control unit receiving a camera image and a lidar image; The controller detecting feature points in the camera image and the lidar image; Checking, by each sensor, whether the position of the detected feature point is the same as the position of the previously stored feature point; If the location of the feature point is not the same as the location of the previously stored feature point, the control unit compares the detected feature point with pre-stored feature point information to check whether a difference occurs over a specified error range; If the position of the detected feature point differs from the position of the previously stored feature point and an error range, the control unit corrects previously stored feature point information with the detected new feature point information; Calculating, by the control unit, a correction parameter corresponding to the difference between the previously stored feature point information and the detected new feature point information; And re-calibrating the corresponding sensor whose feature point information is changed by using the calculated correction parameter by the control unit.

In the present invention, the camera image, it means that the image taken by the camera unit, the lidar image is characterized in that it means the image sensed by the lidar sensor unit.

In the present invention, the feature point is characterized in that it is specified in the rigid body (rigid body) of the vehicle.

In the present invention, it is characterized in that the type and location information of the feature point can be designated at the first calibration.

In the present invention, the location information of the feature point is characterized in that the coordinate information in each sensor image.

In the present invention, the location information of the feature points, each sensor is used as a reference for the same calibration, the location information of the feature points stored for calibration in each sensor corresponding to the position where each sensor is installed is set It is characterized by being.

In the present invention, when each sensor checks whether the detected location of the feature point is the same as the previously stored feature point location, when the feature point location is the same as the previously stored feature point location, the control unit performs And maintaining the calibration state for each sensor.

In the present invention, as a result of comparing the detected feature points with pre-stored feature point information and checking whether a difference over a specified error range occurs, an error range error difference occurs even if the detected feature point location is different from the previously stored feature point location. If not, the control unit is characterized in that it further comprises the step of maintaining each of the previously performed calibration state for each sensor.

In the present invention, the correction parameter is characterized in that it means a parameter for calibration to be performed again according to the change of the feature point information in the previous calibration state.

According to an aspect of the present invention, the present invention enables calibration to be performed automatically using a rigid body of a vehicle without using a special marker or jig for calibration of a vehicle camera and a lidar sensor.

According to another aspect of the present invention, the present invention makes it possible to automatically perform calibration at a desired point in time using the same criteria for a vehicle camera and a lidar sensor.

1 is an exemplary view showing a schematic configuration of a calibration device for a vehicle camera and a lidar sensor according to an embodiment of the present invention.
2 is a flowchart illustrating a method of calibrating a vehicle camera and a lidar sensor according to an embodiment of the present invention.
FIG. 3 is an exemplary view showing an image captured by a vehicle camera and an image sensed using a lidar sensor in FIG. 1.
4 is an exemplary view showing an emblem that can be used as a feature point in the present embodiment in FIG. 1.
5 is an exemplary view shown to schematically explain a method capable of performing calibration on a camera and a lidar sensor and problems that may occur at this time.

Hereinafter, an embodiment of a calibration apparatus and method for a vehicle camera and a lidar sensor according to the present invention will be described with reference to the accompanying drawings.

In this process, the thickness of the lines or the size of components shown in the drawings may be exaggerated for clarity and convenience. In addition, terms to be described later are terms defined in consideration of functions in the present invention, which may vary according to a user's or operator's intention or practice. Therefore, definitions of these terms should be made based on the contents throughout this specification.

1 is an exemplary view showing a schematic configuration of a calibration device for a vehicle camera and a lidar sensor according to an embodiment of the present invention.

As illustrated in FIG. 1, a calibration device for a vehicle camera and a lidar sensor according to the present embodiment includes a camera unit 110, a lidar sensor unit 120, a feature point detection unit 130, a control unit 140, and a feature point It includes an information storage unit 150, a feature point information correction unit 160, and a parameter correction unit 170.

The camera unit 110 photographs and outputs an image in front of the vehicle.

The camera unit 100 may include a monocular camera or a stereo camera.

In the camera unit 100, reference parameters may be respectively set. For example, when the camera unit 100 is a stereo camera, parameters may be set for each of the left camera and the right camera.

The LIDAR (LIDAR, Light Detection and Ranging) sensor unit 120 is a sensor that measures distance and detects an object by using light, and has a similar principle to that of a radar.

However, the radar launches an electromagnetic wave to the outside to check the distance and direction with the electromagnetic wave that is re-received, but the difference is that the rider fires a pulsed laser. That is, since a laser having a short wavelength is used, precision and resolution are high and three-dimensional grasping is possible according to an object.

For example, the lidar sensor is mounted on a vehicle bumper or an upper part of the vehicle to sense an object or structure by sensing the front and rear of the vehicle. For reference, in FIG. 3(b), the upper part is an exemplary view showing an image photographed by a vehicle camera, and in FIG. 3(b), a lower part is a field of view (FOV) of a lidar sensor mounted on the upper part of the vehicle. It is an example.

In addition, FIG. 3(a) is an exemplary view shown to exemplarily explain the positions of the camera and the lidar sensor mounted on the vehicle.

The feature point detecting unit 130 is at least one or more designated in a rigid body image (eg, a camera image, a lidar sensor image) of a vehicle viewed according to the FOV of the camera unit 110 and the lidar sensor unit 120. Feature points (for example, the emblem of FIG. 4) and their location information (ie, coordinate information in the image) are detected.

The feature point information stores a feature point set as a reference (i.e., a reference for calibration) at the time of the last calibration (e.g., factory calibration).

The feature point information storage unit 150 stores feature point information (eg, type and location information of the feature point) set as a reference when the last calibration is performed.

The feature point information corrector 160, when the previously stored calibration feature point information (eg, type and location information of the feature point) is changed, new feature point information detected by the feature point detecting unit 130 (eg, of the feature point) Type and location information).

When the feature point information is changed compared to the previous, the parameter correction unit 170 calculates a corrected parameter in response to a difference between the newly changed feature point information from the previously stored feature point information. For example, the parameter is a parameter for a new calibration to be performed according to the change of the feature point information in the previous calibration state.

Here, since the feature point is set as a rigid body, the fact that the feature point information is changed is because the body itself is hardly deformed, so the camera or lidar sensor is actually distorted due to vibration or shock of the vehicle. it means.

Therefore, when a sensor (ie, a camera or a lidar sensor) whose feature point information has been changed compared to the existing one is detected, it means that the posture of the corresponding sensor (ie, a camera or a lidar sensor) is changed, so the control unit 140 displays the feature point information. It is only necessary to re-calibrate the corresponding sensor (ie, camera or lidar sensor) detected as having been changed.

As described above, the control unit 140 detects a sensor whose feature point information has been changed, and re-calibrates only the corresponding sensor (eg, camera), so that a plurality of sensors calibrated using the same reference (eg, camera, lidar sensor) ), it is not necessary to perform calibration on other sensors (eg, lidar sensors) except for the corresponding sensor (eg, camera) that performs the calibration.

For reference, in order to calculate an extrinsic parameter between a lidar and a camera, as shown in FIG. 5, an arbitrary fitting method is used. That is, if the same data is present in both the camera and the lidar sensor, calibration can be performed based on this.

Here, the same data mostly uses a plane, and as shown in FIG. 5, R/T (Rotation, R/T) of the camera and the lidar using a certain known marker (= chessboard, checkboard). This is the way to calculate the matrix. However, even if the R/T matrix is calculated by the above-described method, since the standard itself may change, some problems may occur in the future.

However, this embodiment prevents the reference itself from being changed by using the rigid body of the vehicle included in the field of view (FOV) of the camera and the lidar as a feature point, and re-calibration is performed accordingly. By accurately detecting the sensor to be calibrated, only the relevant sensor is calibrated, but by calculating a correction parameter corresponding to the difference in the feature point, and recalibrating using the correction parameter, it is automatically performed at a desired time point. Allow calibration to be performed.

2 is a flowchart illustrating a method of calibrating a vehicle camera and a lidar sensor according to an embodiment of the present invention.

Referring to FIG. 2, the controller 140 receives camera images and lidar images (S101). Here, the camera image refers to an image captured by the camera unit 110 (refer to the upper image in FIG. 3B ), and the lidar image refers to an image sensed by the lidar sensor unit 120. (See the lower image in Fig. 3 (b)).

In addition, the control unit 140 detects feature points in the camera image and the lidar image (S102).

Here, the feature point is specified in the rigid body of the vehicle, and the type and location information (ie, coordinate information in the image) of the feature point may be previously specified during the initial calibration. In addition, the feature point itself uses the same plurality of sensors (e.g., camera and lidar sensor) as standard, but calibration is performed at each sensor because the locations where the respective sensors (e.g., camera and lidar sensor) are installed are different. Location information of the feature points stored in order may be different.

Accordingly, the control unit 140 checks each sensor whether the location of the detected feature point is the same as the location of the previously stored feature point (S103).

As a result of the check (S103), when the location of the feature point is the same as the location of the previously stored feature point (YES in S103), the control unit 140 maintains the previously performed calibration state for each sensor (S108).

However, as a result of the check (S103), if the location of the feature point is not the same as the location of the previously stored feature point (No in S103), the control unit 140 compares the detected feature point with pre-stored feature point information and a specified error range It is checked whether an abnormality difference occurs (S104).

As a result of the check (S104), if the difference of the error range or more does not occur even if the detected feature point is different from the previously stored feature point (No in S104), the controller 140 detects the previously performed calibration state. Each star is maintained (S108).

However, as a result of the check (S104), when a difference between the position of the detected feature point and an error range is greater than the previously stored feature point (YES in S104), the control unit 140 previously displays the feature point information storage unit 150. The stored feature point information is corrected (or updated) with the detected new feature point information (S105).

In addition, the control unit 140 calculates a correction parameter corresponding to the difference between the feature point information stored in the feature point information storage unit 150 and the detected new feature point information (S106).

For example, the correction parameter means a parameter for a new calibration (that is, re-calibration) to be performed according to the change of the feature point information in the previous calibration state.

In addition, the control unit 140 performs calibration of the corresponding sensor (ie, a sensor having changed feature point information, a camera or a lidar sensor) using the calculated correction parameter (S107).

As described above, this embodiment does not use a special marker or jig for calibration of a vehicle camera and a lidar sensor, and automatically performs calibration using a rigid body of a vehicle that does not change unless the vehicle is damaged. By doing so, there is an effect of determining a sensor that requires calibration and automatically performing calibration at a desired time point using a correction parameter corresponding to the changed feature point information.

The present invention has been described above with reference to the embodiment shown in the drawings, but this is only exemplary, and those skilled in the art to which the art pertains may have various modifications and other equivalent embodiments. You will understand the point. Therefore, the technical protection scope of the present invention should be defined by the claims below. Also, the implementations described herein can be implemented, for example, as methods or processes, devices, software programs, data streams, or signals. Although discussed only in the context of a single form of implementation (eg, discussed only as a method), implementation of the features discussed may also be implemented in other forms (eg, devices or programs). The device can be implemented with suitable hardware, software and firmware. The method can be implemented in an apparatus, such as a processor, generally referring to a processing device, including, for example, a computer, microprocessor, integrated circuit, or programmable logic device. The processor also includes communication devices such as computers, cell phones, portable/personal digital assistants ("PDAs") and other devices that facilitate communication of information between end-users.

110: camera unit 120: lidar sensor unit
130: feature point detection unit 140: control unit
150: feature point information storage unit 160: feature point information correction
170: parameter correction

Claims (15)

A camera unit that is an image sensor for photographing an image in front of a vehicle;
A lidar sensor unit, which is a sensor for measuring distance and sensing an object using light;
A feature point detector configured to detect at least one feature point designated by a rigid body image of a vehicle viewed according to a field of view (FOV) of the camera unit and the lidar sensor unit and position information thereof;
A feature point information storage unit for storing feature point information set as a reference for the last calibration for each sensor; And
It includes a control unit for detecting the sensor has changed the feature point information of each of the detected sensors compared to the feature points stored in each of the feature point information storage unit, and re-calibrates the corresponding sensor detected that the previous feature point information has been changed; A vehicle camera and a lidar sensor calibration device.
The method of claim 1, wherein the feature point information,
A calibration device for a vehicle camera and a lidar sensor, comprising type and location information of a feature point.
The method of claim 1, wherein the feature point information,
A calibration device for a vehicle camera and a lidar sensor, characterized in that it is information on a feature point set as a reference for the last calibration.
According to claim 1,
If the calibration feature point information previously stored in the feature point information storage unit is changed, a vehicle camera and a lidar sensor further comprising a feature point information correction unit that corrects the new feature point information detected by the feature point detection unit. Calibration device.
According to claim 1,
If the feature point information has been changed compared to the previous, the vehicle camera and the lidar sensor further comprises a parameter correction for calculating a correction parameter in response to the difference of the newly changed feature point information from the previously stored feature point information Calibration device.
The method of claim 5, wherein the correction parameter,
A calibration device for a vehicle camera and a lidar sensor, which is a parameter for calibration to be performed again according to the change of the feature point information in a previous calibration state.
The control unit receives a camera image and a lidar image;
The controller detecting feature points in the camera image and the lidar image;
Checking, by each sensor, whether the position of the detected feature point is the same as the position of the previously stored feature point;
If the location of the feature point is not the same as the location of the previously stored feature point, the control unit compares the detected feature point with pre-stored feature point information to check whether a difference occurs over a specified error range;
If the position of the detected feature point differs from the position of the previously stored feature point and an error range, the control unit corrects previously stored feature point information with the detected new feature point information;
Calculating, by the control unit, a correction parameter corresponding to the difference between the previously stored feature point information and the detected new feature point information; And
And re-calibrating the corresponding sensor whose feature point information is changed by using the calculated correction parameter by the control unit.
The method of claim 7,
The camera image means an image captured by the camera unit,
The lidar image is a vehicle camera and a method for calibrating a lidar sensor, characterized in that means the image sensed by the lidar sensor unit.
The method of claim 7, wherein the feature point,
A method for calibrating a vehicle camera and a lidar sensor, characterized in that it is specified in a rigid body of the vehicle.
The method of claim 7,
A method of calibrating a vehicle camera and a lidar sensor, characterized in that the type and location information of the feature point can be designated at the time of initial calibration.
The location information of the feature point,
A method for calibrating a vehicle camera and a lidar sensor, which is coordinate information in each sensor image.
The location information of the feature point,
Each sensor is equally used as a reference for calibration,
A method for calibrating a vehicle camera and a lidar sensor, characterized in that location information of a feature point stored for calibration in each sensor is set in correspondence to a location where each sensor is installed.
The method of claim 7,
As a result of checking for each sensor whether the location of the detected feature point is the same as the location of the previously stored feature point,
If the position of the feature point is the same as the location of the previously stored feature point, the control unit maintains the previously performed calibration state for each sensor; vehicle camera and lidar sensor calibration method further comprising a.
The method of claim 7,
As a result of comparing the detected feature points with pre-stored feature point information and checking whether a difference occurs over a specified error range,
If the position of the detected feature point differs from the position of the previously stored feature point, and no difference occurs above the error range, the control unit maintains the previously performed calibration state for each sensor. How to calibrate the camera and lidar sensor.
The method of claim 7, wherein the correction parameter,
A calibration method of a vehicle camera and a lidar sensor, characterized in that means a parameter for calibration to be performed again according to the change of the feature point information in the previous calibration state.

Images