KR20130041440A - Image processing apparatus and method thereof - Google Patents

Abstract

PURPOSE: An image processing device and a method thereof are provided to easily detect a response point by only sensing an irregularly placed hole during photographing with a 3D TOF(Time of Flight) sensor by using a 2.5D pattern. CONSTITUTION: An image processing device obtains a color image, an amplitude image, and 3D measurement values(310). The image processing device optimizes an implicitness parameter of a 3D TOF sensor(320). The image processing device corrects a radial distortion from the amplitude image(330). The image processing device optimizes a cost function(340). [Reference numerals] (310) Obtaining a color image, a luminosity image, and a 3D measurement values; (320) Optimizing the implicitness parameter of a 3D TOF sensor; (330) Correcting radial distortion; (340) Optimizing a cost function; (AA) Start; (BB) End;

Description

Image processing apparatus and method {IMAGE PROCESSING APPARATUS AND METHOD THEREOF}

TECHNICAL FIELD The art relates to an external calibration apparatus and method for 3D TOF sensors and color cameras.

A three-dimensional time-of-flight (TOF) sensor measures three-dimensional values by measuring the strength and phase of a signal that is emitted from near-infrared rays and calculating a distance to a photographing object. A two-dimensional laser scanner scans a subject in a row, while a three-dimensional TOF sensor is not limited to measuring distances of a specific height by multiple N-arranged LEDs simultaneously shooting near-infrared rays toward the subject. ).

However, current three-dimensional TOF sensors have lower accuracy of measurement values than two-dimensional race scanners and lower resolution than color images used together.

Therefore, by externally calibrating a conventional two-dimensional laser scanner and a color camera or a method of calibrating only a color image, it is impossible to accurately externally calibrate the three-dimensional TOF sensor and the color camera.

There is a need for a technique for performing external calibration, such as a three-dimensional TOF sensor, to compensate for low resolution and inaccurate measurements.

In one aspect, the image processing method is obtained by capturing a 2.5-dimensional pattern with a color camera to obtain a color image, and by taking a 2.5-dimensional pattern with a three-dimensional time of flight (TOF) sensor and the luminance image and three Obtaining dimensional measurements, optimizing the internal parameters of the three-dimensional TOF sensor to minimize the transfer error of the three-dimensional measurements in consideration of the internal parameters of the three-dimensional TOF sensor and the three-dimensional measurements; Correcting a circumferential distortion in the luminance image in consideration of the internal product parameters, a feature point of the luminosity image with circumferential distortion correction, a feature point of the color image, an internal parameter and an external parameter of the color camera, Optimized cost function considering optimized internal and external parameters and depth constraints of 3D TOF sensor .

In another aspect, the image processing method performs initial calibration on the color image and the luminance image to calculate the internal parameter and the external parameter of the color camera, and the internal parameter and the third parameter of the 3D TOF sensor. The method may further include calculating an external parameter.

The optimizing of the cost function may include acquiring a feature point of the luminosity image in which the circumferential distortion is corrected, and acquiring a feature point of the color image.

In another aspect, the image processing method may further include estimating a position and an angle between the color camera and the three-dimensional TOF sensor using an optimized cost function.

The 2.5-dimensional pattern may have a size that allows the near infrared ray of the 3D TOF sensor to pass through, and may be a pattern in which holes having a symmetrical shape are irregularly disposed.

The 3D TOF sensor may measure a depth value of the 2.5D pattern.

The step of optimizing the internal parameter of the three-dimensional TOF sensor is a result of transferring the three-dimensional measurement values to the luminance image using the internal parameter, the luminance image is mapped to the coordinates of the pixel and the three-dimensional measurement values Optimization can be performed using the Levenberg-Marquardt method so that the error between the coordinates of the pixel is minimized.

The step of correcting the radial distortion may be corrected by using the Levenberg-Marquardt method in consideration of the optimized internal parameter of the radial distortion component included in the luminance image.

The depth constraint may be characterized by estimating values located in the 2.5-dimensional pattern plane among 3D measured values measured by the 3D TOF sensor.

The cost function may include the coordinates of the feature points of the luminosity image in which the circumferential distortion is corrected and the points corresponding to the feature points of the luminosity image in which the circumferential distortion is corrected in the 2.5-dimensional pattern are projected onto the luminosity image in which the circumferential distortion is corrected. The difference between the coordinates of the point, the coordinates of the feature point of the color image and the point corresponding to the feature point of the color image in the 2.5-dimensional pattern may include the difference between the coordinates of the point projected to the color image and the depth constraint.

In an aspect, the image processing apparatus may acquire a color image by capturing a 2.5-dimensional pattern with a color camera, and photograph the 2.5-dimensional pattern with a 3D time of flight (TOF) sensor to obtain an luminance image and a 3 A photographing unit obtaining dimension measurements, the first parameter to optimize the internal parameter of the three-dimensional TOF sensor to minimize the transfer error of the three-dimensional measurement in consideration of the internal parameter of the three-dimensional TOF sensor and the three-dimensional measurement values An optimizer, a corrector for correcting radial distortion in the luminance image in consideration of the optimized inner product parameter, a feature point of the luminance image in which the circumferential distortion is corrected, a feature point of the color image, and an inner parameter of the color camera And cost considering external parameter, optimized internal parameter, external parameter and depth constraint of the 3D TOF sensor The optimization includes two parts to optimize the number.

In another aspect, the image processing apparatus performs initial calibration on the color image and the luminance image to calculate the internal parameter and the external parameter of the color camera, and the internal parameter and the internal parameter of the 3D TOF sensor. The apparatus may further include a calculator configured to calculate the external parameter.

The 2.5-dimensional pattern may be a pattern in which a circular hole having a size enough to pass near infrared rays of the 3D TOF sensor is irregularly disposed.

The first optimizer transfers the three-dimensional measurement values to the luminance image using the internal parameter, and as a result, the error between the coordinates of the pixel and the pixel of the luminance image mapped to the three-dimensional measurement values is minimal. Optimization can be performed as much as possible.

The depth constraint may be characterized by estimating values located in the 2.5-dimensional pattern plane among 3D measured values measured by the 3D TOF sensor.

The second optimizer projects the coordinates of the feature points of the luminosity image in which the circumferential distortion is corrected and the points corresponding to the feature points of the luminosity image in which the circumferential distortion is corrected in the 2.5-dimensional pattern to the luminosity image in which the circumferential distortion is corrected. A cost function comprising the difference between the coordinates of the plotted points, the coordinates of the feature points of the color image and the coordinates of the points corresponding to the feature points of the color image in the 2.5-dimensional pattern projected onto the color image and the depth constraint The minimum value of can be calculated.

By using a 2.5-dimensional pattern of irregularly arranged holes that can pass near infrared rays, the corresponding point can be easily detected only by detecting irregularly arranged holes when photographing with a 3D TOF sensor.

More correspondence can be detected by correcting the radial distortion in the luminosity image captured by the three-dimensional TOF sensor using the optimized internal parameter.

By optimizing the cost function including the transfer error of the color image, the transfer error of the luminance image with the circumferential distortion corrected, and the three-dimensional measurement value, the three-dimensional characteristic that the color camera does not provide may be reflected in the calibration and may be incorrect. The measured value can be complemented.

1 is a view showing a two-dimensional pattern and a 2.5-dimensional pattern according to an embodiment of the present invention.
2 is a view illustrating a color camera, a 3D TOF sensor, and a 2.5D pattern in an image processing apparatus according to an embodiment of the present invention.
3 is a flowchart of an image processing method according to an embodiment of the present invention.
4 is a flowchart of an image processing method according to another exemplary embodiment of the present invention.
FIG. 5 is a diagram illustrating a result of being transferred to an image before and after optimization of an internal parameter in an image processing process according to an embodiment of the present invention.
6 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

In order to know where the distance value measured by the 3D TOF sensor corresponds to the image captured by the color camera, accurate calibration should be performed between the 3D TOF sensor and the color camera. In one aspect, the image processing method relates to the above calibration.

This calibration can be applied throughout the industry using three-dimensional TOF sensors and color cameras together. For example, the calibration may be used when distance measurement, autonomous driving, obstacle detection, and object recognition are performed by mounting a 3D TOF sensor and a color camera together in a robot or a vehicle.

In addition, the calibration may be used to obtain a 3D model and to produce 3D content using a 3D TOF sensor and a color camera. That is, it can be directly applied when using a 3D TOF sensor and a color camera when producing content in related fields including 3D realistic broadcasting, 3D virtual experience for home shopping and internet shopping, and 3D movie production.

1 is a view showing a two-dimensional pattern and a 2.5-dimensional pattern according to an embodiment of the present invention.

Referring to FIG. 1, the color image 110 is an image captured by a two-dimensional checker board with a color camera. The luminance image 120 is an image of a 2D checker board taken with a 3D TOF sensor.

For external calibration between the 3D TOF sensor and the color camera, it is important to accurately determine the point of knowing the position of the 3D image and the corresponding point in the captured image. This is because a parameter of a 3D TOF sensor or a color camera that minimizes a reprojection error may be calculated by using a correspondence between a location and a corresponding point.

When calibrating a color camera, a checkerboard is mainly photographed and a corresponding point on the color image 110 is calculated through corner detection. However, since the brightness image 120 captured by the 3D TOF sensor has low resolution and high noise, it is difficult to accurately calculate a corresponding point through corner detection.

The color image 130 is an image obtained by photographing a 2.5-dimensional pattern with a color camera. The luminance image 140 is an image obtained by capturing a 2.5-dimensional pattern with a 3D TOF sensor.

Here, the 2.5-dimensional pattern means a pattern in which holes are irregularly drilled in the plane. At this time, the hole has a size enough to allow the near infrared ray output from the three-dimensional TOF sensor to pass through. In addition, since the hole appears in the image in the form of a circle, the position of the center of the hole may not be greatly affected by the isotropic noise. Therefore, the 2.5-dimensional pattern may be easily used to accurately calculate the corresponding point in the luminance image 140.

2.5-dimensional pattern is easy to find the center of the hole even in the low-resolution and intensive luminosity image 140. In addition, due to the irregular pattern, only a part of the 2.5-dimensional pattern plane is visible or even when the top and bottom sides are changed, a corresponding point between the luminance image and the 2.5-dimensional pattern may be automatically detected.

2 is a view illustrating a color camera, a 3D TOF sensor, and a 2.5D pattern in an image processing apparatus according to an embodiment of the present invention.

Referring to FIG. 2, the color camera 210 captures a 2.5D pattern 230 to obtain a color image 211, and the 3D TOF sensor 220 captures a 2.5D pattern 230 to measure 3D. Values 221 may be obtained.

Extrinsic calibration refers to the method of determining the mutual position and angle between two or more heterogeneous / homogeneous sensors. Through external calibration, an image may be processed as if a plurality of sensors photographed a photographing target at the same location.

External calibration should be made between the color camera 210 and the three-dimensional TOF sensor 220.

By photographing the 2.5-dimensional pattern 230 through the color camera 210, the feature point in the color image 211 can be easily detected. An initial calibration of the color camera 210 may be performed in consideration of the relationship between the feature point of the 2.5D pattern 230 and the corresponding point corresponding to the feature point in the color image 211.

By capturing the 2.5D pattern 230 through the 3D TOF sensor 220, the feature point may be easily detected in the luminance image. An initial calibration of the 3D TOF sensor 220 may be performed in consideration of the relationship between the feature point of the 2.5D pattern 230 and the corresponding point corresponding to the feature point in the luminance image. The specific method of performing the calibration may be performed through a camera calibration method proposed by Z. Zhang.

3 is a flowchart of an image processing method according to an embodiment of the present invention.

In operation 310, the image processing apparatus acquires a color image, an amplitude image, and three-dimensional measurement values. The image processing apparatus may acquire a color image by photographing the 2.5D pattern with a color camera. The color image refers to an image containing color components of a 2.5-dimensional pattern.

The image processing apparatus may acquire the luminance image and the 3D measurement values by photographing the 2.5D pattern with a 3D time of flight (TOF) sensor. Luminance images are measured through the intensity of the reflected and refracted rays from the 2.5-dimensional pattern, allowing for the identification of 2.5-dimensional patterns, and the 3-dimensional measurements indicate how far the 2.5-dimensional pattern is from the 3D TOF sensor, ie perspective. Can be. The 3D measurement value refers to a 3D coordinate value measured for each pixel of the luminance image.

The image processing apparatus may calculate an initial value of the internal parameter and an external parameter of the color camera by performing initial calibration on the color image. The image processing apparatus may calculate an initial value of an internal parameter and an external parameter of the 3D TOF sensor by performing initial calibration on the luminance image.

The internal parameter represents the focal length of the camera and the center of the image taken by the camera, and the external parameter may indicate at which position and angle the 2.5-dimensional pattern plane is from the camera.

Initial calibration can be done using Qilong Zhang and Robert Pless, IROS 2004.

The relationship between the color image or the luminance image and the 2.5 pattern to be photographed may be expressed by the following equation.

s is any scale factor, [u, v] is the point where the point on the 2.5-dimensional pattern is projected on the image-color image or luminosity image, and in [u, v, 1] 1 is the extension vector added for matrix calculation (augmented vector), [u, v, 1] ^T is the transposed matrix of [u, v, 1], K is an internal parameter, [r ₁ , r ₂ , t] is an external parameter and R = [ r ₁ , r ₂ , r ₃ ] is the rotation matrix, t is the translation vector, [X, Y] is the point on the 2.5-dimensional pattern, e.g. the position of the center of each hole, 1 in X, Y, 1] is an extension vector added for matrix calculation, [X, Y, 1] ^T is a transpose of [X, Y, 1], H = [h ₁ , h ₂ , h ₃ ] Denotes homography between the points on the 2.5-dimensional pattern and the points projected on the image. λ means any constant.

The internal parameter K can be calculated to satisfy the following condition.

The three-dimensional measurement measured at the three-dimensional TOF sensor includes the depth value of the point on the 2.5-dimensional pattern. In this case, assuming that the measured depth value is accurate, the external parameter may be calculated using a constraint that the depth values measured by the 3D TOF sensor are located on the same plane.

In this case, the external parameter may be calculated to satisfy the following condition.

,

,

은 M에 확장 벡터가 추가된 행렬을 의미한다.M is a matrix representing points on a 2.5D pattern photographed in the coordinate system of the 3D TOF sensor, Q is a matrix representing points projected on the 2.5D pattern in the luminosity image of the 3D TOF sensor, and [R _L t _L ] is an external product. Parameter, N is the normal,

Denotes a matrix in which an extension vector is added to M.

The 2.5-dimensional pattern may be a pattern in which a near-infrared ray of the 3D TOF sensor is large enough to pass through and a hole having a symmetrical shape is irregularly disposed. The 2.5-dimensional pattern may be a pattern consisting of irregularly arranged circular holes.

In the case of a 2.5-dimensional pattern composed of ellipses, even when there is isotropic noise, the center of the ellipse does not change in the captured image, and therefore, even when the resolution is low, the center of the ellipse may be used as a feature point.

For example, in the case of a 2.5-dimensional pattern in which a 4 cm hole is irregularly arranged, the near infrared ray (NIR) can pass through the hole, and thus, even in a color image, the hole can be easily detected by the color difference. . In addition, in the luminance image, the hole can be easily detected by the difference in the depth value.

The three-dimensional TOF sensor can measure the depth value of the point on the 2.5-dimensional pattern.

When the 3D measurement values are projected to the luminance image by using the initial dot product parameters of the 3D TOF sensor, the 3D TOF sensors are often transferred to positions different from the image coordinate values of each 3D measurement value.

In operation 320, the image processing apparatus optimizes an internal parameter of the 3D TOF sensor. The image processing apparatus may optimize the internal parameter such that the 3D measured values are transferred to the original coordinates, that is, the coordinates of the 3D measured values.

The image processing apparatus may optimize the internal parameter of the 3D TOF sensor to minimize the transfer error of the 3D measured values to the luminance image in consideration of the internal parameter and the 3D measured values of the 3D TOF sensor. In this case, the mapping relationship between the 3D measurement value and the coordinates of the luminance image may be modeled by a pinhole camera model to optimize an internal parameter.

The image processing apparatus may transfer the 3D measured values to the luminance image by using an internal parameter. The image processing apparatus may perform optimization using the Levenberg-Marquardt algorithm so that an error between the coordinates of the pixel from which the 3D measurements are transferred and the coordinates of the pixel of the luminance image to which the 3D measurements are originally mapped is minimized.

In operation 330, the image processing apparatus corrects circumferential distortion in the luminance image. The image processing apparatus may correct the circumferential distortion in the luminance image in consideration of the optimized internal parameter.

When the circumferential distortion is compensated for, the feature points on the luminosity image which were not detected due to the circumferential distortion may be detected to obtain more corresponding points. For example, the feature point may be a hole in a 2.5 dimensional pattern.

The image processing apparatus may detect a feature point in the luminance image in which circumferential distortion is compensated. The more feature points spread widely in the luminance image, the more accurate the external calibration can be.

The image processing apparatus may correct the circumferential distortion component included in the luminance image by using the Levenberg-Marquardt algorithm in consideration of the optimized internal parameter. The image processing apparatus may correct the circumferential distortion by using a constraint that the 3D point measured at one pixel of the 3D TOF sensor must be transferred back to the original pixel to be transferred.

In operation 340, the image processing apparatus optimizes the cost function. The cost function takes into account the characteristic points of the luminosity image with the correction of the circumferential distortion, the characteristic points of the color image, the internal and external parameters of the color camera, the optimized internal and external parameters of the three-dimensional TOF sensor, and the depth constraint. It can be a function.

The depth constraint may be to estimate values located in the 2.5D pattern plane among the 3D measured values measured by the 3D TOF sensor.

The cost function is the difference between the coordinates of the feature points of the luminance image corrected for the circumferential distortion and the coordinates of the points projected to the luminance image corrected for the circumferential distortion in the 2.5-dimensional pattern, and the coordinates of the feature points of the color image. A point corresponding to a feature point of the color image in the 2.5-dimensional pattern may include a difference between the coordinates of the point projected into the color image and a depth constraint.

One point on the 2.5-dimensional pattern should coincide with the feature points detected in the color image when projected onto the color image. In addition, one point on the 2.5-dimensional pattern should coincide with the feature point detected in the luminance image when projected onto the luminance image. In addition, the points on one plane measured by the three-dimensional TOF sensor should be on the 2.5-dimensional pattern plane. These conditions can be reflected in the cost function.

The image processing apparatus may correct the external parameters of the color camera and the external parameters of the 3D TOF sensor by optimizing the cost function using the Levenberg-Marquardt algorithm.

The parameters can be expressed as follows.

R _c2t and t _c2t are the external parameters of the three-dimensional TOF sensor and represent rotation and translation matrices. R _c1 , t _c1 , R _ci , t _ci , R _cN and t _cN are the external parameters of the color camera and represent the rotation matrix and the translation matrix.

The cost function f (P) can be defined as

는 컬러 영상에서 검출된 특징점의 좌표,

는 컬러 영상에 투영된 특징점의 좌표,

는 3차원 TOF 센서에서 촬영된 광도 영상에서 검출된 특징점의 좌표,

는 광도 영상에 투영된 특징점의 좌표,

는 깊이 제약조건,

는 2.5차원 패턴 평면 상의 점에 대해 측정된 3차원 측정값을 나타낸다. here,

Is the coordinate of the feature point detected in the color image,

Is the coordinate of the feature point projected on the color image,

Is the coordinate of the feature point detected in the luminance image captured by the 3D TOF sensor,

Is the coordinate of the feature point projected on the luminance image,

Is the depth constraint,

Denotes a three dimensional measurement measured for a point on a 2.5 dimensional pattern plane.

The image processing apparatus may acquire a feature point of the luminance image in which the circumferential distortion is corrected, and may acquire a feature point of the color image.

The image processing apparatus may estimate the position and angle between the color camera and the 3D TOF sensor using the optimized cost function.

4 is a flowchart of an image processing method according to another exemplary embodiment of the present invention.

Referring to FIG. 4, the color camera 403 may acquire the color image 407 by photographing the 2.5D pattern 401. The TOF sensor 405 may acquire the luminance image 409 and the 3D measured value 411 by photographing the 2.5D pattern 401. The image processing apparatus may acquire a color image 407, a luminance image 409, and a 3D measurement value 411. The color camera 403 and the TOF sensor 405 may be included as one component of the image processing apparatus, or may be an external device separate from the image processing apparatus.

The image processing apparatus may perform an initial calibration 413 through the color image 407, and calculate an internal parameter 417 and an external parameter 419. The internal parameter 417 may represent the focal length of the color camera 403 and the center of the color image 407. The external parameter 419 can indicate at what position and angle the plane of the 2.5-dimensional pattern 401 is from the color camera 403.

The image processing apparatus may perform an initial calibration 415 through the luminance image 409, and calculate an internal parameter 421 and an external parameter 423. The internal parameter 421 may represent the focal length of the TOF sensor 405 and the center of the luminance image 409. The external parameter 423 can indicate at what position and angle the plane of the 2.5-dimensional pattern 401 is from the TOF sensor 405.

The 3D measurement value 411 means a 3D coordinate value measured for each pixel of the luminance image 409.

When the 3D measurement value 411 is transferred to the luminance image 409 using the internal parameter 421, the transferred coordinate value is different from the 3D coordinate value to which the 3D measurement value 411 is to be transferred. . That is, an error occurs.

The image processing apparatus optimizes 425 the internal parameter 421 to minimize the error. In this case, the image processing apparatus may use the 3D measurement value 411 and the Levenberg-Marquardt algorithm to calculate the optimized internal parameter 427 in which the error is minimal.

The image processing apparatus may remove circumferential distortion 429 from the luminance image 409 using the optimized inner product parameter 427.

The image processing apparatus includes a corresponding point of the image 431 without distortion—a point corresponding to a feature point of a 2.5-dimensional pattern—a corresponding point of the color image 403, an internal parameter 417 of the color camera 403, an external parameter 419, The final function 433 of the optimized internal parameter 427, the external parameter 423, and the three-dimensional measured value 411 of the TOF sensor 405 can be finally optimized 433. have.

The image processing apparatus may reflect more correspondence points in the luminance image 409 by considering the correspondence points of the optimized internal parameter 427 of the TOF sensor 405 and the image 431 without distortion. In addition, by taking into account the measured value of the 2.5-dimensional pattern plane of the dimensional measurement value 411, the three-dimensional characteristics that the color camera 403 does not provide may be reflected in the external calibration.

The image processing apparatus calculates using the dot product parameter 417 of the color camera 403, the optimized dot product 427 of the TOF sensor 405, the corresponding point of the image 431 without distortion, and the corresponding point of the color image 403. Optimization may be performed to minimize the transfer error of the feature point of the 2.5D pattern and the error of the 3D measured value 411. In this case, the image processing apparatus may minimize the error by using the Levenberg-Marquardt algorithm.

FIG. 5 is a diagram illustrating a result of being transferred to an image before and after optimization of an internal parameter in an image processing process according to an embodiment of the present invention.

Referring to FIG. 5, the transferred result 510 is a result of transferring a 3D measured value to an image using an internal parameter obtained as a result of performing an initial calibration using only a luminance image. It can be found that the three-dimensional measurements are not transferred to the corresponding coordinate values in the image.

For accurate external calibration, it is important to find many correspondences that are spread out in the image. Luminance images with no circumferential distortion corrected are circumferentially distorted, so that even if a hole is detected, the homography constraint is not satisfied, and thus, the detected hole is not detected as points corresponding to the 2.5-dimensional pattern. In particular, the hole at the edge of the image has a higher probability of not being detected as a corresponding point. In order to detect more correspondence points, it is necessary to remove the circumferential distortion of the luminance image, and to remove the circumferential distortion, accurate internal parameters of the three-dimensional TOF sensor are required.

The exact dot product can be obtained through optimization of the dot product. The optimized dot product can be obtained by minimizing an error between the coordinates from which the 3D measurements are transferred and the coordinates from which the original 3D measurements are to be transferred using the initial value of the dot product.

The transferred result 520 is a result of transferring the 3D measurement value to the image by using the optimized internal parameter. Compared with the transferred result 510, the transferred result 520 may confirm that the 3D measurement values are also transferred to the edge and transferred to image coordinates to be transferred. In addition, it may be expected that more corresponding points corresponding to the feature points may be detected in the transferred result 520.

6 is a block diagram of an image processing apparatus according to an embodiment of the present invention.

Referring to FIG. 6, the image processing apparatus includes a photographing unit 610, a calculator 620, a first optimizer 630, a controller 640, a corrector 650, and a second optimizer 660. do.

The photographing unit 610 acquires a color image by capturing a 2.5-dimensional pattern with a color camera. The photographing unit 610 captures a 2.5-dimensional pattern with a 3D TOF sensor to obtain an intensity image and 3D measurement values.

The 2.5-dimensional pattern may be a flat plate having irregularly arranged circular holes that are large enough to pass near-infrared rays emitted from the 3D TOF sensor.

The calculator 620 may calculate an initial value of an internal parameter and an external parameter of the color camera by performing initial calibration on the color image. The calculator 620 may calculate an initial value of an internal parameter and an external parameter of the 3D TOF sensor by performing initial calibration on the luminance image.

The first optimizer 630 may optimize the internal parameter of the 3D TOF sensor to minimize the transfer error of the 3D measured values in consideration of the internal parameter and the 3D measured values of the 3D TOF sensor.

The first optimizer 630 may transfer the 3D measured values to the luminance image using the internal parameter. The first optimizer 630 performs optimization using the Levenberg-Marquardt algorithm so that an error between the coordinates of the pixel to which the 3D measurement values are transferred and the coordinates of the pixel of the luminance image to which the 3D measurement values are originally mapped is minimized. can do.

The correction unit 650 may correct the circumferential distortion in the luminance image in consideration of the optimized internal parameter.

The second optimizer 660 optimizes the cost function. The cost function takes into account the characteristic points of the luminosity image with the correction of the circumferential distortion, the characteristic points of the color image, the internal and external parameters of the color camera, the optimized internal and external parameters of the three-dimensional TOF sensor, and the depth constraint. It can be a function.

The depth constraint may be to estimate values located in the 2.5D pattern plane among the 3D measured values measured by the 3D TOF sensor.

The second optimizer 660 may include a difference between the coordinates of the feature points of the luminosity image in which the circumferential distortion is corrected and the coordinates of the points corresponding to the feature points in the 2.5-dimensional pattern as the luminescence image in which the circumferential distortion is corrected, the color image. The minimum value of the cost function including the depth constraint and the difference between the coordinates of the feature points of the point and the coordinates of the point projected to the color image of the point corresponding to the feature point of the color image in the 2.5-dimensional pattern can be calculated.

The controller 640 is responsible for overall control of the image processing apparatus, and functions of the photographing unit 610, the calculator 620, the first optimizer 630, the corrector 650, and the second optimizer 660. Can be performed. In the embodiment of FIG. 6, this configuration is illustrated separately to describe each function. Therefore, in the case of actually implementing a product, all of them may be configured to be processed by the controller 640, and only some of them may be configured to be processed by the controller 640.

The above-described methods may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. The computer readable medium may include program instructions, data files, data structures, etc. alone or in combination. The program instructions recorded on the medium may be those specially designed and constructed for the present invention or may be available to those skilled in the art of computer software.

As described above, the present invention has been described by way of limited embodiments and drawings, but the present invention is not limited to the above embodiments, and those skilled in the art to which the present invention pertains various modifications and variations from such descriptions. This is possible.

Therefore, the scope of the present invention should not be limited to the described embodiments, but should be determined by the equivalents of the claims, as well as the claims.

Claims (16)

Capturing a 2.5-dimensional pattern with a color camera to obtain a color image, and capturing the 2.5-dimensional pattern with a 3D time of flight (TOF) sensor to obtain luminance images and 3D measurements;
Optimizing the internal parameter of the three-dimensional TOF sensor to minimize the transfer error of the three-dimensional measured values in consideration of the internal parameter of the three-dimensional TOF sensor and the three-dimensional measured values;
Correcting radial distortion in the luminance image in consideration of the optimized internal parameter; And
Features of luminosity image with circumferential distortion correction, features of the color image, internal and external parameters of the color camera, optimized internal and external parameters and depth constraints of the 3D TOF sensor are considered The optimized cost function
And an image processing method. The method of claim 1,
Performing initial calibration on the color image and the luminance image to calculate the internal parameter and the external parameter of the color camera, and calculating the internal parameter and the external parameter of the three-dimensional TOF sensor.
Further comprising the steps of: The method of claim 1,
Optimizing the cost function
Acquiring a feature point of the luminosity image in which the circumferential distortion is corrected, and obtaining a feature point of the color image
And an image processing method. The method of claim 1,
Estimating the position and angle between the color camera and the three-dimensional TOF sensor using an optimized cost function
Further comprising the steps of: The method of claim 1,
The 2.5-dimensional pattern is
Near-infrared rays of the three-dimensional TOF sensor has a size enough to pass, characterized in that the pattern is irregularly arranged holes having a symmetrical shape
Image processing method. The method of claim 1,
The three-dimensional TOF sensor
To measure the depth value of the point on the 2.5-dimensional pattern
And an image processing method. The method of claim 1,
Optimizing the internal parameter of the three-dimensional TOF sensor
Levenberg-Marquardt so that the error between the coordinates of the pixel and the coordinates of the pixel of the luminance image mapped to the three-dimensional measurement value is minimized as a result of transferring the three-dimensional measurement values to the luminance image using the internal parameter To optimize using
Video conversion method. The method of claim 1,
Correcting the radial distortion
The circumferential distortion component included in the luminance image is corrected by using the Levenberg-Marquardt method in consideration of the optimized internal parameter.
Video conversion method. The method of claim 1,
The depth constraint is
Estimating values located in the 2.5-dimensional pattern plane among three-dimensional measured values measured by the three-dimensional TOF sensor
Video conversion method. 10. The method of claim 9,
The cost function is
Between a coordinate of a feature point of the luminosity image with the circumferential distortion corrected and a point corresponding to a feature point of the luminosity image with the circumferential distortion corrected in the 2.5-dimensional pattern between the coordinates of the point projected into the luminosity image with the circumferential distortion corrected The difference, the difference between the coordinates of the feature points of the color image and the coordinates of the points corresponding to the feature points of the color image in the 2.5-dimensional pattern to the color image and the depth constraint
Image conversion method comprising a. A photographing unit which acquires a color image by capturing a 2.5-dimensional pattern with a color camera, and acquires a luminance image and three-dimensional measurement values by capturing the 2.5-dimensional pattern with a three-dimensional TOF sensor;
A first optimizer for optimizing an internal parameter of the 3D TOF sensor in order to minimize a transfer error of the 3D measured values in consideration of the internal parameter of the 3D TOF sensor and the 3D measured values;
A correction unit for correcting radial distortion in the luminance image in consideration of an optimized internal parameter; And
Features of luminosity image with circumferential distortion correction, features of the color image, internal and external parameters of the color camera, optimized internal and external parameters and depth constraints of the 3D TOF sensor are considered Second optimization unit for optimizing the estimated cost function
And the image processing apparatus. The method of claim 11,
A calculation unit configured to calculate the internal parameter and the external parameter of the color camera by performing initial calibration on the color image and the luminance image, and calculate the internal parameter and the external parameter of the 3D TOF sensor
Further comprising: The method of claim 11,
The 2.5-dimensional pattern is
Characterized in that the circular holes having a size that can pass through the near infrared rays of the three-dimensional TOF sensor is irregularly arranged
Image processing device. The method of claim 11,
The first optimization unit
As a result of transferring the three-dimensional measured values to the luminance image using the internal parameter, optimization is performed such that an error between the coordinate of the pixel and the coordinate of the pixel of the luminance image mapped to the three-dimensional measured values is minimized. doing
Video conversion device. The method of claim 11,
The depth constraint is
Estimating values located in the 2.5-dimensional pattern plane among three-dimensional measured values measured by the three-dimensional TOF sensor
Video conversion device. 16. The method of claim 15,
The second optimizer
Between a coordinate of a feature point of the luminosity image with the circumferential distortion corrected and a point corresponding to a feature point of the luminosity image with the circumferential distortion corrected in the 2.5-dimensional pattern between the coordinates of the point projected into the luminosity image with the circumferential distortion corrected Calculating a minimum value of a cost function including the depth constraint and the difference between the coordinates of the feature points of the color image and the point corresponding to the feature points of the color image in the 2.5-dimensional pattern;
Video conversion device.

Images