CN109325981B - Geometric parameter calibration method for micro-lens array type optical field camera based on focusing image points - Google Patents

Abstract

The invention discloses a method for calibrating geometric parameters of a microlens array type light field camera based on a focused image point. The method includes the following steps: S1, according to a focused imaging optical path diagram of the microlens array type light field camera, obtain the object point and the focus The mapping relationship between the image point and the main lens; S2, according to the focusing imaging optical path diagram of the microlens array light field camera, the mapping relationship between the focusing image point and the detector image point on the microlens is obtained; S3, according to the detected detector image point, solve the coordinates of the focused image point; S4, solve the camera internal parameter matrix and external parameter moment in the calibration model according to the coordinates of the focused image point obtained in S3; S5, obtain the camera internal parameter matrix and external parameter matrix through S4, Calibrate the geometric parameters of the microlens array light field camera. By adopting the method provided by the present invention to calibrate the geometric parameters of the microlens array light field camera, reliable parameters can be provided for subsequent light field data calibration and computational imaging.

Description

Geometric parameter calibration method of microlens array light field camera based on focused image point

technical field

The invention relates to the technical field of computer vision and digital image processing, in particular to a method for calibrating geometric parameters of a microlens array light field camera based on a focused image point.

Background technique

Light field imaging technology can simultaneously record the spatial and angular information of light, which can break through the limitations of conventional lens imaging. Using light field data, computational imaging techniques such as digital refocusing, depth of field extension, scene depth calculation, and scene 3D reconstruction can be realized, and are widely used in the fields of computer vision and computational imaging.

Different sampling methods of the light field form different hardware systems for acquiring light field data, such as microlens array light field cameras, structured camera array light field cameras, and mask light field cameras. Among them, the microlens array type light field camera completes the light field data collection by adding a microlens array to the imaging system and splitting the light beams of the main lens to discretize the continuous light field. The microlens array light field camera has the advantages of simple hardware structure, portable equipment, and the ability to obtain light field data in a single exposure, and is currently the mainstream light field acquisition device. A typical representative of the microlens array light field camera is the light field camera (Plenoptic 1.0) designed by Ng Ren et al. In this imaging device, a microlens array is placed at the focal plane of a conventional camera, and an image detector is placed at one focal length of the microlens. Georgiew et al. proposed a focusing light field camera (Plenoptic2.0), where the imaging detector is not on the focal plane of the microlens array, which reduces the sampling of the directional dimension of the light, and trades a lower directional resolution for a relatively higher spatial resolution , which effectively improves the imaging resolution of the refocused image.

The calibration of the geometric parameters of the light field imaging system is an important prerequisite for the calibration of light field data and the realization of computational imaging technology. The calibration process of the geometric parameters of the camera is divided into the modeling of the positive process from the object point to the image point of the detector, and the process of solving the geometric parameters of the camera by the corner detection and inversion. Positive process modeling is to establish the mapping relationship between the three-dimensional coordinates of the object point and the two-dimensional coordinates of the image point on the imaging plane, and this mapping relationship is described by the camera geometric parameters. The process of solving the geometric parameters of the camera is to use the coordinates of the corner points and object points on the detected imaging surface to invert and solve the geometric parameters of the camera based on the forward process modeling, which corresponds to the nonlinear solution problem. That is to say, the existing method for calibrating the geometric parameters of the camera integrates the geometric parameters into a model, detects the corner points, and then calculates the geometric parameters of the microlens array light field camera from the corner points. This method of calibrating the geometric parameters of the camera has a high degree of model coupling, and the calculation is complicated.

SUMMARY OF THE INVENTION

The purpose of the present invention is to provide a method for calibrating the geometric parameters of a microlens array light field camera based on a focused image point to solve the geometric parameters of the camera.

In order to achieve the above purpose, the present invention provides a method for calibrating geometric parameters of a microlens array light field camera based on a focused image point, and the method for calibrating geometric parameters of a microlens array light field camera based on a focused image point includes the following steps: S1 , according to the focusing imaging optical path diagram of the microlens array light field camera, the mapping relationship between the object point and the focusing image point on the main lens is obtained; S2, according to the focusing imaging optical path diagram of the microlens array light field camera, the focusing image point and the focusing image point are obtained. The mapping relationship between the detector image point and the microlens; S3, according to the detected detector image point, solve the coordinates of the focus image point; S4, according to the coordinates of the focus image point obtained in S3, solve the camera internal parameter matrix in the calibration model and external parameter matrix; and S5, the camera internal parameter matrix and external parameter matrix obtained by S4, calibrate the geometric parameters of the microlens array light field camera, the geometric parameters include the distance between the microlens array and the detector plane and the distance from the main lens to the main lens Microlens distance.

Further, "the mapping relationship between the object point and the focus image point on the main lens" in S1 is expressed as formula (1), and it is set that: the object point is expressed as M, and its coordinates are (x _w , y _w , z _w ); The focused image point is denoted as m', and its coordinates are (u', v'):

sm′=A[R t]M, (1)

In formula (1), s is the scale factor; M=[x _w , y _w , z _w , 1] ^T ; m′=[u′, v′, 1] ^T ; [R t] represents the microlens array type The external parameter matrix of the light field camera, A represents the internal parameter matrix of the microlens array light field camera;

[R t[ is represented by the following formulas (3) and (2):

In formula (2) and formula (3), q ₁ , q ₂ , and q ₃ respectively represent the rotation angles of the three coordinate axes during the transformation of the camera coordinate system and the world coordinate system, and t _x , _ty , and t _z represent the world coordinates The distance that the origin of the system is translated along the three coordinate axes;

A is represented by the following formula (4):

In formula (4), dx×dy is the pixel size of the detector, u ₀ and v ₀ are the coordinates of the center of the main lens in the image coordinate system, and θ is the non-perpendicular tilt angle of the two coordinate axes;

Among them: the camera coordinate system takes the center of the main lens as the origin O, the x _c and y _c axes are parallel to the detector plane, and the z _c axis is perpendicular to the detector plane; the world coordinate system takes the center of the calibration object plane as the origin O _w , x The _w and y _w axes are parallel to the calibration object plane, and the z _w axis is perpendicular to the calibration object plane; the image coordinate system takes the center of the detector plane as the origin O _c , and the u and v axes are parallel to the x _c and y _c axes.

Further, the "mapping relationship between the focus image point and the detector image point with respect to the microlens" in S2 is the formula (8). It is set that the detector image point includes the image point m ₁ and the image point m ₂ , and the image point m ₁ The coordinates in the image coordinate system are (u ₁ , v ₁ ), and the coordinates of the center of the corresponding microlens in the image coordinate system are (p ₁ , q ₁ ); the coordinates of the image point m ₂ in the image coordinate system are (u 1 , q 1 ) ₂ , v ₂ ), the coordinates of the corresponding microlens center in the image coordinate system are (p ₂ , q ₂ );

sm′=sT ⁻¹ m ₁ =A[R t]M (8)

In formula (8): m ₁ represents the coordinate vector of the image point m ₁ ; T is the transformation matrix between the image point m' and the image point m ₁ ;

m ₁ is expressed as m ₁ =[u ₁ ,v ₁ ,1] ^T ;

T is represented by the following equations (6) and (7):

δ=b/a=(u ₁ -u ₂ )/(p ₁ -p ₂ )-1 (7).

Further, S3 specifically is to solve the coordinates of the focused image point according to the detected image point of the detector, and use the geometrical relationship in the focused imaging optical path diagram of the microlens array type;

"Detected detector image points" are image points m ₁ , m ₂ ;

"The geometric relationship in the focused imaging optical path diagram of the microlens array type" is expressed as formula (9):

Then formula (10) can be obtained according to formula (8):

First, substitute the coordinates of the image points m ₁ and m ₂ in the image coordinate system and the coordinates of the center of the corresponding microlens in the image coordinate system into formula (7), and calculate the δ value;

Then, the calculated δ value is substituted into formula (9), and the coordinates (u', v') of the focus image point m' are solved.

Further, S4 specifically includes: S41, set the calibration model plane z _w = 0 in the world coordinate system, substitute formula (1), and obtain a 3 × 3 homography according to the coordinates of the object point M and the corresponding focus image point m' property matrix H, in order to solve the maximum likelihood estimation of the matrix H, iteratively solve the nonlinear least squares problem of the focused image points in the image; S42, according to the constraints of r ₁ and r ₂ , compare the r ₁ obtained by the matrix H with the Substitute the expression of r ₂ , and then bring the symmetric matrix B and its calculation formula into the obtained equation, and finally obtain a 2 × 6 matrix V, which can be substituted into the homography matrix h _i of the three images to solve the matrix b, and then Solve the unknown parameters in matrix A; S43, matrix A has been solved, substituting matrix A into the expressions of r ₁ and r ₂ can solve r ₁ , r ₂ , and combining matrix H can solve matrix [R t] The unknown parameters of the The parameters are optimized nonlinearly.

Further, S41 specifically includes:

Set the calibration model plane z _w = 0 in the world coordinate system, and substitute it into equation (1), there is equation (11):

According to the coordinates of the object point M and the corresponding focus image point m', a 3×3 homography matrix H can be obtained in the following form:

H=[h ₁ h ₂ h ₃ ]=λA[r ₁ r ₂ t] (12)

In formula (12), h _i = [h _i1 , h _i2 , h _i3 ] ^T , λ is an arbitrary scalar;

To solve the maximum likelihood estimate of the matrix H, iteratively solve the nonlinear least squares problem for the focused image points in the image:

根据式(11)、式(12)由物点M_i计算得到。In formula (13), m′ _i is the actual coordinate vector of the i-th focused image point in the same image,

According to formula (11) and formula (12), it is calculated from the object point _Mi.

Further, S42 specifically includes:

According to the properties of the rotation matrix, r ₁ and r ₂ in the matrix R satisfy the following two constraints:

From formula (12), it can be known that

r ₁ =λA ^-1 h ₁ ,r ₂ =λA ^-1 h ₂ (15)

In formula (15), λ=1/||A ^-1 h ₁ ||=1/||A ^-1 h ₂ ||, and substituting formula (15) into formula (14) can get:

make

B is a symmetric matrix, and formula (18) can be obtained by calculation:

In formula (18):

b=[B ₁₁ , B ₁₂ , B ₂₂ , B ₁₃ , B ₂₃ , B ₃₃ ] ^T (19)

v _ij =[h _i1 h _j1 ,h _i1 h _j2 +h _i2 h _j1 ,h _i2 h _j2 ,h _i3 h _j1 +h _i1 h _j 3,h _i3 h _j 2+h _i2 h _j3 ,h _i3 h _j3 ] ^T (20)

Substituting equations (17) and (18) into equation (16) can obtain equation (21):

That is, there is formula (22):

In formula (22), V is a 2 × 6 matrix, which contains 6 unknown parameters. Substitute into the homography matrix h _i of the three images to solve the matrix b, and then solve the unknown parameters in the matrix A:

γ=-B ₁₂ α ² β/λ (26)

u ₀ =γv ₀ /β-B ₁₃ α ² /λ (27)

Thus, the internal parameter matrix A of the microlens array type light field camera is determined.

Further, S43 specifically includes:

Substitute the matrix A obtained from S42 into formula (15) to obtain r ₁ and r ₂ , and the specific calculation expression is formula (29):

r ₃ =r ₁ ×r ₂ (29)

Formula (30) can be obtained from formula (12):

t=λA ^-1 h ₃ (30).

Further, S44 specifically includes:

The n images are calibrated, each image has m focus points, and the parameters obtained by the above solution method are iteratively optimized to solve the nonlinear least squares problem:

根据式(8)由第i幅图像的第j个物点坐标向量M_j及A_i，R_i，t_i计算得到，A和[R t]的初始值由S41到S43计算得到。In formula (31), m′ _ij is the focus point coordinate of the jth corner of the ith image,

According to formula (8), the jth object point coordinate vector M _j and A _i , R _i , t _i of the ith image are calculated, and the initial values of A and [R t] are calculated from S41 to S43.

Further, S5 specifically includes:

The pupil diameter D of the main lens and the diameter d of the microlens image satisfy the following formula:

D/L=d/b (32)

Formula (33) can be obtained from formula (7):

a=b/δ (33)

Substituting equations (32) and (33) into α=(L-a)/(dx) in equation (4), equations (34) and (35) can be obtained:

L=a+αdx (34)

b=αδddx/(δD-d) (35)

In equations (34) and (35), the parameters α and δ are known parameters.

The method provided by the invention divides the model into two conjugate relationships, detects the corner points, obtains the focused image point from the corner points according to the second conjugate relationship, and then calculates the geometry of the microlens array type light field camera based on the second conjugate relationship. parameters, this method has low model coupling and is easy to calculate.

Description of drawings

1 is a flowchart of a method for calibrating geometric parameters of a focused image point-based microlens array light field camera provided by the present invention;

Fig. 2 is the light path diagram of focusing imaging of the microlens array type provided by the present invention;

FIG. 3 is a schematic diagram of the correspondence between the image coordinate system, the camera coordinate system and the world coordinate system in the present invention.

Detailed ways

In the drawings, the same or similar reference numerals are used to denote the same or similar elements or elements having the same or similar functions. The embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

As shown in FIG. 1 , the method for calibrating geometric parameters of a microlens array light field camera based on a focused image point provided in this embodiment includes the following steps:

S1 , according to the focused imaging optical path diagram of the microlens array light field camera, the mapping relationship between the object point and the focused image point on the main lens is obtained, that is, the "first conjugate relationship" mentioned below.

S2, according to the focused imaging optical path diagram of the microlens array light field camera, the mapping relationship between the focused image point and the detector image point with respect to the microlens is obtained, that is, the "second conjugate relationship" mentioned below.

S3, according to the detected image point of the detector, use an inverse transformation matrix or a geometric relationship to find the coordinates of the focused image point.

S4, according to the coordinates of the focused image point obtained in S3, an iterative algorithm is used to solve the camera internal parameter matrix and external parameter matrix in the calibration model. The "camera internal parameter matrix" in the text refers to the internal parameters of the microlens array light field camera, and the "camera external parameter matrix" refers to the external parameters of the microlens array light field camera.

S5, through the camera internal parameter matrix and external parameter matrix obtained in S4, the geometric parameters of the microlens array light field camera are calibrated, the geometric parameters are the distance b between the microlens array and the detector plane and the distance L from the main lens to the microlens.

The five steps of the present invention will be described in detail below.

In one embodiment, in S1, the microlens array type light field camera acquires the light field information of the scene by placing a microlens array in front of a detector in a conventional camera.

The light path diagram of the microlens array type light field camera is shown in Figure 2. As shown in FIG. 2 , the microlens array is a structure composed of a plurality of microlenses arranged in an array, for example, some of the microlenses in the microlens array shown in FIG. 2 , namely the first microlens l1 and the second microlens l2 , a third microlens l3, and a fourth microlens l4.

In FIG. 2, the order from left to right is: the dotted straight line P1 where the image point m' is located indicates the plane where the focused imaging point is located (hereinafter referred to as "focusing imaging plane"). The solid straight line P2 where the focused image point m ₁ and the focused image point m ₂ are located indicates the plane where the detector is located (hereinafter referred to as "detector plane"). The solid line P3 where the first microlens 11, the second microlens 12, the third microlens 13, and the fourth microlens 14 are located at the same time shows the plane where the microlens array is located (hereinafter referred to as "microlens array surface") . The solid line P3 where the main lens lm is located represents the plane of the main lens. The imaginary straight line P4 where the object point M is located indicates the plane where the object point M is located (hereinafter referred to as "object plane").

Wherein: the distance between the microlens array surface and the detector surface is b, which can also be said to be: the distance b between the microlens array and the detector plane, that is, one of the geometric parameters to be calibrated in the present invention mentioned in S5 above . The distance between the main lens plane and the microlens array surface is L, which can also be said to be the distance L from the main lens to the microlens, which is the second geometric parameter to be calibrated in the present invention mentioned in S5 above.

The distance from the focusing imaging surface to the microlens array surface is a, the focal length of the microlens is f, and it can be known from the lens imaging formula that 1/a+1/b=1/f. Figure 2 corresponds to the case of b<f, at this time a<0, the image point formed by the object point through the main lens is on the other side of the microlens array, which is a virtual image.

The distance from the focusing imaging plane to the main lens plane is b _L , and the distance from the object plane to the main lens plane is a _L . A is the diaphragm, the pupil diameter of the main lens is D, and the diameter of the microlens image is d.

In S1, for the microlens array arrangement of the above-mentioned microlens array type, considering the microlens arrays with the same focal length, the modeling of the positive problem of generating an image point from an object point is given. Positive problem modeling includes establishing the mapping relationship between the object point and the focus image point on the main lens, and the mapping relationship between the focus image point and the detector image point on the microlens.

The microlens array type focusing imaging process shown in FIG. 2 has two conjugate relationships, that is, the object point M and the focused image point m' are conjugate with respect to the main lens plane, and the image point m' and the two on the detector are conjugated. The focusing image points m ₁ , m ₂ are conjugate with respect to the microlens array surface, respectively corresponding to the “first conjugate relationship” and “second conjugate relationship” mentioned above.

The "mapping relationship between the object point and the focus image point on the main lens" in S1 is specifically:

The mapping relationship between the object point M and the focused image point m' on the main lens is described by the first conjugate relationship. The process that the object point M forms the image point m' through the main lens can be described by the coordinate transformation relationship between the world coordinate system, the camera coordinate system and the image coordinate system.

As shown in Figure 3, it is set that the three-dimensional camera coordinate system takes the center of the main lens as the origin O, the x _c and y _c axes are parallel to the detector plane, and the z _c axis is perpendicular to the detector plane. The two-dimensional image coordinate system takes the center of the detector plane as the origin O _c , and the u and v axes are parallel to the x _c and y _c axes. The three-dimensional world coordinate system takes the center of the calibration object plane as the origin O _w , the x _w and y _w axes are parallel to the calibration object plane, and the z _w axis is perpendicular to the calibration object plane. The unit of the 3D camera coordinate system is pixel, the unit of the 2D image coordinate system is mm, and the unit of the 3D world coordinate system is mm. The coordinates of the object point M are (x _w , y _w , z _w ), the coordinates of the image point m' are (u', v'), and u' corresponds to u' shown in FIG. 2 .

The process of converting the object point M to the image point m' in the two-dimensional image coordinate system can be expressed as formula (1):

sm′=A[R t]M (1)

In formula (1):

s is the scale factor;

M can be expressed as M=[x _w , y _w , z _w , 1] ^T ;

m' can be expressed as m'=[u',v',1] ^T ;

[R t] represents the external parameter matrix of the microlens array light field camera;

A represents the internal parameter matrix of the microlens array type light field camera.

The camera's external parameter matrix [R t] describes the transformation relationship between the world coordinate system and the camera coordinate system, which can be expressed as equations (2) and (3):

In equations (2) and (3), q ₁ , q ₂ , and q ₃ represent the rotation angles of the three coordinate axes in the process of transforming the camera coordinate system and the world coordinate system; t _x , _ty , and t _z represent the origin O _w The distance to translate along the three coordinate axes respectively.

The camera's internal parameter matrix A can be expressed as equation (4), which describes the conversion relationship between the camera coordinate system and the image coordinate system:

In formula (4): dx×dy is the pixel size of the detector; u ₀ , v ₀ are the coordinates of the center of the main lens in the image coordinate system, and θ is the non-perpendicular inclination angle between the camera coordinate system and the world coordinate system.

In one embodiment, in S2, it is set that the coordinates of the image point m ₁ in the image coordinate system are (u ₁ , v ₁ ), and the coordinates of the center of the corresponding microlens in the image coordinate system are (p ₁ , q ₁ ), u ₁ , p ₁ correspond to u ₁ , p ₁ shown in FIG. 2 ; the coordinates of the image point m ₂ in the image coordinate system are (u ₂ , v ₂ ), and the corresponding microlens center is in the image coordinates The coordinates of the system are (p ₂ , q ₂ ), and u ₂ and p ₂ correspond to u ₂ and p 2 shown in FIG. ₂ . From the geometric relationship of the microlens under the pinhole model in Figure 3, the coordinate conversion relationship between the image point m' formed by the focusing of the main lens and the image point m ₁ on the detector is equation (5):

m ₁ =Tm' (5)

In formula (5):

m ₁ =[u ₁ ,v ₁ ,1] ^T is the coordinate vector of the image point m ₁ ;

T is the transformation matrix between the image point m' and the image point m ₁ , which can be specifically expressed as the following formulas (6) and (7):

δ=b/a=(u ₁ -u ₂ )/(p ₁ -p ₂ )-1 (7)

Substituting Equation (1) into Equation (5), the coordinate conversion relationship between the object point M and the focused image point m ₁ on the detector can be obtained as Equation (8):

sm′=sT ⁻¹ m ₁ =A[R t]M (8)

In one embodiment, in S3, firstly, the image points detected on the detector surface are used for grouping, and the detector image points corresponding to the same object point are taken as a group, and each group of detector image points is based on T ^-1 Or geometric relationship to get focus image point coordinates.

Formula (9) can be obtained according to the geometric relationship in the focused imaging optical path diagram of the microlens array type:

That is, in formula (8)

Two image points are selected from the detector image points corresponding to the same object point, and the coordinates of the image points and the center coordinates of the microlens corresponding to the image points are substituted into the formula (1) to calculate the δ value, and then the focused image point in the formula (9) is solved. the coordinates (u', v').

In one embodiment, in S4, in order to solve the internal parameter matrix A and the external parameter matrix [R t] of the microlens array type light field camera in Equation (8), the focused image point coordinates obtained in S3 are used, and the calibration method of Zhang Zhengyou is used for reference. It is solved by an iterative algorithm, and its specific methods include:

S41 , set the calibration model plane z _w =0 in the world coordinate system, and substitute into formula (1). A 3 × 3 homography matrix H can be obtained according to the coordinates of the object point M and the corresponding focus image point m'. In order to solve the maximum likelihood estimation of the matrix H, the focus image points in the iterative image are solved to solve the nonlinear least squares problem.

S42, according to the constraints of r ₁ and r ₂ , substitute the expressions of r ₁ and r ₂ obtained from the matrix H into the obtained equation, and then bring the symmetric matrix B and its calculation formula into the obtained equation, and finally obtain a 2× 6 Matrix _V , which can be substituted into the homography matrix hi of the three images to solve the matrix b, and then solve the unknown parameters in the matrix A.

S43, the matrix A has been solved. Substituting it into the expressions of r ₁ and r ₂ can solve r ₁ and r ₂ , and combining with the matrix H, the unknown parameters in the matrix [R t] can be solved.

S44 , calibrate the n images, each image has m focus points, iteratively optimize the parameters obtained by the above solution method, and solve the nonlinear least squares problem, so as to perform the nonlinear least squares problem on the parameters obtained by the above steps. optimization.

In one embodiment, "substituting into formula (1)" in S41 is specifically as formula (11):

The "3 × 3 homography matrix H" in S41 is specifically as formula (12):

H=[h ₁ h ₂ h ₃ ]=λA[r ₁ r ₂ t] (12)

In S41 "iteratively solve the non-linear least squares problem with the focused image points in the image" as shown in formula (13):

可以根据式(11)、式(12)由物点M_i计算得到。m′ _i in formula (13) is the actual coordinate vector of the i-th focused image point in the same image,

It can be calculated from the object point _Mi according to formula (11) and formula (12).

In one embodiment, the "constraints of r ₁ and r ₂ " in S42 are specifically as in formula (14):

The "substitute the expressions of r ₁ and r ₂ obtained from H into" in S42 is specifically as formula (15):

r ₁ =λA ^-1 h ₁ ,r ₂ =λA ^-1 h ₂ (15)

In formula (15), λ=1/||A ⁻¹ h ₁ ||=1/||A ⁻¹ h ₂ ||.

After substituting Equation (15) into Equation (14), Equation (16) can be obtained:

"Bring the symmetric matrix B and its calculation formula into the obtained equation" in S42 is specifically as formula (17):

make

B is a symmetric matrix, and formula (18) can be obtained by calculation:

In formula (18):

b=[B ₁₁ , B ₁₂ , B ₂₂ , B ₁₃ , B ₂₃ , B ₃₃ ] ^T (19)

v _ij =[h _i1 h _j1 ,h _i1 h _j2 +h _i2 h _j1 ,h _i2 h _j2 ,h _i3 h _j1 +h _i1 h _j3 ,h _i3 h _j2 +h _i2 h _j3 ,h _i3 h _j3 ] ^T (20)

"Obtaining a 2 × 6 matrix V" in S42 is to substitute formula (17) and formula (18) into formula (16) to obtain formula (21):

That is, there is formula (22):

In formula (22), V is a 2×6 matrix, which contains 6 unknown parameters.

"Solving the unknown parameters in matrix A" in S42 specifically includes:

Substituting _V into the homography matrix hi of the three images can solve the matrix b, and then get:

γ=-B ₁₂ α ² β/λ (26)

u ₀ =γv ₀ /β-B ₁₃ α ² /λ (27)

Thus, the internal parameter matrix A of the microlens array type light field camera is determined.

In one embodiment, "substituting the expressions of r ₁ and r ₂ into the expressions of r 1 and r 2 can solve r ₁ and r ₂ , and combining with H can solve the unknown parameters in the matrix [R t]" specifically includes:

Substituting matrix A into equation (15) can solve r ₁ , r ₂ :

r ₃ =r ₁ ×r ₂ (29)

Formula (30) can be obtained from formula (12):

t=λA ^-1 h ₃ (30)

In one embodiment, the "nonlinear least squares problem" in S44 is specifically as formula (31):

In formula (31):

m' _ij is the focus image point coordinate of the jth corner of the ith image;

根据式(8)由第i幅图像的第j个物点坐标向量M_j及A_i，R_i，t_i计算得到；

According to formula (8), it is calculated from the j-th object point coordinate vector M _j and A _i , R _i , t _i of the i-th image;

The initial values of A and [R t] are calculated from S41 to S43.

In one embodiment, in S5, after obtaining the internal and external parameter matrix of the microlens array type light field camera, since the parameters α and δ become known parameters, the distance b between the microlens array surface and the detector surface and the main lens are calculated. The distance L from the surface to the microlens array surface is as follows:

It can be seen from the similar triangular relationship in Figure 2 that the pupil diameter of the main lens and the diameter of the microlens image satisfy the following formula (32):

D/L=d/b (32)

Among them, D is the pupil diameter of the main lens, and d is the diameter of the microlens image. Formula (33) can be obtained from formula (7):

a=b/δ (33)

Substituting equations (32) and (33) into α=(L-a)/(dx) in equation (4), equations (34) and (35) can be obtained:

L=a+αdx (34)

b=αδddx/(δD-d) (35)

Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present invention, but not to limit them. It should be understood by those of ordinary skill in the art that the technical solutions described in the foregoing embodiments can be modified, or some of the technical features thereof can be equivalently replaced; these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the various aspects of the present invention. The spirit and scope of the technical solutions of the embodiments.

Claims (9)

1. a method for calibrating geometric parameters of a microlens array type light field camera based on a focused image point, is characterized in that, comprises the following steps: S1, according to the focused imaging optical path diagram of the microlens array light field camera, the mapping relationship between the object point and the focused image point on the main lens is obtained; S2, according to the focused imaging light path diagram of the microlens array light field camera, obtain the mapping relationship between the focused image point and the detector image point with respect to the microlens; S3, according to the detected image point of the detector, solve the coordinates of the focus image point; S4, according to the coordinates of the focused image point obtained in S3, solve the camera internal parameter matrix and external parameter matrix in the calibration model; S4 specifically includes: S41, set the calibration model plane z _w = 0 in the world coordinate system, substitute the object point in S1 and the focus image point in the mapping relationship with the main lens, and set: the object point is represented as M, and its coordinates are (x _w , y _w , z _w ); the focus image point is represented as m', and its coordinates are (u', v'). According to the coordinates of the object point M and the corresponding focus image point m', a 3×3 homography matrix H can be obtained , in order to solve the maximum likelihood estimation of the matrix H, iteratively solve the nonlinear least squares problem of the focused image points in the image; S42, according to the constraints of r ₁ and r ₂ , substitute the expressions of r ₁ and r ₂ obtained from the matrix H into the obtained equation, and then bring the symmetric matrix B and its calculation formula into the obtained equation, and finally obtain a 2× 6 matrix V, substituted into the homography matrix h _i of the three images can solve the matrix b, and then solve the unknown parameters in the matrix A; S43, the matrix A has been solved. Substituting the matrix A into the expressions of r ₁ and r ₂ can solve r ₁ and r ₂ , and combining with the matrix H, the unknown parameters in the matrix [R t] can be solved; S44, calibrate the n images, each image has m focus points, iteratively optimize the parameters obtained from S41 to S43, and solve the nonlinear least squares problem, so as to compare the parameters obtained from S41 to S43 perform nonlinear optimization; S5, calibrate the geometric parameters of the microlens array light field camera through the camera internal parameter matrix and external parameter matrix obtained in S4, the geometric parameters include the distance between the microlens array and the detector plane and the distance from the main lens to the microlens. 2. The method for calibrating geometric parameters of a microlens array light field camera based on a focused image point as claimed in claim 1, wherein the “mapping relationship between the object point and the focused image point on the main lens” in S1 is expressed as the formula (1): sm′=A[R t]M, (1) In formula (1), s is the scale factor; M=[x _w , y _w , z _w , 1] ^T ; m′=[u′, v′, 1] ^T ; [R t] represents the microlens array type The external parameter matrix of the light field camera, A represents the internal parameter matrix of the microlens array light field camera; [R t] is represented by the following formulas (3) and (2):

In formula (2) and formula (3), q ₁ , q ₂ , and q ₃ respectively represent the rotation angles of the three coordinate axes during the transformation of the camera coordinate system and the world coordinate system, and t _x , _ty , and t _z represent the world coordinates The distance that the origin of the system is translated along the three coordinate axes; A is represented by the following formula (4):

In formula (4), dx×dy is the pixel size of the detector, u ₀ , v ₀ are the coordinates of the center of the main lens in the image coordinate system, θ is the non-perpendicular tilt angle of the two coordinate axes, and L is the main lens plane The distance to the microlens array surface, a is the distance from the focusing imaging surface to the microlens array surface, α, β, γ are unknown parameters; Among them: the camera coordinate system takes the center of the main lens as the origin O, the x _c and y _c axes are parallel to the detector plane, and the z _c axis is perpendicular to the detector plane; the world coordinate system takes the center of the calibration object plane as the origin O _w , x The _w and y _w axes are parallel to the calibration object plane, and the z _w axis is perpendicular to the calibration object plane; the image coordinate system takes the center of the detector plane as the origin O _c , and the u and v axes are parallel to the x _c and y _c axes. 3. The method for calibrating geometric parameters of a microlens array light field camera based on a focused image point as claimed in claim 2, wherein the “mapping relationship between the focused image point and the detector image point with respect to the microlens” in S2 is: Equation (8), setting: the detector image point includes image point m ₁ and image point m ₂ , the coordinates of image point m ₁ in the image coordinate system are (u ₁ , v ₁ ), and the center of the corresponding microlens is at The coordinates of the image coordinate system are (p ₁ , q ₁ ); the coordinates of the image point m ₂ in the image coordinate system are (u ₂ , v ₂ ), and the coordinates of the corresponding microlens center in the image coordinate system are (p ₂ , q ₂ ); sm′=sT ⁻¹ m ₁ =A[R t]M (8) In formula (8): m ₁ represents the coordinate vector of the image point m ₁ ; T is the transformation matrix between the image point m' and the image point m ₁ ; m ₁ is represented as m ₁ =[u ₁ , v ₁ , 1] ^T ; T is represented by the following equations (6) and (7):

δ=(u ₁ -u ₂ )/(p ₁ -p ₂ )-1 (7). 4. The method for calibrating geometric parameters of a microlens array type light field camera based on a focused image point as claimed in claim 3, wherein S3 is specifically based on the detected image point of the detector, and utilizes the focus of the microlens array type The geometric relationship in the imaging optical path diagram is used to solve the coordinates of the focused image point; "Detected detector image points" are image points m ₁ , m ₂ ; "The geometric relationship in the focused imaging optical path diagram of the microlens array type" is expressed as formula (9):

Then formula (10) can be obtained according to formula (8):

First, substitute the coordinates of the image points m ₁ and m ₂ in the image coordinate system and the coordinates of the center of the corresponding microlens in the image coordinate system into formula (7), and calculate the δ value; Then, the calculated δ value is substituted into formula (9), and the coordinates (u', v') of the focus image point m' are solved. 5. The method for calibrating geometric parameters of a microlens array light field camera based on a focused image point as claimed in claim 1, wherein S41 specifically comprises: Set the calibration model plane z _w = 0 in the world coordinate system, and substitute it into equation (1), there is equation (11):

According to the coordinates of the object point M and the corresponding focus image point m', a 3×3 homography matrix H can be obtained in the following form: H=[h ₁ h ₂ h ₃ ]=λA[r ₁ r ₂ t] (12) In formula (12), h _i =[h _i1 , h _i2 , h _i3 ] ^T , λ is an arbitrary scalar; To solve the maximum likelihood estimate of the matrix H, iteratively solve the nonlinear least squares problem for the focused image points in the image:

In formula (13), m′ _i is the actual coordinate vector of the i-th focused image point in the same image,

According to formula (11) and formula (12), it is calculated from the object point _Mi. 6. The method for calibrating geometric parameters of a microlens array light field camera based on a focused image point as claimed in claim 5, wherein S42 specifically comprises: According to the properties of the rotation matrix, r ₁ and r ₂ in the matrix R satisfy the following two constraints:

From formula (12), it can be known that r ₁ =λA ^-1 h ₁ , r ₂ =λA ^-1 h ₂ (15) In formula (15), λ=1/||A ^-1 h ₁ ||=1/||A ^-1 h ₂ ||, and substituting formula (15) into formula (14) can get:

make

B is a symmetric matrix, and formula (18) can be obtained by calculation:

In formula (18), b=[B ₁₁ , B ₁₂ , B ₂₂ , B ₁₃ , B ₂₃ , B ₃₃ ] ^T (19) v _ij =[h _i1 h _j1 , h _i1 h _j2 +h _i2 h _j1 , h _i2 h _j2 , h _i3 h _j1 +h _i1 h _j3 , h _i3 h _j2 +h _i2 h _j3 , h _i3 h _j3 ] ^T (20) Substituting equations (17) and (18) into equation (16) can obtain equation (21):

That is, there is formula (22):

In formula (22), V is a 2 × 6 matrix, which contains 6 unknown parameters. Substitute into the homography matrix h _i of the three images to solve the matrix b, and then solve the unknown parameters in the matrix A:

γ=-B ₁₂ α ² β/λ (26) u ₀ =γv ₀ /β-B ₁₃ α ² /λ (27)

Thus, the internal parameter matrix A of the microlens array type light field camera is determined. 7. The method for calibrating geometric parameters of a microlens array type light field camera based on a focused image point as claimed in claim 6, wherein S43 specifically comprises: Substitute the matrix A obtained from S42 into formula (15) to obtain r ₁ and r ₂ , and the specific calculation expression is formula (29): r ₃ =r ₁ ×r ₂ (29) Formula (30) can be obtained from formula (12): t=λA ^-1 h ₃ (30). 8. The method for calibrating geometric parameters of a microlens array type light field camera based on a focused image point as claimed in claim 7, wherein S44 specifically comprises: Calibrate n images, each image has m focus points, and iteratively optimize the parameters obtained from S41 to S43 to solve the nonlinear least squares problem:

In formula (31), m′ _ij is the focus point coordinate of the jth corner of the ith image,

According to formula (8), the jth object point coordinate vector M _j and A _i , R _i , t _i of the ith image are calculated, and the initial values of A and [R t] are calculated from S41 to S43. 9. The method for calibrating geometric parameters of a microlens array type light field camera based on a focused image point as claimed in claim 1, wherein S5 specifically comprises: The pupil diameter D of the main lens and the diameter d of the microlens image satisfy the following formula: D/L=d/b (32) b is the distance between the microlens array surface and the detector surface; Formula (33) can be obtained from formula (7): a=b/δ (33) Substituting equations (32) and (33) into α=(L-a)/(dx) in equation (4), equations (34) and (35) can be obtained: L=a+αdx (34) b=αδddx/(δD-d) (35) In equations (34) and (35), the parameters α and δ are known parameters.

Images