CN115451866B - Three-dimensional measurement method of highly reflective surface based on light field equivalent camera array model - Google Patents

Abstract

Based on the traditional fringe projection 3D measurement technology, the present invention proposes a method based on a light field equivalent camera array model and combines a light field camera to perform 3D measurement of highly reflective surfaces. The light field equivalent camera array model of the present invention avoids the cumbersome calibration process and complex calibration results of the light field camera, and realizes fast and accurate calibration. The highly reflective surface 3D measurement method based on the light field equivalent camera array model of the present invention effectively solves the problem of information loss after 3D reconstruction of highly reflective surfaces.

Description

Three-dimensional measurement method of highly reflective surface based on light field equivalent camera array model

Technical Field

The present invention relates to a three-dimensional reconstruction method, and in particular to a three-dimensional reconstruction method for an object with a high-reflective area.

Background technique

Fringe projection profilometry is often used to measure the three-dimensional surface morphology of an object. The phase-shifted fringe pattern is projected onto the surface of the object and deformed, and then collected by the camera. The three-dimensional coordinate information of the corresponding point of the camera's pixel can be calculated through the phase information contained in the collected deformed fringe pattern. However, due to different surface processing methods and surface roughness of objects, specular reflection and diffuse reflection will occur when light is projected onto the surface of the object. Specular reflection is directional and is the main factor causing image oversaturation. This makes the fringe projection three-dimensional measurement method face the problem of image oversaturation, that is, the incident light intensity far exceeds the maximum grayscale value of the camera image sensor. As a result, there are large errors in the subsequent phase demodulation and phase unfolding process, which leads to phase disorder, and ultimately introduces too many errors or excessive sparseness to the calculated point cloud data.

Summary of the invention

Light field imaging technology can simultaneously record the position information and direction information of light. Therefore, compared with traditional cameras, light field cameras can capture four-dimensional light field information. By extracting the four-dimensional light field information, multi-perspective images of the scene, namely sub-aperture images, are obtained. The light information collected by sub-aperture images of different viewing angles for the same world point is not the same, which makes the light field sub-aperture images have certain differences and complementarities. Theoretically, the scene information lost in the image information recorded by a certain sub-aperture due to overexposure can be supplemented by sub-aperture images at other viewing angles. Therefore, based on the traditional fringe projection three-dimensional measurement technology, the present invention proposes a high-reflective surface three-dimensional measurement method and system based on a light field equivalent camera array model, which effectively solves the problem of information loss after three-dimensional reconstruction of high-reflective surfaces.

According to a first aspect of an embodiment of the present invention, a light field equivalent camera array model is proposed, which avoids the cumbersome calibration process and complex calibration results of the light field camera and achieves fast and accurate calibration.

According to a second aspect of an embodiment of the present invention, a method for three-dimensional measurement of a highly reflective surface is proposed, wherein a calibrated light field camera is used to perform multi-view three-dimensional reconstruction of the highly reflective surface, thereby obtaining multi-directional point cloud information of the highly reflective surface at different viewing angles, wherein the light field camera is regarded as an equivalent camera array consisting of a main camera and an auxiliary camera, and the central viewing angle in the sub-aperture image is selected as the viewing angle of the main camera, and the imaging of the auxiliary camera is regarded as the imaging of other viewing angles except the central viewing angle, and the coordinate systems of all auxiliary cameras are unified into the coordinate system of the main camera; the point cloud reconstructed under the main camera is selected as the target point cloud, and the point cloud reconstructed under the remaining auxiliary cameras is selected as the target point cloud. The point cloud is used as an auxiliary point cloud. All reconstructed point clouds are unified into the main camera coordinate system, and then all reconstructed point clouds unified into the main camera coordinate system are projected onto the XOY plane. After obtaining the projection image, the pixel position and size of the missing part in each point cloud projection image are extracted through connected domain analysis, and the different-view point cloud information that is meaningful for repairing the target point cloud is determined. Then, complementary point clouds are extracted from the different-view point clouds and spliced to the corresponding area of the target point cloud. In this way, the target point cloud is iteratively repaired until the information lost due to overexposure is completely restored, thereby realizing complete three-dimensional measurement of objects on highly reflective surfaces.

According to a third aspect of an embodiment of the present invention, a high-reflective surface 3D measurement system is proposed, comprising: a light field camera; and a 3D reconstruction module, which is configured to use a calibrated light field camera to perform multi-view 3D reconstruction on the high-reflective surface, thereby obtaining multi-directional point cloud information of the high-reflective surface under different viewing angles, wherein the light field camera is regarded as an equivalent camera array composed of a main camera and an auxiliary camera, the central viewing angle in the sub-aperture image is selected as the viewing angle of the main camera, the imaging of the auxiliary camera is regarded as imaging of other viewing angles except the central viewing angle, the coordinate systems of all auxiliary cameras are unified into the coordinate system of the main camera, the point cloud reconstructed under the main camera is selected as the target point cloud, and the remaining The point cloud reconstructed under the auxiliary camera is used as the auxiliary point cloud. All reconstructed point clouds are unified under the main camera coordinate system, and then all reconstructed point clouds unified under the main camera coordinate system are projected to the XOY plane. After obtaining the projection image, the pixel position and size of the missing part in each point cloud projection image are extracted through connected domain analysis, and the different-viewing angle point cloud information that is meaningful for repairing the target point cloud is determined. Then, complementary point clouds are extracted from the different-viewing angle point clouds and spliced to the corresponding area of the target point cloud. In this way, the target point cloud is iteratively repaired until the information lost due to overexposure is completely restored, thereby realizing complete three-dimensional measurement of objects on highly reflective surfaces.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solution of the embodiment of the present invention, the drawings of the embodiment are briefly introduced below.

FIG. 1( a ) is a schematic diagram of a light field camera model provided by an embodiment of the present invention.

FIG. 1( b ) is a schematic diagram of an equivalent camera array model provided by an embodiment of the present invention.

FIG. 2 is a schematic diagram of a three-dimensional measurement system based on a light field equivalent array model provided by an embodiment of the present invention.

FIG. 3 is a schematic diagram of sinusoidal phase-shifted fringes provided by an embodiment of the present invention.

FIG. 4 is a schematic diagram of a light field equivalent camera array three-dimensional measurement system provided by an embodiment of the present invention.

FIG5 is a schematic diagram of point cloud adaptive restoration provided by an embodiment of the present invention.

FIG6 is a schematic diagram of surface reconstruction of a metal cylinder provided by an embodiment of the present invention, wherein (a) the original light field image, (b) the traditional fringe projection 3D reconstruction result, and (c) the 3D reconstruction result of the present invention.

Detailed ways

The sub-aperture image extracted from the original light field image is an image of the scene from different perspectives. Therefore, the focusing light field camera can be equivalent to a camera array, so that the four-dimensional light field L(s, t, u, v) can be simplified into a two-dimensional L(u, v) array, and the angular resolution (s, t) of the light field camera is used for positioning on the array. Based on this, the present invention proposes an equivalent camera array model based on a focusing light field camera. In order to calibrate the focusing light field camera, the focusing light field camera with unknown structure is regarded as an equivalent camera array composed of a main camera and an auxiliary camera, and the central perspective in the sub-aperture image is selected as the perspective of the main camera. Considering the extraction process of the sub-aperture image, all cameras in the model can be regarded as having the same intrinsic parameter matrix, and the different perspectives of the sub-aperture image of the light field camera are reflected in the posture changes of the camera. The imaging of the auxiliary camera can be regarded as imaging of other perspectives except the central perspective. Therefore, the perspective projection transformation from the world point _Pw = (x, y, z) ^T on the main camera image plane to the point _mm = ( _um , _vm ) ^T on the main camera plane and the point _ma = ( _ua , _va ) ^T on the auxiliary camera image plane can be expressed as:

in, represents homogeneous coordinates, s _m and _sa are scale factors, _AC is the intrinsic matrix of the camera, [R _m |T _m ] and [R _a |T _a ] are the extrinsic matrices of the main camera and the auxiliary camera, respectively.

In order to avoid inconsistent reconstruction results of the same world point by the main camera and the auxiliary camera, the coordinate systems of all auxiliary cameras need to be unified into the main camera coordinate system. Therefore, the main camera coordinate system defaults to the equivalent camera array coordinate system. In this case, the position and posture of the auxiliary camera can be represented by the external parameter matrix of the main camera. At the same time, the change of posture from the auxiliary camera to the main camera is a rigid body transformation process. Therefore, the rigid body transformation from the auxiliary camera coordinate system to the main camera coordinate system can be derived, which can be expressed as:

Among them, [ _Rs | _Ts ] is the structural parameter of the auxiliary camera, which represents the rigid body transformation matrix from the auxiliary camera coordinate system to the main camera coordinate system. _Pm and _Pa are the three-dimensional coordinates of a point in the main camera coordinate system and the auxiliary camera coordinate system, respectively. In order to identify the different postures of the auxiliary camera, C(i, j) is used to represent the camera located in the i-th row and j-th column of the equivalent camera array, _Pa,ij represents the coordinates in the C(i, j) camera coordinate system, and _ma,ij is the coordinates in the corresponding image pixel coordinate system. In the main camera coordinate system, the transformation between the main camera and the auxiliary camera can be expressed as follows:

Where s _a,ij is the scale factor of the equivalent camera in the i-th row and j-th column, I is the unit matrix, and R _s,ij and T _s,ij are the structural parameters of C(i, j). The _Pa,ij calculated under different auxiliary camera viewing angles are unified into the main coordinate system to obtain the equivalent camera array model of the focusing light field camera, as shown in Figure 1.

A high-reflective surface three-dimensional measurement system based on the light field equivalent camera array model as shown in Figure 2 was built, including a DLP4500 structured light projector, a Raytrix R8 light field camera, a host computer, etc. Three sets of phase-shifted sinusoidal fringe patterns with a resolution of 912×1140 were generated using MATLAB programming, with frequencies of 15, 12, and 10, respectively. Each set includes 4 horizontal and vertical phase-shifted fringes, and the phase difference between adjacent phase-shifted fringe patterns is π/2, totaling 24 sinusoidal fringe patterns, as shown in Figure 3. The prepared checkerboard calibration plate is placed at different positions in the field of view of the light field camera, trying to cover different angles. At each position, the projector projects 24 sinusoidal fringe patterns. The calibration plate is placed in 10 different positions, and 10 sets of sinusoidal fringe pattern sequences can be obtained. Each set of patterns contains a checkerboard image and 24 sinusoidal fringe patterns, for a total of 25 patterns. After the image acquisition is completed, the original light field data collected by the light field camera is decoded according to the light field sub-aperture image extraction algorithm. In order to ensure the accuracy of the three-dimensional reconstruction of the measurement system, it is necessary to balance the angular resolution and spatial resolution of the light field camera. Finally, the angular resolution is adjusted to 6×6, that is, 36 sub-aperture images of different viewing angles can be extracted from the original image, and the spatial resolution is 1430×784, that is, the resolution of the sub-aperture image extracted from the original light field image is 1430×784. The collected calibration patterns are decoded to obtain 36 groups of patterns, each group of patterns is a calibration pattern collected at each viewing angle, with a total of 250 patterns. The open source calibration software is used for calibration to obtain the internal and external parameters of each equivalent camera, the camera array structure parameters, and the internal and external parameters of the projector.

After the equivalent camera array model and the projector intrinsic and extrinsic matrix are obtained through calibration, the three-dimensional reconstruction results of the object under test at different viewing angles in a unified coordinate system can be obtained by the following formula:

Where, E _ij is the structural parameter of C(i, j) in the equivalent camera array, s _p is the projector scale factor, and s _c,ij is the equivalent camera scale factor in the i-th row and j-th column. _{A p} is the intrinsic matrix of the projector, [R _p |T _p ] is the extrinsic parameter matrix of the projector, u _p is the horizontal coordinate of the corresponding point in the projector pixel coordinate system, Φ _i (u _c ,v _c ) is the absolute phase value at the point (u _c ,v _c ), W is the projection fringe width, and N is the maximum fringe period integer. P _c =R _c P _w +T _c is the coordinate in the camera coordinate system, and R _c ,T _c are the extrinsic parameters of the camera. The main camera in the camera array is the camera under the central perspective, so the main camera coordinate system is selected as the coordinate system of the measurement system, then E _ij can be defined as:

Among them, (s, t) is the angular resolution of the focusing light field camera. Therefore, the focusing light field can be used for multi-view 3D reconstruction, and R _s,ij ,T _s,ij represent the extrinsic parameters of the equivalent camera in the i-th row and j-th column. Thus, the multi-directional point cloud information of the measured object under different viewing angles is obtained, as shown in Figure 4.

In the measurement system, due to the presence of highly reflective areas on the surface of the measured object, the point cloud reconstructed from multiple perspectives may be incomplete, that is, the image is overexposed, resulting in holes in the reconstructed point cloud. The light intensity recorded by the camera from the same point in the world coordinate system at different perspectives is not equal, so the position and size of the lost information in the different perspective point cloud reconstructed from the highly reflective surface object are different. In other words, the incomplete point cloud can be repaired by the point cloud with complementary information reconstructed from other perspectives, as shown in Figure 5.

In the light field equivalent camera array measurement system, the point cloud {P} _m reconstructed under the main camera is used as the target point cloud, and the rest are used as auxiliary point clouds {P} _a . In order to avoid redundant calculations, all reconstructed point clouds are unified to the main camera coordinate system, and then the point clouds are projected to the XOY plane. The projection image can be obtained by the following formula:

Among them, _Bm represents the binary projection image obtained by projecting the target point cloud, and the zero pixel area represents the missing area of the point cloud; _Ba represents the binary projection image obtained by projecting the source point cloud; _Ωz represents the projection of the point cloud onto the XOY plane. After obtaining the projection image, the pixel position and size of the missing part in each point cloud projection image can be extracted through connected domain analysis, so as to determine the different-viewpoint point cloud information that has the significance of repairing the target point cloud, which can be expressed as:

Λ＝Ψ(B _m )ΔΨ(B _a )

Among them, Ψ represents the connected domain analysis, and AΔB represents the extraction of the coordinate set Λ from the point set B according to the position of the zero pixel in the point set A. Then, the complementary point cloud can be extracted from the different-view point cloud and spliced to the corresponding area of the target point cloud. The repaired point cloud {P} _re can be expressed as:

Among them, δ _Λ represents the point cloud extracted according to the coordinate set Λ, Represents point cloud stitching. Repeat the above steps to iteratively repair the target point cloud until the information lost due to overexposure is completely restored, so as to perform complete three-dimensional measurement of highly reflective surface objects using the light field equivalent camera array measurement system.

The above processing of the point cloud can be implemented by a three-dimensional reconstruction module configured on the host computer. When the three-dimensional reconstruction module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

Taking a polished metal cylinder as an example, the original light field image is shown in FIG6(a), the traditional fringe projection 3D reconstruction method is shown in FIG6(b), and the reconstruction result of the present invention is shown in FIG6(c).

Claims (2)

1. The three-dimensional measurement method of the high reflection surface is characterized in that a calibrated light field camera is utilized to carry out multi-view three-dimensional reconstruction on the high reflection surface, so that multi-directional point cloud information of the high reflection surface under different view angles is obtained, the light field camera is regarded as an equivalent camera array consisting of a main camera and an auxiliary camera, a central view angle in a sub-aperture image is selected as the view angle of the main camera, imaging of the auxiliary camera is regarded as imaging of other view angles except the central view angle, and a coordinate system of all the auxiliary cameras is integrated into a main camera coordinate system;

The method comprises the steps of selecting point clouds reconstructed under a main camera as target point clouds, reconstructing the other auxiliary cameras to obtain the point clouds as auxiliary point clouds, unifying all reconstructed point clouds under a main camera coordinate system, projecting all reconstructed point clouds unified under the main camera coordinate system onto an XOY plane to obtain projection images, analyzing and extracting through a connected domain to obtain pixel positions and sizes of missing parts in each point cloud projection image, determining the information of the point clouds with restoration significance on the target point clouds, extracting complementary point clouds from the point clouds, splicing the complementary point clouds to the corresponding areas of the target point clouds, and iteratively restoring the target point clouds until the lost information due to overexposure is completely restored, so that the complete three-dimensional measurement work of the high-reflection surface object is realized.

2. A highly reflective surface three-dimensional measurement system, comprising:

A light field camera;

The three-dimensional reconstruction module is configured to reconstruct the high reflection surface in a multi-view manner by using a calibrated light field camera, so as to obtain multi-directional point cloud information of the high reflection surface under different view angles, the light field camera is regarded as an equivalent camera array formed by a main camera and an auxiliary camera, a central view angle in a sub-aperture image is selected as the view angle of the main camera, imaging of the auxiliary camera is regarded as imaging of other view angles except the central view angle, coordinate systems of all the auxiliary cameras are integrated into the coordinate systems of the main camera, point clouds are reconstructed under the main camera are selected to obtain point clouds as target point clouds, the point clouds are reconstructed under the other auxiliary cameras to obtain the point clouds as auxiliary point clouds, all the reconstructed point clouds are integrated into the main camera coordinate systems, all the reconstructed point clouds under the main camera coordinate systems are projected onto an XOY plane, after a projected image is obtained, the pixel position and the size of a missing part in each point cloud projected image are obtained through connected domain analysis, the point cloud information with a restoration meaning for the target point clouds is determined, and the point clouds are extracted from the point clouds of different view points, and the point clouds are complementary point clouds are spliced to the corresponding areas of the target point clouds, so that the point clouds are completely restored, and the target point clouds is completely restored, and the three-dimensional reflection surface is completely, and the reflection object is completely restored, and the reflection object is completely and completely restored.