CN116310120A - Multi-view 3D reconstruction method, device, equipment and storage medium - Google Patents

Abstract

The application provides a multi-view three-dimensional reconstruction method, device, equipment and storage medium, and relates to the technical field of three-dimensional reconstruction. The method comprises the following steps: acquiring two-dimensional images of a target object under a plurality of view angles; respectively determining point cloud data under a plurality of view angles according to the two-dimensional images under the plurality of view angles; sampling point clouds in a preset voxel space in point cloud data under a plurality of view angles based on the preset voxel space to obtain target voxel points; determining a three-dimensional surface of the target object according to the target voxel point; determining an intersection point interval of each sampling ray and the three-dimensional surface according to the camera pose under each view angle; determining the surface color of the target object based on each intersection point interval by adopting a color network; the target object is rendered based on the surface color. Compared with the prior art, a large amount of redundant and invalid calculation is avoided, and the efficiency and the accuracy of three-dimensional reconstruction are improved.

Description

Multi-view 3D reconstruction method, device, equipment and storage medium

technical field

The present application relates to the technical field of three-dimensional reconstruction, and in particular, relates to a multi-view three-dimensional reconstruction method, device, equipment, and storage medium.

Background technique

Multi-view 3D reconstruction is a technique for reconstructing the shape of a three-dimensional (3D) object by taking pictures of the object to be reconstructed from different perspectives. In recent years, with the development of 3D games, architectural design, and virtual reality, the demand for 3D reconstruction of real-world objects is increasing. At present, the development of 3D reconstruction based on deep learning is rapid.

The prior art generally uses a neural surface reconstruction method using a signed distance function (Signed Distance Function, SDF). This method obtains the initial bounding box (BBox) of the object based on the sparse point cloud calculated by the Structure From Motion (SFM) method. Based on the camera pose estimation results, the observed sampling ray is designed and effectively sampled inside the BBox. For each target voxel point, the SDF value and the color at the target voxel point are predicted by a multilayer perceptron (MLP). Then, through the Logistic probability density distribution function, the SDF value is converted into a volume density value, combined with the predicted color of each target voxel point, and the neural volume rendering method is used to train the SDF representation.

However, in this reconstruction method, because the actual object to be reconstructed only occupies a part of the BBox, there are a lot of redundancy and invalid calculations based on the ray sampling of the entire BBox, which reduces the efficiency of optimization and leads to a decline in the final reconstruction effect.

Contents of the invention

The purpose of the present application is to provide a multi-view 3D reconstruction method, device, equipment and storage medium for the deficiencies in the above-mentioned prior art, so as to solve the problem of a large number of redundant and invalid calculations in the prior art, which reduces the optimization efficiency and leads to A problem with reduced rebuild performance.

In order to achieve the above purpose, the technical solution adopted in the embodiment of the present application is as follows:

In the first aspect, an embodiment of the present application provides a multi-view 3D reconstruction method, the method comprising:

Acquire two-dimensional images of the target object under multiple viewing angles;

According to the two-dimensional images under the multiple viewing angles, respectively determine the point cloud data under the multiple viewing angles;

Based on the preset voxel space, sampling the point cloud in the preset voxel space in the point cloud data under multiple viewing angles to obtain the target voxel point;

determining a three-dimensional surface of the target object according to the target voxel points;

Determining the intersection interval between each sampling ray and the three-dimensional surface according to the camera pose under each of the viewing angles;

Using a color network to determine the surface color of the target object based on each of the intersection intervals;

Render the target object based on the surface color.

In the second aspect, another embodiment of the present application provides a multi-view 3D reconstruction device, the device includes: an acquisition module, a determination module, a sampling module and a rendering module, wherein:

The acquiring module is configured to acquire two-dimensional images of the target object under multiple viewing angles;

The determining module is configured to respectively determine point cloud data under the multiple viewing angles according to the two-dimensional images under the multiple viewing angles;

The sampling module is configured to, based on a preset voxel space, sample point clouds in the preset voxel space in the point cloud data under multiple viewing angles to obtain target voxel points;

The determining module is specifically configured to determine the three-dimensional surface of the target object according to the target voxel points; determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera poses under each of the viewing angles; The color network determines the surface color of the target object based on each of the intersection intervals;

The rendering module is configured to render the target object based on the surface color.

In the third aspect, another embodiment of the present application provides a multi-view 3D reconstruction device, including: a processor, a storage medium, and a bus, and the storage medium stores machine-readable instructions executable by the processor. When the perspective three-dimensional reconstruction device is running, the processor communicates with the storage medium through a bus, and the processor executes the machine-readable instructions to perform the steps of any one of the methods described in the first aspect above.

In a fourth aspect, another embodiment of the present application provides a storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the method described in any one of the above-mentioned first aspects are executed.

The beneficial effect of the present application is: adopt the multi-view three-dimensional reconstruction method provided by the present application to obtain two-dimensional images of the target object under multiple viewing angles, and then determine the point cloud data under multiple viewing angles based on the multiple two-dimensional images, and Based on the preset voxel space, sample the point cloud in the preset voxel space, determine the target voxel point, determine the three-dimensional surface of the target object based on the target voxel point, and then determine the surface of the target object based on the sampling ray When using color, first determine the intersection interval between each sampling ray and the three-dimensional surface, and then determine the color of each sampling ray in the focus area based on the color network, thereby determining the surface color of the target object. This method of determination can reduce the size of the sampling ray. Sampling range, so as to improve the efficiency of determining the surface color of the target object, and improve the rendering efficiency of the target object.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings that are required in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and thus It should be regarded as a limitation on the scope, and those skilled in the art can also obtain other related drawings based on these drawings without creative work.

FIG. 1 is a schematic flowchart of a multi-view three-dimensional reconstruction method provided by an embodiment of the present application;

FIG. 2 is a schematic flowchart of a multi-view 3D reconstruction method provided by another embodiment of the present application;

FIG. 3 is a schematic flowchart of a multi-view three-dimensional reconstruction method provided by another embodiment of the present application;

FIG. 4 is a schematic flowchart of a multi-view 3D reconstruction method provided by another embodiment of the present application;

FIG. 5 is a schematic structural diagram of a multi-view 3D reconstruction device provided by an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a multi-view 3D reconstruction device provided by another embodiment of the present application;

FIG. 7 is a schematic structural diagram of a multi-view three-dimensional reconstruction device provided by an embodiment of the present application.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments It is a part of the embodiments of this application, not all of them.

The components of the embodiments of the application generally described and illustrated in the figures herein may be arranged and designed in a variety of different configurations. Accordingly, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative efforts belong to the scope of protection of the present application.

Additionally, the flowcharts used in this application illustrate operations implemented in accordance with some embodiments of the application. It should be understood that the operations of the flowcharts may be performed out of order, and steps that have no logical context may be performed in reverse order or concurrently. In addition, those skilled in the art may add one or more other operations to the flowchart or remove one or more operations from the flowchart under the guidance of the content of the present application.

For the convenience of understanding the application, some nouns designed in the application are explained below:

Salient object detection (SOD): Aims to distinguish the most visually salient regions. Use traditional computer vision techniques or data-driven deep learning algorithms to mark parts of the image that are salient foregrounds.

The implicit expression of the signed distance function (Signed Distance Function, SDF) three-dimensional object surface determines the distance from a point to the object surface on a limited area in space and defines the sign of the distance at the same time: the point is positive inside the object, Negative outside, 0 when on the surface. SDFs can be implicitly represented by means of a multi-layer perceptron (MLP) network.

Bounding Box (BoundingBox, BBox), refers to the smallest bounding box surrounding an object in 3D space.

Structure From Motion (SFM) is a technique for estimating three-dimensional structures from a series of multiple two-dimensional images containing visual motion information. Using SFM, camera parameters and sparse point clouds of each view can be obtained.

Marching Cubes: Surface extraction algorithm, also known as surface rendering algorithm, is an algorithm for extracting isosurfaces in volume data. The core concept is to determine which sides of the voxel intersect with the isosurface by comparing the vertices with the user-specified threshold after setting the 3D space and vertex scalar values, create triangular patches, and connect the isosurface boundaries All the facets of the cube to get a surface.

A method for multi-view 3D reconstruction provided in an embodiment of the present application may be executed by a terminal or a server. A terminal is any device having computing hardware capable of supporting and executing a corresponding software product. A multi-view three-dimensional reconstruction method provided in the embodiment of the present application is explained below in conjunction with multiple specific application examples. Fig. 1 is a schematic flowchart of a multi-view 3D reconstruction method provided by an embodiment of the present application. As shown in Fig. 1, the method includes:

S101: Acquire two-dimensional images of a target object under multiple viewing angles.

In the embodiments of the present application, in some possible embodiments, the multiple viewing angles may be, for example, 50-80 viewing angles. Multiple viewing angle images are sufficient for the 360° surround view of the target object, or any integer number less than 50, or any integer number greater than 80. The specific number of multiple viewing angles can be limited. It can be flexibly adjusted according to user needs, and is not limited to those given in the above-mentioned embodiments.

In some possible embodiments, the manner of acquiring two-dimensional images under multiple viewing angles may be, for example: acquiring multiple two-dimensional images obtained by shooting a target object in a real environment with a preset camera under multiple viewing angles, the multiple Each two-dimensional image corresponds to multiple viewing angles.

S102: According to the two-dimensional images under multiple viewing angles, respectively determine point cloud data under multiple viewing angles.

In some possible embodiments, for example, multiple two-dimensional images may be estimated separately to obtain three-dimensional point cloud data under multiple viewing angles, wherein the three-dimensional point cloud under each viewing angle may be, for example, a sparse point cloud. As an example, the preset SFM network can be used to estimate the 2D image at each viewing angle, to obtain the 3D point cloud data at each viewing angle, and the camera poses at multiple viewing angles.

S103: Based on the preset voxel space, sampling point clouds in the preset voxel space in the point cloud data under multiple viewing angles to obtain target voxel points.

In one embodiment of the present application, before S103, for example, according to the point cloud data under multiple viewing angles, the information of the bounding box of the target object can be determined, wherein the information of the bounding box of the target object is the minimum bounding box surrounding the target object Information.

The method of determining the bounding box of the target object can be, for example, as follows: the camera pose at each view angle can be calibrated according to the 3D point cloud data at each view angle, and the target camera pose at each view angle can be obtained, and updated according to The target camera pose estimates the bounding box of the target object, thereby obtaining the bounding box information of the target object; that is, the frame of the target object is estimated based on the calibrated target camera pose under multiple viewing angles in the embodiment of the present application. Estimation, the information of the bounding box of the target object is obtained, and the automatic estimation of the bounding box of the target object is realized, which can ensure that the estimated bounding box of the target object is more compact.

For the information of the bounding box of the target object, in some possible embodiments, the estimation results of the above-mentioned SFM on the two-dimensional images of multiple viewing angles can also be used to perform automatic object frame estimation, so as to obtain the information of the bounding box of the target object .

In the embodiment of the present application, after determining the bounding box BBox of the target object, it is necessary to first normalize the coordinates of the bounding box of the target object in the three-dimensional space, and normalize to [-1, 1] to obtain the normalized The normalized point cloud data can be obtained as follows:

i∈(x,y,z)。Normalize the coordinates of the above BBox in the three-dimensional space to [-1,1], and calculate the scaling matrix as scale_mat∈R ^4×4 . Among them, B ∈ R ³ ;

i∈(x,y,z).

R ^4×4 represents a 4x4 matrix, R ³ represents a three-dimensional vector space, and c _x , _cy , and c _z represent offsets in the x, y, and z directions. B _x , B _y , B _z represent the coordinate values of the BBOX in the x, y, and z directions.

In the method provided by the present application, the bounding box of the target object is calculated and estimated automatically instead of artificially, so the application complexity in the three-dimensional reconstruction process is reduced.

In the embodiment of the present application, firstly, the preset voxel space needs to be initialized, and the initialization method may be, for example, setting the number of voxels sampled in the preset voxel space according to the target accuracy of the target object.

Wherein, the target precision represents the fineness of the three-dimensional model of the object to be obtained, and different finenesses correspond to different target precisions, and the higher the fineness, the higher the target precision. For example, for example, medium fineness corresponds to sampling in a manner of downsampling 100 voxels in each dimension in a three-dimensional space. For example, based on the preset voxel space, according to the information of the bounding box of the target object, the point cloud in the preset voxel space in the normalized point cloud data under multiple viewing angles can be sampled to determine the correspondence between multiple viewing angles. target voxel point.

定义每个体素属于物体内部的计数数组为O∈[-1,1]³。Initialize the preset voxel space V∈[-1,1] ³ in the three-dimensional space, and set the number of voxels sampled in each dimension to ρ, where the size of ρ is determined according to the target accuracy of the target object , the higher the target accuracy, the larger the ρ setting, the lower the target accuracy, the smaller the ρ setting, the coordinates in each dimension in the three-dimensional space

Define the count array that each voxel belongs to inside the object as O∈[-1,1] ³ .

其中横纵坐标表示为/>

p＝(p₀,p₁,p₂)。则/>

For the i-th viewing angle, assume that the currently calculated transformation matrix from the world coordinate system to the current 2D image is w2c_mat _i ∈R ^3×4 . For each point (i,j,k) in V∈[-1,1] ³ , the coordinates projected to the current 2D image are

The horizontal and vertical coordinates are expressed as />

p = (p ₀ , p ₁ , p ₂ ). Then />

R ^3×4 represents a 3x4 matrix, and p ₀ , p ₁ , and p ₂ represent the components of P in the x, y, and z directions. O _{i, j, k} represents the component of the count array O whose coordinates are (i, j, k) in the three-dimensional space.

S104: Determine the three-dimensional surface of the target object according to the target voxel points.

In an embodiment of the present application, the manner of determining the three-dimensional surface of the target object may be, for example, using an isosurface extraction algorithm to determine the three-dimensional surface of the target object based on the target voxel points.

则认为该体素点属于物体内部，否则输入物体外部。其中阈值γ∈(0,1)。根据等值面提取Marching cubes算法提取属于物体内部的体素点构成的三维表面(3D凸包)。In the embodiment of the present application, the above-mentioned process of determining the target voxel point is repeated for the two-dimensional images under all viewing angles, and the final count array O is obtained. For each position (i,j,k) in O, if

It is considered that the voxel point belongs to the inside of the object, otherwise it is input outside the object. where the threshold γ∈(0,1). According to the isosurface extraction Marching cubes algorithm, the three-dimensional surface (3D convex hull) composed of voxel points belonging to the interior of the object is extracted.

S105: Determine an intersection interval between each sampling ray and the three-dimensional surface according to the camera poses at each viewing angle.

(假设该射线为经过相机原点和2D图像中(i,j)的直线)，则定义中心点到相机原点的距离为/>

因此可以得到射线与第1步中计算得到的BBox的近交点/>

远交点为

此外如果射线与3D凸包存在交点，并记o_n与3D凸包的第一个交点为

o_n相对于/>

的对称点为/>

并记其与3D凸包的第一个交点为/>

则远交点/>

当射线与3D凸包无交点时，使用射线与BBox的交点作为/>

和/>

即如果射线d_i,j与3D凸包无交点，则使用下式替换：For the camera pose at each viewing angle, calculate the surface intersection point of each sampling ray and the 3D convex hull of the object calculated above. Assuming that the coordinates of the nth camera origin in the world coordinate system are o _n , the unit vector of the ray to be sampled is

(Assuming that the ray is a straight line passing through the origin of the camera and (i,j) in the 2D image), the distance from the center point to the origin of the camera is defined as />

Therefore, the close intersection point of the ray and the BBox calculated in step 1 can be obtained />

far node is

In addition, if there is an intersection point between the ray and the 3D convex hull, and the first intersection point between o _n and the 3D convex hull is

o _n relative to />

The symmetry point of

And record its first intersection with the 3D convex hull as />

then far node />

When the ray has no intersection with the 3D convex hull, use the intersection of the ray and the BBox as />

and />

That is, if the ray d _{i, j} has no intersection with the 3D convex hull, use the following formula to replace:

和远交点/>

So far, the effective near intersection point of each sampled ray is obtained

and far node />

If the sampling ray is tangent to the target object, that is, the first intersection point and the second intersection point of the sampling ray coincide.

S106: Using the color network to determine the surface color of the target object based on each intersection interval.

Among them, the color network is a pre-trained network model, which is constructed using a Multilayer Perceptron (MLP) network, and network parameters are obtained through training. The output of the network is the predicted color value of each point in the space.

That is to say, the data and surface color of the 3D surface of the target have been obtained so far, that is, the 3D model data of the target object has been obtained, and the target object can be directly rendered in the 3D world based on the obtained 3D model data.

S107: Render the target object based on the surface color.

After obtaining the 3D model data of the target object, the 3D model data can be stored, and according to the subsequent actual business needs, after opening and loading the 3D model data through the model rendering software, the rendering and display of the 3D model corresponding to the target object can be realized .

In the 3D reconstruction method provided by the embodiment of the present application, in the process of collecting the target voxel points in the 3D point cloud, the target object is obtained by estimating the bounding boxes of multiple 2D images under multiple viewing angles. Therefore, compared with the artificially estimated bounding box, the automatic estimation method of this application can realize the estimation of the compact object bounding box, so that The sampling of the 3D point cloud is made more accurate, and then the 3D model data obtained by 3D reconstruction can be made more accurate, thereby improving the reconstruction accuracy of the 3D model and the integrity of the model details.

Using the multi-view 3D reconstruction method provided by the present application, two-dimensional images of the target object under multiple viewing angles are obtained, and then the point cloud data under multiple viewing angles and the information of the bounding box of the target object are respectively determined based on the multiple two-dimensional images, And based on the preset voxel space, the point cloud in the preset voxel space is sampled, the target voxel point is determined, the three-dimensional surface of the target object is determined based on the target voxel point, and then the target object is determined based on the sampling ray For the surface color, first determine the intersection interval between each sampling ray and the three-dimensional surface, and then determine the color of each sampling ray in the focus area based on the color network, thereby determining the surface color of the target object. This method of determination can reduce the sampling ray The sampling range, thereby improving the efficiency of determining the surface color of the target object, and improving the rendering efficiency of the target object.

Optionally, on the basis of the above embodiments, the embodiments of the present application may also provide a multi-view three-dimensional reconstruction method. The implementation process of determining the surface color of the target object in the above method is illustrated as follows with reference to the accompanying drawings. Fig. 2 is a schematic flowchart of a multi-view three-dimensional reconstruction method provided by another embodiment of the present application. As shown in Fig. 2, S106 may include:

S111: According to the signed distance function network, determine the signed distance function value of the target voxel point, the normal vector, and the distance between the target voxel points.

在/>

和

之间进行3D采样，通过上述SDF网络确定各目标体素点的符号距离函数值、法向量以及目标体素点间距。In the embodiment of this application, for the sampling ray at the nth viewing angle

at />

and

3D sampling is performed between them, and the signed distance function value, normal vector and target voxel point spacing of each target voxel point are determined through the above SDF network.

S112: Determine the color value along each sampling ray according to the color network.

在

和/>

之间进行3D采样，通过颜色网络，确定沿着每条采样射线的颜色值。Similarly, when determining the color value of each sampling ray, it is also for the sampling ray at the nth viewing angle

exist

and />

3D sampling is carried out between, and the color value along each sampling ray is determined through the color network.

S113: Obtain the target rendering color value of each sampling ray.

In the embodiment of the present application, the manner of determining the target rendering color value of each sampling ray may be, for example, converting the weighted SDF value into a volume density value through a Logistic cumulative distribution function. And perform volume rendering according to the volume density value, color value and sampling interval of all sampling points on each ray, and obtain the final rendered color value of each ray.

Optionally, on the basis of the above-mentioned embodiments, the embodiments of the present application may also provide a multi-view three-dimensional reconstruction method, and the implementation process of determining the target voxel point in the above-mentioned method is illustrated as follows with reference to the accompanying drawings. Fig. 3 is a schematic flowchart of a multi-view three-dimensional reconstruction method provided by another embodiment of the present application. As shown in Fig. 3, S103 may include:

S121: Determine whether each pixel in the two-dimensional image under multiple viewing angles is a salient foreground.

A salient foreground is an object instance with a clear outline in a two-dimensional image; first, this application can perform saliency detection based on a salient foreground detection algorithm. Generally, an object surrounded by high-frequency details in a two-dimensional image is an object instance. That is, one or more object instances with clear outlines in the two-dimensional images under multiple viewing angles are detected through saliency detection, and then, whether the pixel position of each pixel is inside the object instance with clear outlines is judged respectively , if it is, that is, the pixel falls within the range of the object instance with a clear outline, that is, the pixel is a salient foreground pixel; otherwise, it means that the pixel is between the object instance with a clear outline In addition, that is, the pixel is the background and does not belong to the salient foreground.

For example, for a two-dimensional image at each viewing angle, for each pixel in the image, if it belongs to the salient foreground, it is marked as 1, or if it does not belong to the salient foreground (belongs to the background), it is marked as 0. Assuming that there are N viewing angles, the 2D saliency image at each viewing angle is I _n ∈ ^{R w×h} , where w, h are the image width and height, respectively. If the coordinate [x, y] belongs to the salient foreground, I _n (x, y)=1, otherwise I _n (x, y)=0.

R ^w×h represents a matrix of w, x, and h. I _n (x, y) represents the pixel whose coordinates are (x, y) in I _n .

S122: According to the determination result of whether it is a salient foreground, based on the preset voxel space, determine the target voxel points belonging to the inside of the target object in the point cloud data under multiple viewing angles.

In the embodiment of the present application, the method of determining the target voxel point belonging to the inside of the target object may be, for example: determining the first ratio according to the target voxel point and the number of viewing angles of multiple viewing angles; if the first ratio is greater than the preset threshold , it is determined that the target voxel point belongs to the interior of the target object.

Fig. 4 is a schematic flow chart of the multi-view 3D reconstruction method provided by an embodiment of the present application. As shown in Fig. 4, the training process of the symbolic distance function network and the color network in the 3D reconstruction method provided by the present application is explained with a complete flow chart illustrate:

S201: Input the 2D image, camera pose and sparse point cloud of the target object under various viewing angles.

S202: Obtain a BBox and normalize the sparse point cloud.

Among them, the bounding box BBox of the target object is obtained according to the sparse point cloud result obtained by SFM, and the x, y, z coordinates of the object in the three-dimensional space are normalized to [-1,1], and the scaling matrix is calculated.

S203: Calculate the salient foreground of the two-dimensional image under each viewing angle.

Calculate the salient foreground area in the two-dimensional image taken at each viewing angle. For each pixel in the two-dimensional image, if it belongs to the foreground, it is marked as 1, and if it belongs to the background, it is marked as 0.

S204: Initialize the voxel space and determine whether it belongs to the interior of the target object.

Initialize the voxel space in the unit space in the three-dimensional space, and set the voxel interval to ρ. Each point in the above-mentioned initialized voxel space is projected into the 2D image according to the camera parameters, and it is determined that each voxel point projected into the 2D image belongs to the salient foreground or background.

Judging the proportion of each voxel point projected into the 2D map belonging to the foreground at each viewing angle, if the proportion exceeds the threshold, it is determined that the voxel point belongs to the inside of the target object, otherwise it belongs to the outside of the target object.

S205: Generate a three-dimensional surface of the target object according to the target voxel points.

The points inside the target object are determined as the target voxel points, and the three-dimensional surface of the target object, that is, the 3D convex hull is established according to the determination result of the target voxel points.

S206: Calculate the intersection interval between each sampling ray and the target object.

For the camera pose at each viewing angle, calculate the surface intersection point between each sampling ray and the 3D convex hull of the object calculated above, calculate the closest point and the farthest point, and determine the relationship between each sampling ray and the farthest point based on the closest point and the farthest point The intersection interval of the target object.

S207: Randomly sample some rays and calculate 3D model data.

Then, 3D sampling is performed along each observation ray between the nearest point and the farthest point of the 3D convex hull, and the SDF value and normal vector of the sampling point are calculated through the above-mentioned Signed Distance Function (SDF) network and Color network , sampling point spacing, and color values along the observation ray; wherein, the above-mentioned SDF value, normal vector, sampling point spacing, and color values along the observation ray are collectively referred to as 3D model data.

S208: volume density rendering.

Then, the weighted SDF value is converted into a volume density value through the Logistic cumulative distribution function. And perform volume rendering according to the volume density value, color value and sampling interval of all sampling points on each ray, and obtain the final rendered color value of each ray.

S209: Calculate a loss function.

S210: Optimizing network parameters.

Wherein, the optimized network parameters respectively include optimized signed distance function network parameters and network parameters of the color network.

S211: Whether the maximum number of iterations is reached.

If not, go back to S207 and continue to randomly sample and calculate some rays.

If yes, execute S212.

S212: Save network parameters.

If the maximum number of iterations is reached, it is determined that the iteration of the signed distance function network and the color network is completed, and at this time the iterated model is obtained and put into use.

Using the multi-view 3D reconstruction method provided by this application, in solving the extrusion of 3D reconstruction and rendering of 2D images, a method of 3D reconstruction of 2D images based on 3D surfaces is proposed, which can optimize the effective The sampling of three-dimensional sampling rays within the range only determines the color of each sampling ray in the focal area based on the color network, thereby determining the surface color of the target object. This determination method can narrow the sampling range of the sampling ray, thereby improving the determination of the target object. Improve the efficiency of surface color, improve the rendering efficiency of target objects, and improve the efficiency and accuracy of 3D reconstruction.

The multi-view 3D reconstruction device provided by the present application will be explained below in conjunction with the accompanying drawings. The multi-view 3D reconstruction device can execute any of the above-mentioned multi-view 3D reconstruction methods in Figures 1-4. For its specific implementation and beneficial effects, refer to the above. No further details will be given below.

FIG. 5 is a schematic structural diagram of a multi-view 3D reconstruction device provided by an embodiment of the present application. As shown in FIG. 5 , the device includes: an acquisition module 201, a determination module 202, a sampling module 203, and a rendering module 204, wherein:

An acquisition module 201, configured to acquire two-dimensional images of the target object under multiple viewing angles;

A determining module 202, configured to determine point cloud data under multiple viewing angles according to the two-dimensional images under multiple viewing angles;

The sampling module 203 is configured to sample the point cloud in the preset voxel space in the point cloud data under multiple viewing angles based on the preset voxel space, to obtain the target voxel point;

The determining module 202 is specifically used to determine the three-dimensional surface of the target object according to the target voxel points; determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera pose under each viewing angle; use the color network to determine the target based on each intersection interval the surface color of the object;

The rendering module 204 is configured to render the target object based on the surface color.

Optionally, the determining module 202 is specifically configured to determine the signed distance function value, the normal vector, and the target voxel point spacing of the target voxel point according to the signed distance function network; determine the color along each sampling ray according to the color network Value; get the target render color value for each sampled ray.

Optionally, the determining module 202 is specifically configured to determine the three-dimensional surface of the target object based on the target voxel points using an isosurface extraction algorithm.

Optionally, on the basis of the above-mentioned embodiments, the embodiments of the present application may also provide a multi-view three-dimensional reconstruction device. The implementation process of the above-mentioned device shown in FIG. 5 is illustrated as follows with reference to the accompanying drawings. Fig. 6 is a schematic structural diagram of a multi-view 3D reconstruction device provided by another embodiment of the present application. As shown in Fig. 6, the device further includes: a setting module 205, which is used to set a preset voxel space according to the target accuracy of the target object The number of voxels to sample within.

Optionally, the determination module 202 is specifically configured to determine whether each pixel in the two-dimensional image under multiple viewing angles is a salient foreground; according to the determination result of whether it is a salient foreground, based on a preset voxel space, in In the point cloud data under multiple viewing angles, the target voxel points belonging to the interior of the target object are determined.

Optionally, the determining module 202 is specifically configured to determine a first ratio according to the target voxel point and the number of viewing angles of multiple viewing angles; if the first ratio is greater than a preset threshold, determine that the target voxel point belongs to the inside of the target object.

Optionally, the determination module 202 is specifically configured to determine the information of the bounding box of the target object according to the point cloud data under the multiple viewing angles;

The acquisition module 201 is specifically used to normalize the bounding box of the target object, and acquire normalized bounding box data;

The sampling module 203 is specifically configured to, based on the preset voxel space, sample point clouds in the preset voxel space in the normalized bounding box data under multiple viewing angles to obtain target voxel points.

The above-mentioned device is used to execute the methods provided in the foregoing embodiments, and its implementation principles and technical effects are similar, and details are not repeated here.

The above modules may be one or more integrated circuits configured to implement the above method, for example: one or more specific integrated circuits (Application Specific Integrated Circuit, referred to as ASIC), or, one or more microprocessors, or, One or more Field Programmable Gate Arrays (Field Programmable Gate Array, FPGA for short), etc. For another example, when one of the above modules is implemented in the form of a processing element scheduling program code, the processing element may be a general-purpose processor, such as a central processing unit (Central Processing Unit, CPU for short) or other processors that can call program codes. For another example, these modules can be integrated together and implemented in the form of a system-on-a-chip (SOC for short).

FIG. 7 is a schematic structural diagram of a multi-view 3D reconstruction device provided by an embodiment of the present application. The multi-view 3D reconstruction device may be integrated into a terminal device or a chip of the terminal device.

As shown in FIG. 7 , the multi-view three-dimensional reconstruction device includes: a processor 501 , a bus 502 and a storage medium 503 .

The processor 501 is configured to store a program, and the processor 501 invokes the program stored in the storage medium 503 to execute the above-mentioned method embodiments corresponding to FIGS. 1-4 . The specific implementation manner and technical effect are similar, and will not be repeated here.

Optionally, the present application further provides a program product, such as a storage medium, on which a computer program is stored, including a program. When the program is run by a processor, the corresponding embodiment of the above method is executed.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are only illustrative. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated. to another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit. The above-mentioned integrated units can be implemented in the form of hardware, or in the form of hardware plus software functional units.

The above-mentioned integrated units implemented in the form of software functional units may be stored in a computer-readable storage medium. The above-mentioned software functional units are stored in a storage medium, and include several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) or a processor (English: processor) to execute the functions described in various embodiments of the present application. part of the method. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (English: Read-Only Memory, abbreviated: ROM), random access memory (English: Random Access Memory, abbreviated: RAM), magnetic disk or optical disc, etc. Various media that can store program code.

Claims (10)

1. A method for multi-view three-dimensional reconstruction, characterized in that the method comprises: Acquire two-dimensional images of the target object under multiple viewing angles; According to the two-dimensional images under the multiple viewing angles, respectively determine the point cloud data under the multiple viewing angles; Based on the preset voxel space, sampling the point cloud in the preset voxel space in the point cloud data under multiple viewing angles to obtain the target voxel point; determining a three-dimensional surface of the target object according to the target voxel points; Determining the intersection interval between each sampling ray and the three-dimensional surface according to the camera pose under each of the viewing angles; Using a color network to determine the surface color of the target object based on each of the intersection intervals; Render the target object based on the surface color. 2. The method according to claim 1, wherein said adopting a color network to determine the surface color of said target object based on each said intersection interval comprises: According to the signed distance function network, determine the signed distance function value of the target voxel point, the normal vector and the target voxel point spacing; determining a color value along each of the sampling rays according to the color network; Gets the target render color value for each of said sampled rays. 3. The method according to claim 1, wherein said determining the three-dimensional surface of the target object according to the target voxel points comprises: An isosurface extraction algorithm is used to determine the three-dimensional surface of the target object based on the target voxel points. 4. The method according to claim 1, wherein, based on the preset voxel space, the point cloud in the preset voxel space in the point cloud data under multiple viewing angles is sampled to obtain Before the target voxel point, the method also includes: According to the target accuracy of the target object, the number of voxels sampled in the preset voxel space is set. 5. The method according to claim 1, wherein, based on the preset voxel space, the point cloud in the preset voxel space in the point cloud data under multiple viewing angles is sampled to obtain Target voxel points, including: determining whether each pixel in the two-dimensional images under the plurality of viewing angles is a salient foreground; According to the determination result of whether it is a salient foreground, based on the preset voxel space, the target voxel points belonging to the inside of the target object are determined in the point cloud data under multiple viewing angles. 6. The method according to claim 5, wherein the determining the target voxel points belonging to the inside of the target object comprises: determining a first ratio according to the target voxel point and the number of viewing angles of multiple viewing angles; If the first ratio is greater than a preset threshold, it is determined that the target voxel point belongs to the inside of the target object. 7. The method according to claim 1, wherein, based on the preset voxel space, the point cloud in the preset voxel space in the point cloud data under multiple viewing angles is sampled to obtain Before the target voxel point, the method also includes: Determine the information of the bounding box of the target object according to the point cloud data under the plurality of viewing angles; normalizing the bounding box of the target object, and obtaining normalized bounding box data; Based on the preset voxel space, the point cloud in the preset voxel space in the point cloud data under multiple viewing angles is sampled to obtain the target voxel point, including: Based on the preset voxel space, the point cloud in the preset voxel space in the normalized bounding box data under multiple viewing angles is sampled to obtain the target voxel point. 8. A multi-view three-dimensional reconstruction device, characterized in that the device comprises: an acquisition module, a determination module, a sampling module and a rendering module, wherein: The acquiring module is configured to acquire two-dimensional images of the target object under multiple viewing angles; The determining module is configured to respectively determine point cloud data under the multiple viewing angles according to the two-dimensional images under the multiple viewing angles; The sampling module is configured to, based on a preset voxel space, sample point clouds in the preset voxel space in the point cloud data under multiple viewing angles to obtain target voxel points; The determining module is specifically configured to determine the three-dimensional surface of the target object according to the target voxel points; determine the intersection interval between each sampling ray and the three-dimensional surface according to the camera poses under each of the viewing angles; The color network determines the surface color of the target object based on each of the intersection intervals; The rendering module is configured to render the target object based on the surface color. 9. A multi-view three-dimensional reconstruction device, characterized in that the device comprises: a processor, a storage medium and a bus, the storage medium stores machine-readable instructions executable by the processor, and when the multi-view When the three-dimensional reconstruction device is running, the processor communicates with the storage medium through a bus, and the processor executes the machine-readable instructions to perform the method described in any one of claims 1-7. 10. A storage medium, wherein a computer program is stored on the storage medium, and the computer program executes the method according to any one of claims 1-7 when run by a processor.

Images