WO2022237249A1

WO2022237249A1 - Three-dimensional reconstruction method, apparatus and system, medium, and computer device

Info

Publication number: WO2022237249A1
Application number: PCT/CN2022/075636
Authority: WO
Inventors: 曹智杰; 汪旻; 刘文韬; 钱晨; 马利庄
Original assignee: 上海商汤智能科技有限公司
Priority date: 2021-05-10
Filing date: 2022-02-09
Publication date: 2022-11-17
Also published as: JP2023547888A; KR20230078777A; TW202244853A; CN113160418A

Abstract

The present disclosure provides a three-dimensional reconstruction method, apparatus and system, a medium, and a computer device. The method comprises: carrying out three-dimensional reconstruction on a target object in an image by means of a three-dimensional reconstruction network, and obtaining an initial value of a parameter of the target object, the initial value of the parameter being used for building a three-dimensional model of the target object; optimizing the initial value of the parameter on the basis of pre-acquired supervision information used for indicating features of the target object, so as to obtain an optimized value of the parameter; and performing skinned mesh processing on the basis of the optimized value of the parameter, and establishing a three-dimensional model of the target object.

Description

Three-dimensional reconstruction method, device and system, medium and computer equipment

Cross References to Related Applications

This disclosure claims the priority of the Chinese patent application with the application number 202110506464X and the title of the invention "three-dimensional reconstruction method, device and system, medium and computer equipment" submitted on May 10, 2021, which is incorporated by reference into this article.

technical field

The present disclosure relates to the technical field of computer vision, and in particular to a three-dimensional reconstruction method, device and system, media and computer equipment.

Background technique

3D reconstruction is one of the important technologies in computer vision, and has many potential applications in fields such as augmented reality and virtual reality. By performing three-dimensional reconstruction on the target object, the posture and limb rotation of the target object can be reconstructed. However, traditional 3D reconstruction methods cannot balance the accuracy and reliability of reconstruction results.

Contents of the invention

The present disclosure provides a three-dimensional reconstruction method, device and system, medium and computer equipment.

According to the first aspect of the embodiments of the present disclosure, there is provided a 3D reconstruction method, the method comprising: performing 3D reconstruction on a target object in an image through a 3D reconstruction network to obtain an initial value of a parameter of the target object, wherein the The initial value of the parameter is used to establish the three-dimensional model of the target object; the initial value of the parameter is optimized based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain the optimized value of the parameter; based on the obtained The optimized values of the above parameters are used for bone skinning processing, and the three-dimensional model of the target object is established.

In some embodiments, the supervision information includes first supervision information, or the supervision information includes first supervision information and second supervision information; the first supervision information includes at least one of the following: the initial Two-dimensional key points, semantic information of multiple pixel points on the target object in the image; the second supervisory information includes an initial three-dimensional point cloud of the target object surface. In the embodiments of the present disclosure, only the initial two-dimensional key points or semantic information of pixels of the target object can be used as supervisory information to optimize the initial value of the parameter, which has high optimization efficiency and low optimization complexity; or, can also use The initial 3D point cloud of the surface of the target object and the semantic information of the aforementioned initial 2D key points or pixels are used as supervisory information, thereby improving the accuracy of the optimal value of the obtained parameters.

In some embodiments, the method further includes: extracting information of initial two-dimensional key points of the target object from the image through a key point extraction network. Using the information of the initial two-dimensional key points extracted by the key point extraction network as supervision information can generate more natural and reasonable actions for the three-dimensional model.

In some embodiments, the image includes a depth image of the target object; the method further includes: extracting depth information of a plurality of pixels on the target object from the depth image; A plurality of pixel points on the target object in the depth image are back-projected to a three-dimensional space to obtain an initial three-dimensional point cloud of the surface of the target object. By extracting the depth information and back-projecting the pixels on the two-dimensional image to the three-dimensional space based on the depth information, the initial three-dimensional point cloud of the target object surface can be obtained, so that the initial three-dimensional point cloud can be used as the supervision information to optimize the initial parameters. value, further improving the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; the extracting the depth information of a plurality of pixels on the target object from the depth image includes: performing image processing on the RGB image Segmentation, determining the image area where the target object is located in the RGB image based on the results of the image segmentation, determining the image area where the target object is located in the depth image based on the image area where the target object is located in the RGB image; acquiring the depth image Depth information of multiple pixels in the image area where the target object is located. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.

In some embodiments, the method further includes: filtering outliers from the initial three-dimensional point cloud, and using the filtered initial three-dimensional point cloud as the second supervisory information. By filtering the outliers, the interference of the outliers is reduced, and the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is acquired by an image acquisition device, and the parameters include: the global rotation parameter of the target object, the key point rotation parameters of each key point of the target object, the target object The body parameters and the displacement parameters of the image acquisition device; the initial value of the parameter is optimized based on the pre-acquired supervision information used to represent the characteristics of the target object, including: the initial value of the body parameter and the key Under the condition that the initial value of the point rotation parameter remains unchanged, based on the supervisory information and the initial value of the displacement parameter, optimize the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter , to obtain the optimized value of the displacement parameter and the optimized value of the global rotation parameter; based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter are performed Optimization, to obtain the optimal value of the key point rotation parameter and the optimal value of the body parameter. During the optimization process, changing the position of the image acquisition device and changing the position of the 3D key points can lead to changes in the 2D projection of the 3D key points, which will lead to an unstable optimization process. By adopting a two-stage optimization method, firstly fix the initial value of the key point rotation parameter and the initial value of the posture parameter to optimize the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter, and then fix the initial value of the displacement parameter value and the initial value of the global rotation parameter, optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter, and improve the stability of the optimization process.

In some embodiments, the supervision information includes the initial two-dimensional key points of the target object; the current value of the displacement parameter of the image acquisition device based on the supervision information and the initial value of the displacement parameter And optimizing the initial value of the global rotation parameter includes: obtaining the target two-dimensional projection key points corresponding to the two-dimensional projection key points corresponding to the three-dimensional key points of the target object belonging to the preset position of the target object; wherein, The 3D key points of the target object are obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the posture parameter, and the 2D projection key point is based on the current value of the displacement parameter and the global The initial value of the rotation parameter is obtained by projecting the three-dimensional key point of the target object; obtaining the first loss between the target two-dimensional projection key point and the initial two-dimensional key point; obtaining the initial value of the displacement parameter and a second loss between the current value of the displacement parameter; optimizing the current value of the displacement parameter and the initial value of the global rotation parameter based on the first loss and the second loss. The preset part can be the torso and other parts. Since different actions have little influence on the key points of the torso, the first loss can be determined by using the key points of the torso, which can reduce the influence of different actions on the position of the key points and improve Optimize the accuracy of the results. Since the two-dimensional key points are supervisory information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, it is possible to reduce the deviation of the optimization result from falling into the local optimal point on the two-dimensional plane. The real situation.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the optimized value based on the displacement parameter and the optimized value of the global rotation parameter, and the initial value of the key point rotation parameter Optimizing with the initial value of the posture parameter includes: obtaining the third loss between the optimized two-dimensional projection key point of the target object and the initial two-dimensional key point, and the optimized two-dimensional projection key point is based on the The optimized value of the displacement parameter and the optimized value of the global rotation parameter are obtained by projecting the optimized three-dimensional key point of the target object, and the optimized three-dimensional key point is based on the optimized value of the global rotation parameter and the initial value of the key point rotation parameter and the initial value of the posture parameter is obtained; the fourth loss is obtained, and the fourth loss is used to characterize the rationality of the posture corresponding to the optimal value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter; Optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the third loss and the fourth loss. This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, which improves the stability of the optimization process. At the same time, the fourth loss ensures the optimization The latter parameters correspond to the rationality of the pose.

In some embodiments, the method further includes: after optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter , performing joint optimization on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. In this embodiment, on the basis of the aforementioned optimization, the optimized parameters are jointly optimized, thereby further improving the accuracy of the optimization result.

In some embodiments, the supervision information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; based on the supervision information and the initial value of the displacement parameter, the Optimizing the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter includes: obtaining presets belonging to the target object in the two-dimensional projection key points corresponding to the three-dimensional key points of the target object The target two-dimensional projection key point of the part; wherein, the three-dimensional key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the posture parameter, and the two-dimensional projection key The point is obtained by projecting the 3D key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the first 2D key point between the target 2D projection key point and the initial 2D key point A loss; obtain the second loss between the initial value of the displacement parameter and the current value of the displacement parameter; obtain the fifth loss between the first three-dimensional point cloud of the surface of the target object and the initial three-dimensional point cloud loss; the first three-dimensional point cloud is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter; based on the first loss, the second loss and the fifth loss to the The current value of the displacement parameter and the initial value of the global rotation parameter are optimized. In this embodiment, the three-dimensional point cloud is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

In some embodiments, the joint optimization of the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the posture parameter and the optimized value of the displacement parameter includes: obtaining the A sixth loss between an optimized two-dimensional projection keypoint of the target object and the initial two-dimensional keypoint, the optimized two-dimensional projection keypoint is based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter on the The optimized three-dimensional key points of the target object are obtained by projection, and the optimized three-dimensional key points are obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the body shape parameter; the seventh loss is obtained, and the first Seven losses are used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the body shape parameter; the second three-dimensional point cloud of the surface of the target object is obtained and the initial The eighth loss between three-dimensional point clouds; the second three-dimensional point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized value of the body shape parameter; based on the sixth loss, the first The seventh loss and the eighth loss jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter. In this embodiment, the three-dimensional point cloud is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

According to the second aspect of the embodiments of the present disclosure, there is provided a 3D reconstruction device, the device comprising: a first 3D reconstruction module, configured to perform 3D reconstruction on a target object in an image through a 3D reconstruction network, to obtain the target object The initial value of the parameter, wherein the initial value of the parameter is used to establish the three-dimensional model of the target object; the optimization module is used to adjust the initial value of the parameter based on the pre-acquired supervision information used to represent the characteristics of the target object Perform optimization to obtain the optimized value of the parameter; the second three-dimensional reconstruction module is used to perform bone skinning processing based on the optimized value of the parameter, and establish a three-dimensional model of the target object.

In some embodiments, the device further includes: a two-dimensional key point extraction module, configured to extract initial two-dimensional key point information of the target object from the image through a key point extraction network. Using the information of the initial two-dimensional key points extracted by the key point extraction network as supervision information can generate more natural and reasonable actions for the three-dimensional model.

In some embodiments, the image includes a depth image of the target object; the device further includes: a depth information extraction module, configured to extract depth information of multiple pixels on the target object from the depth image a back-projection module, configured to back-project multiple pixel points on the target object in the depth image to a three-dimensional space based on the depth information, to obtain an initial three-dimensional point cloud on the surface of the target object. By extracting the depth information and back-projecting the pixels on the two-dimensional image to the three-dimensional space based on the depth information, the initial three-dimensional point cloud of the target object surface can be obtained, so that the initial three-dimensional point cloud can be used as the supervision information to optimize the initial parameters. value, further improving the accuracy of parameter optimization.

In some embodiments, the image further includes an RGB image of the target object; the depth information extraction module includes: an image segmentation unit for performing image segmentation on the RGB image, and an image area determination unit for based on The result of image segmentation determines the image area where the target object is located in the RGB image, and determines the image area where the target object is located in the depth image based on the image area where the target object is located in the RGB image; the depth information acquisition unit is used to acquire Depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation on the RGB image, the position of the target object can be accurately determined, thereby accurately extracting the depth information of the target object.

In some embodiments, the device further includes: a filtering module, configured to filter out outliers from the initial 3D point cloud, and use the filtered initial 3D point cloud as the second supervisory information. By filtering the outliers, the interference of the outliers is reduced, and the accuracy of the parameter optimization process is further improved.

In some embodiments, the image of the target object is acquired by an image acquisition device, and the parameters include: the global rotation parameter of the target object, the key point rotation parameters of each key point of the target object, the target object The posture parameters of the body parameters and the displacement parameters of the image acquisition device; the optimization module includes: a first optimization unit, for when the initial values of the body posture parameters and the initial values of the key point rotation parameters remain unchanged, based on The supervision information and the initial value of the displacement parameter are optimized by optimizing the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter to obtain an optimized value of the displacement parameter and an optimized value of the global rotation parameter ; The second optimization unit is used to optimize the initial value of the key point rotation parameter and the initial value of the body posture parameter based on the optimal value of the displacement parameter and the optimal value of the global rotation parameter to obtain the key point rotation parameter The optimal value of and the optimal value of body parameters. During the optimization process, changing the position of the image acquisition device and changing the position of the 3D key points can lead to changes in the 2D projection of the 3D key points, which will lead to an unstable optimization process. By adopting a two-stage optimization method, firstly fix the initial value of the key point rotation parameter and the initial value of the posture parameter to optimize the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter, and then fix the initial value of the displacement parameter value and the initial value of the global rotation parameter, optimize the initial value of the key point rotation parameter and the initial value of the body shape parameter, and improve the stability of the optimization process.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the first optimization unit is configured to: obtain the two-dimensional projection key points corresponding to the three-dimensional key points of the target object belonging to the The target two-dimensional projection key point of the preset part of the target object; wherein, the three-dimensional key point of the target object is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, The two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the target two-dimensional projection key point and the initial two-dimensional a first loss between key points; obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter; based on the first loss and the second loss on the current value of the displacement parameter and the initial value of the global rotation parameter for optimization. The preset part can be the torso and other parts. Since different actions have little influence on the key points of the torso, the first loss can be determined by using the key points of the torso, which can reduce the influence of different actions on the position of the key points and improve Optimize the accuracy of the results. Since the two-dimensional key points are supervisory information on the two-dimensional plane, and the displacement parameters of the image acquisition device are parameters on the three-dimensional plane, by obtaining the second loss, it is possible to reduce the deviation of the optimization result from falling into the local optimal point on the two-dimensional plane. The real situation.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object; the second optimization unit is configured to: obtain the optimized two-dimensional projection key points of the target object and the initial two-dimensional key points The third loss between points, the optimized two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized three-dimensional key point The point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the posture parameter; the fourth loss is obtained, and the fourth loss is used to characterize the optimal value of the global rotation parameter, the key point The rationality of the attitude corresponding to the initial value of the rotation parameter and the initial value of the posture parameter; based on the third loss and the fourth loss, the initial value of the key point rotation parameter and the initial value of the posture parameter are optimized . This embodiment optimizes the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, which improves the stability of the optimization process. At the same time, the fourth loss ensures the optimization The latter parameters correspond to the rationality of the pose.

In some embodiments, the device further includes: a joint optimization module, configured to perform an initial value of the key point rotation parameter and the body shape parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter. After the initial value of is optimized, the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized. In this embodiment, on the basis of the aforementioned optimization, the optimized parameters are jointly optimized, thereby further improving the accuracy of the optimization result.

In some embodiments, the supervisory information includes the initial two-dimensional key points of the target object and the initial three-dimensional point cloud of the surface of the target object; the first optimization unit is configured to: acquire the three-dimensional key points of the target object Among the two-dimensional projection key points corresponding to the point, the target two-dimensional projection key point belonging to the preset part of the target object; wherein, the three-dimensional key point of the target object is based on the initial value of the global rotation parameter, the key point rotation parameter The initial value of the initial value and the initial value of the posture parameter are obtained, and the two-dimensional projection key point is obtained by projecting the three-dimensional key point of the target object based on the current value of the displacement parameter and the initial value of the global rotation parameter; obtaining the target The first loss between the two-dimensional projection key point and the initial two-dimensional key point; obtain the second loss between the initial value of the displacement parameter and the current value of the displacement parameter; obtain the target object surface The fifth loss between the first 3D point cloud and the initial 3D point cloud; the first 3D point cloud is obtained based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter ; optimizing the current value of the displacement parameter and the initial value of the global rotation parameter based on the first loss, the second loss and the fifth loss. In this embodiment, the three-dimensional point cloud is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

In some embodiments, the joint optimization module includes: a first acquisition unit, configured to acquire the sixth loss between the optimized 2D projection keypoint of the target object and the initial 2D keypoint, the optimization The two-dimensional projection key point is obtained by projecting the optimized three-dimensional key point of the target object based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, and the optimized three-dimensional key point is based on the optimized value of the global rotation parameter, The optimized value of the key point rotation parameter and the optimized value of the posture parameter are obtained; the second acquisition unit is used to obtain the seventh loss, and the seventh loss is used to represent the optimized value of the global rotation parameter and the optimized value of the key point rotation parameter value and the rationality of the posture corresponding to the optimized value of the body posture parameter; the third acquisition unit is used to acquire the eighth loss between the second 3D point cloud on the surface of the target object and the initial 3D point cloud; the first 3D point cloud The 2D and 3D point cloud is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the body shape parameter; the joint optimization unit is used for pairing based on the sixth loss, the seventh loss and the eighth loss The optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized. In this embodiment, the three-dimensional point cloud is added to the supervisory information to optimize various initial parameters, thereby improving the accuracy of the optimization result.

According to a third aspect of an embodiment of the present disclosure, there is provided a three-dimensional reconstruction system, the system comprising: an image acquisition device, configured to acquire an image of a target object; and a processing unit communicatively connected to the image acquisition device, configured to The three-dimensional reconstruction network performs three-dimensional reconstruction on the target object in the image to obtain the initial value of the parameter of the target object, and the initial value of the parameter is used to establish the three-dimensional model of the target object; Optimizing the initial value of the parameter based on the supervisory information representing the characteristics of the target object to obtain the optimized value of the parameter; performing bone skinning processing based on the optimized value of the parameter to establish a three-dimensional model of the target object.

According to a fourth aspect of the embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any embodiment is implemented.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computer device, including a memory, a processor, and a computer program stored in the memory and operable on the processor. When the processor executes the computer program, any The method described in the examples.

According to a sixth aspect of the embodiments of the present disclosure, a computer program product is provided, the computer program product is stored in a storage medium and includes a computer program that can run on a processor, and when the processor executes the computer program, any A method described in one embodiment.

In the embodiment of the present disclosure, the initial value of the parameter is obtained by three-dimensionally reconstructing the image of the target object through the three-dimensional reconstruction network, and then the initial value of the parameter is optimized based on the supervisory information, and the optimized value of the parameter obtained based on the parameter optimization is established. 3D model of the target object. The advantage of the parameter optimization method is that it can give more accurate 3D reconstruction results that conform to the 2D observation characteristics of the image, but it often gives unnatural and unreasonable action results with low reliability. Network regression through the 3D reconstruction network can give more natural and reasonable action results. Therefore, optimizing the output of the 3D reconstruction network as the initial value of the parameters can ensure the reliability of the 3D reconstruction results. Accuracy of 3D reconstruction.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Description of drawings

The accompanying drawings here are incorporated into the description and constitute a part of the present description. These drawings show embodiments consistent with the present disclosure, and are used together with the description to explain the technical solution of the present disclosure.

1A and 1B are schematic illustrations of three-dimensional models of some embodiments.

Fig. 2 is a flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure.

FIG. 3 is an overall flowchart of an embodiment of the present disclosure.

FIG. 4A and FIG. 4B are schematic diagrams of application scenarios of embodiments of the present disclosure, respectively.

FIG. 5 is a block diagram of a three-dimensional reconstruction device according to an embodiment of the present disclosure.

FIG. 6 is a schematic diagram of a three-dimensional reconstruction system according to an embodiment of the present disclosure.

FIG. 7 is a schematic structural diagram of a computer device according to an embodiment of the present disclosure.

Detailed ways

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numerals in different drawings refer to the same or similar elements unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatuses and methods consistent with aspects of the present disclosure as recited in the appended claims.

The terminology used in the present disclosure is for the purpose of describing particular embodiments only, and is not intended to limit the present disclosure. As used in this disclosure and the appended claims, the singular forms "a", "the", and "the" are intended to include the plural forms as well, unless the context clearly dictates otherwise. It should also be understood that the term "and/or" as used herein refers to and includes any and all possible combinations of one or more of the associated listed items. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality.

It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, the information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of the present disclosure, first information may also be called second information, and similarly, second information may also be called first information. Depending on the context, the word "if" as used herein may be interpreted as "at" or "when" or "in response to a determination."

In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure, and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are described below in conjunction with the accompanying drawings The program is described in further detail.

The 3D reconstruction of the target object needs to reconstruct the body posture and limb rotation of the target object. Usually, a parametric model is used to express the body posture and limb rotation of the target object, not just the 3D key points. For example, by performing 3D reconstruction on different people, a 3D model of a thinner person (as shown in Figure 1A) and a 3D model of a fatter person (as shown in Figure 1B ) are respectively reconstructed. The person shown in Figure 1B is in the same posture as the person shown in Figure 1B, and the key point information is the same, and the difference in posture between the two cannot be represented only through the key point information.

In related technologies, 3D reconstruction is generally carried out by means of parameter optimization and network regression. The parameter optimization method usually selects a set of standard parameters, and uses the gradient descent method to iteratively optimize the initial value of the parameters of the 3D model of the target object according to the 2D visual features of the image of the target object, where the 2D visual features of the image can be Select 2D keypoints, etc. The advantage of the parameter optimization method is that it can give more accurate parameter estimation results that conform to the two-dimensional visual characteristics of the image, but it often gives unnatural and unreasonable action results, and the final performance of parameter optimization is very dependent on the initial value of the parameters , leading to low reliability of the 3D reconstruction method based on parameter optimization.

Methods for network regression typically train an end-to-end neural network to learn the mapping from images to 3D model parameters. The advantage of the network regression method is that it can give more natural and reasonable action results. However, due to the lack of a large amount of training data, the 3D reconstruction results may not match the 2D visual features in the image. Therefore, the accuracy of the 3D reconstruction method based on network regression is relatively low. Low. The 3D reconstruction method in the related art cannot take into account the accuracy and reliability of the 3D reconstruction results.

Based on this, an embodiment of the present disclosure provides a three-dimensional reconstruction method, as shown in FIG. 2 , the method includes:

Step 201: Perform 3D reconstruction on the target object in the image through a 3D reconstruction network to obtain initial values of parameters of the target object, wherein the initial values of the parameters are used to establish a 3D model of the target object;

Step 202: Optimizing the initial value of the parameter based on the pre-acquired supervisory information representing the characteristics of the target object to obtain the optimized value of the parameter;

Step 203: Perform bone skinning processing based on the optimized values of the parameters, and establish a 3D model of the target object.

In step 201, the target object may be a three-dimensional object, such as a person, an animal, a robot, etc. in a physical space, or one or more regions on the three-dimensional object, such as a human face or a limb. For the convenience of description, the target object is a human being, and the three-dimensional reconstruction performed on the target object is a human body reconstruction as an example for description. The image of the target object may be a single image, or may include multiple images obtained by shooting the target object from multiple different angles of view. 3D human body reconstruction based on a single image is called monocular 3D human body reconstruction, and 3D human body reconstruction based on multiple images from different perspectives is called multi-eye 3D human body reconstruction. Each image can be a grayscale image, RGB image or RGBD image. The image may be an image collected in real time by an image acquisition device (for example, a camera or a camera) around the target object, or an image collected and stored in advance.

The image of the target object can be reconstructed in 3D through a 3D reconstruction network, wherein the 3D reconstruction network can be a pre-trained neural network. The 3D reconstruction network can perform 3D reconstruction based on images, and estimate the initial values of natural and reasonable parameters. The initial values of the parameters here can be represented by a vector. The dimension of the vector can be 85 dimensions, for example, and the vector contains The rotation information of the moving limbs of the human body (that is, the initial value of the posture parameters, including the initial values of the global rotation parameters of the human body and the initial values of the key point rotation parameters of 23 key points), the initial values of the posture parameters and the initial values of the camera parameters These three parts of information. The human body can be represented by key points and limb bones connecting these key points. The key points of the human body can include the top of the head, nose, neck, left and right eyes, left and right ears, chest, left and right shoulders, left and right elbows, left and right wrists, left and right hips, left and right buttocks, One or more of key points such as left and right knees, left and right ankles, etc., the initial value of the pose parameter is used to determine the position of the key points of the human body in three-dimensional space. The initial value of the body shape parameter is used to determine body shape information such as height, shortness, fatness, and thinness of the human body. The initial value of the parameter of the camera is used to determine the absolute position of the human body in the three-dimensional space under the camera coordinate system, and the parameter of the camera includes a displacement parameter between the camera and the human body and a posture parameter of the camera, wherein the posture parameter of the camera is The initial value can be replaced by the initial value of the global rotation parameter of the human body. The parameters of the human body can be expressed using a parametric form of a Skinned Multi-Person Linear (SMPL) model (referred to as SMPL parameters). After obtaining the value of the SMPL parameter, the bone skinning process can be performed based on the value of the SMPL parameter, that is, a mapping function M(θ,β) is used to map the initial value of the body parameter and the initial value of the attitude parameter to the three-dimensional model of the human body surface , the 3D model includes 6890 vertices, and the vertices form a triangular patch through a fixed connection relationship. A pre-trained regressor W can be used to further regress the 3D key points of the human body from the vertices of the human surface model

which is:

In step 202, the supervisory information can be two-dimensional visual features of the image (also called two-dimensional observation features), for example, the two-dimensional key points of the target object in the image and the semantics of multiple pixel points on the target object at least one of the information. The semantic information of a pixel is used to represent which area the pixel is located on the target object, and the area may be, for example, the area where the head, arm, torso, leg, etc. are located. In the case of using two-dimensional key point information as supervision information, the two-dimensional key point extraction network can be used to estimate the position of human key points in the image. Here, any two-dimensional pose estimation method can be used, such as OpenPose. In addition to using 2D visual features as supervision information, 2D visual features and the initial 3D point cloud of the target object surface can also be used as supervision information to further improve the accuracy of 3D reconstruction.

When the image includes a depth image (for example, the image is an RGBD image), the depth information of multiple pixels on the target object can be extracted from the depth image, and the Multiple pixel points on the target object in the depth image are projected into a three-dimensional space to obtain an initial three-dimensional point cloud on the surface of the target object.

The plurality of pixels may be part or all of the pixels on the target object in the image. For example, it may include pixel points of various areas on the target object that need to be three-dimensionally reconstructed, and the number of pixel points in each area should be greater than or equal to the number required for three-dimensional reconstruction.

Because the image generally includes both the target object and the background area. Therefore, image segmentation can be performed on the RGB image included in the image, the image area where the target object is located in the RGB image is obtained, and the target object in the depth image is determined based on the image area where the target object is located in the RGB image. the image area; acquiring depth information of multiple pixels in the image area where the target object is located in the depth image. By performing image segmentation, the image area where the target object that needs to be three-dimensionally reconstructed is located can be extracted from the image, and the influence of the background area in the image on the three-dimensional reconstruction can be avoided. In some embodiments, the pixels in the depth image correspond one-to-one to the pixels in the RGB image. For example, the image may also be an RGBD image.

Further, outliers can also be filtered out from the 3D point cloud (ie, the initial 3D point cloud), and the supervision information can include the filtered 3D point cloud. The filtering can be implemented using a point cloud filter. By filtering out outliers, a finer 3D point cloud of the surface of the target object can be obtained, thereby further improving the accuracy of 3D reconstruction. For each target 3D point in the 3D point cloud, obtain the average distance from the n 3D points nearest to the target 3D point to the target 3D point, assuming that the average distance corresponding to each target 3D point obeys a statistical distribution (for example, Gaussian distribution), the mean and variance of the statistical distribution can be calculated, and a threshold s can be set based on the mean and variance, then the three-dimensional points whose average distance is outside the range of the threshold s can be regarded as outliers and analyzed from the three-dimensional filtered from the point cloud.

In practical applications, if the image is an RGB image, the initial values of the parameters can be iteratively optimized using the two-dimensional observation features as supervisory information. If the image is an RGBD image, the two-dimensional observation features and the three-dimensional point cloud of the surface of the target object can be used as supervisory information to iteratively optimize the initial value of the parameter. The optimization method may, for example, use a gradient descent method, which is not limited in the present disclosure.

In step 203, bone skinning processing may be performed based on the optimized values of the parameters to obtain a three-dimensional model of the target object.

As shown in FIG. 3 , it is an overall flowchart of the embodiment of the present disclosure. In the case that the input is an RGB image, the RGB image can be reconstructed three-dimensionally through the three-dimensional reconstruction network to obtain the human body parameter value of the person in the image, and the key point extraction network can be used to extract the key points of the person in the image to obtain the two-dimensional human body key point. Then, the human body parameter value is used as the initial value of the parameter, and the two-dimensional key points of the human body are used as the supervision information, and the initial value of the human body parameter is optimized through the parameter optimization module to obtain the optimized value of the human body parameter, and based on the optimized value of the human body parameter. Skinning processing to obtain the human body reconstruction model.

In the case that the input is an RGBD image, the image can be decomposed into an RGB image and a TOF (Time of Flight, time of flight) depth map. The TOF depth map includes the depth information of each pixel in the RGB image. The RGB image can be reconstructed three-dimensionally through the three-dimensional reconstruction network to obtain the human body parameter value of the person in the image, and the key point extraction network can be used to extract the key point of the person in the image to obtain the two-dimensional key point of the human body. The point cloud reconstruction module can also be used to reconstruct the surface point cloud of the human body based on the depth information in the TOF depth map. Then, the human body parameter value is used as the initial value of the parameter, and the two-dimensional key points of the human body and the point cloud of the human body surface are jointly used as supervision information. The optimal value of the parameters is processed by bone skinning to obtain the human body reconstruction model.

Further, after the human body reconstruction model is obtained, color processing may be performed on the human body reconstruction model based on the color information in the RGB image or the RGBD image, so that the human body reconstruction model matches the color information of the person in the image.

In the embodiment of the present disclosure, the target object in the image is reconstructed three-dimensionally through the three-dimensional reconstruction network, so as to obtain the initial value of the parameter, and then optimize the initial value of the parameter based on the supervision information, and establish the target object based on the optimized value of the parameter 3D model of . The advantage of the parameter optimization method is that it can give more accurate 3D reconstruction results that conform to the 2D observation characteristics of the image, but it often gives unnatural and unreasonable action results with low reliability. The network regression through the 3D reconstruction network can give more natural and reasonable action results. Therefore, using the output of the 3D reconstruction network as the initial value of the parameters for parameter optimization can ensure the reliability of the 3D reconstruction results. Taking into account the accuracy of 3D reconstruction.

In some embodiments, in the parameter optimization stage, a multi-stage optimization method may be used. The multi-stage optimization method may include a camera optimization stage and a pose optimization stage. In the camera optimization stage, the optimization targets are the value R of the global rotation parameter and the current value t of the displacement parameter between the image acquisition device and the target object. Among them, t and R are three-dimensional vectors, and R is expressed in the form of axis and angle. In the pose optimization stage, the optimization targets are the values of key point rotation parameters and body posture parameters.

Because in the optimization process, changing the position of the camera and changing the position of the 3D key points of the human body can cause changes in the 2D projection of the 3D key points, which will make the optimization process very unstable. Therefore, in the camera optimization stage, the human body pose is fixed, and in the pose optimization stage, the camera position is fixed, thereby improving the stability of the optimization process. That is, when the initial value of the body posture parameter and the initial value of the key point rotation parameter remain unchanged, based on the supervision information and the initial value of the displacement parameter, the current displacement parameter of the image acquisition device value and the initial value of the global rotation parameter are optimized to obtain the optimal value of the displacement parameter and the optimal value of the global rotation parameter; then keep the optimal value of the displacement parameter and the optimal value of the global rotation parameter unchanged, and based on the The optimized value and the optimized value of the global rotation parameter are optimized by optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter to obtain the optimized value of the key point rotation parameter and the optimized value of the body shape parameter.

Further, among the 2D projection key points corresponding to the 3D key points of the target object, target 2D projection key points belonging to preset parts of the target object can be acquired; wherein, the 3D key points of the target object are based on the The initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter are obtained; the two-dimensional projection key point is based on the current value of the displacement parameter and the initial value of the global rotation parameter for the target object The 3D key points of are obtained by projection. A first loss between the target 2D projection keypoint and the initial 2D keypoint is obtained. A second loss between an initial value of the displacement parameter and a current value of the displacement parameter is obtained. The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss and the second loss.

Wherein, the preset part may be a trunk part, and the key points of the target two-dimensional projection may include left and right shoulder points, left and right hip points, spine center points and other key points. Since different actions have less influence on the key points of the torso, by using the key points of the torso to establish the first loss, the influence of different actions on the position of the key points can be reduced and the accuracy of the optimization result can be improved. The first loss can also be called torso key point projection loss, and the second loss can also be called camera displacement regularization loss. The first loss can be obtained by the following formula (1), and the second loss can be obtained by the following formula (2) get:

L _cam = ||tt _net || ₂ (2);

Among them, L _torso and L _cam denote the first loss and the second loss respectively, x _torso and

represent target two-dimensional projection key points and initial two-dimensional key points respectively, and t and t _net represent the current value of the displacement parameter between the image acquisition device and the target object and the initial value of the displacement parameter respectively. The first target loss L ₁ can be determined based on the first loss and the second loss. For example, the first target loss can be determined as the sum of the first loss and the second loss, which can be determined by the following formula (3) Sure:

L ₁ =L _torso +L _cam (3).

A third loss between an optimized 2D projection keypoint of the target object and the initial 2D keypoint may be obtained, wherein the optimized 2D projection keypoint is based on an optimized value of the displacement parameter and a global rotation parameter The optimized value of is obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter. The fourth loss is obtained, and the fourth loss is used to characterize the rationality of the posture corresponding to the optimal value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body posture parameter. Optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the third loss and the fourth loss.

The third loss can also be called the two-dimensional key point projection loss, the fourth loss can also be called the attitude rationality loss, and the third loss can be determined by the following formula (4):

where L _2d is the third loss, x and

represent the optimized two-dimensional projection key points and the initial two-dimensional key points respectively. The second target loss may be determined based on the third loss and the fourth loss. For example, the second target loss may be determined as the sum of the third loss and the fourth loss, which may be determined by the following formula (5):

L ₂ =L _2d +L _prior (5);

Among them, L ₂ is the second target loss, and L _prior is the fourth loss, which can be obtained by using a Gaussian Mixture Model (GMM), which is used to judge the optimal value of the global rotation parameter, the initial and body posture of the key point rotation parameter Whether the attitude corresponding to the initial value of the parameter is reasonable, and output a large loss for the unreasonable attitude.

After optimizing the initial value of the key point rotation parameter and the initial value of the posture parameter based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the optimized value of the global rotation parameter may also be optimized, The optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized, that is, a three-stage optimization method is adopted. For the case where the supervision information includes the information of the 3D point cloud on the surface of the target object, the three-stage optimization method can be adopted, including the camera optimization stage, the attitude optimization stage and the point cloud optimization stage.

In the camera optimization stage, target two-dimensional projection key points belonging to preset parts of the target object among the two-dimensional projection key points corresponding to the three-dimensional key points of the target object can be obtained; wherein, the three-dimensional key points of the target object Based on the initial value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter, the two-dimensional projection key point is based on the current value of the displacement parameter and the initial value of the global rotation parameter. The 3D key points of the target object are obtained by projection. A first loss between the target 2D projection keypoint and the initial 2D keypoint is obtained. A second loss between an initial value of the displacement parameter and a current value of the displacement parameter is obtained. Acquiring a fifth loss between the first 3D point cloud of the surface of the target object and the initial 3D point cloud; wherein, the first 3D point cloud is based on the initial value of the global rotation parameter, the key point rotation parameter Initial values and initial values of body parameters are obtained. The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss. The fifth loss can also be called the nearest point iteration (Iterative Closest Point, ICP) point cloud registration loss, which can be determined by the following formula (6):

In the formula, L _icp is the fifth loss, the initial 3D point cloud is regarded as point cloud P, and the first 3D point cloud is regarded as point cloud Q, K ₁ ={(p,q)} is Each point in point cloud P is a set of point pairs formed by the closest point in point cloud Q, K ₂ ={(p,q)} is the point pair set from each point in point cloud Q to the closest point in point cloud P A set of point pairs composed of points. The first loss and the second loss are represented by the following formula (7) and formula (8) respectively:

L _cam = ||tt _net || ₂ (8);

respectively represent the target two-dimensional projection key point and the initial two-dimensional key point, t and t _net represent the current value of the displacement parameter and the initial value of the displacement parameter respectively. The first target loss L ₁ can be determined based on the sum of the first loss, the second loss and the fifth loss, and then optimize the current value of the displacement parameter and the initial value of the global rotation parameter based on the first target loss, that is, as The following formula (9):

L ₁ =L _torso +L _cam +L _icp (9).

The attitude optimization stage in the three-stage optimization process is the same as the attitude optimization stage in the two-stage optimization process, and will not be repeated here.

In the point cloud optimization stage, the sixth loss between the optimized 2D projection keypoint of the target object and the initial 2D keypoint can be obtained, wherein the optimized 2D projection keypoint is based on the displacement parameter The optimized value and the optimized value of the global rotation parameter are obtained by projecting the optimized three-dimensional key point of the target object, and the optimized three-dimensional key point is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the body shape parameter. The optimized value is obtained. A seventh loss is obtained, and the seventh loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter. Obtain an eighth loss between the second 3D point cloud of the surface of the target object and the initial 3D point cloud; wherein, the second 3D point cloud is based on the optimized value of the global rotation parameter and the key point rotation parameter The optimal value and the optimal value of body parameters are obtained. Jointly optimize the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the posture parameter and the optimized value of the displacement parameter based on the sixth loss, the seventh loss and the eighth loss , can be optimized by the following formula (10) and formula (11):

In the formula,

for the sixth loss,

To optimize the 2D projection keypoints,

is the initial two-dimensional keypoint. The seventh loss can be obtained by using a Gaussian mixture model, which is used to judge whether the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter is reasonable, and outputs a large loss for an unreasonable posture .

is the eighth loss, P is the initial 3D point cloud as a point cloud,

is the second 3D point cloud,

For each point in the point cloud P to the point cloud

A set of point pairs consisting of the closest points in the middle,

for the point cloud

A set of point pairs from each point in point cloud P to the nearest point in point cloud P. Further, the sum of the sixth loss, the seventh loss and the eighth loss can be determined as the third target loss L ₃ , and based on the third target loss, optimize the value of the global rotation parameter, the key point rotation parameter The optimal value of the optimized value, the optimized value of the body shape parameter and the optimized value of the displacement parameter are jointly optimized, and can be jointly optimized by the following formula (12):

L ₃ =L _2d +L _prior +L _icp (12).

In the case where the image of the target object is an RGB image, parameter optimization can be performed based on the aforementioned two-stage optimization method including the camera optimization stage and the attitude optimization stage; The parameters are optimized by the three-stage optimization method of the stage, attitude optimization stage and point cloud optimization stage.

This solution can be used in a wide range of scenarios, and can provide natural, reasonable and accurate human body reconstruction models in scenarios such as virtual fitting rooms, virtual anchors, and video action migration.

As shown in FIG. 4A , it is a schematic diagram of an application scene of a virtual fitting room according to an embodiment of the present disclosure. The image of the user 401 can be collected by the camera 403, and the collected image is sent to a processor (not shown in the figure) for three-dimensional human body reconstruction, so as to obtain the human body reconstruction model 404 corresponding to the user 401, and the human body reconstruction model 404 is displayed on The display interface 402 is for the user 401 to watch. At the same time, the user 401 can select the required clothing 405, including but not limited to clothing 4051 and hat 4052, etc., and the clothing 405 can be displayed on the display interface 402 based on the human body reconstruction model 404, so that the user 401 can watch the wearing effect of the clothing 405.

As shown in FIG. 4B , it is a schematic diagram of an application scenario of a virtual live broadcast room according to an embodiment of the present disclosure. During the live broadcast, the image of the anchor user 406 can be collected through the anchor client 407, and the image of the anchor user 406 can be sent to the server 408 for three-dimensional reconstruction to obtain the human body reconstruction model of the anchor user, that is, the virtual anchor. The server 408 can return the human body reconstruction model of the host user to the host client 407 for display, as shown in the model 4071 in the figure. In addition, the host client 407 can also collect the voice information of the host user, and send the voice information to the server 408, so that the server 408 can fuse the reconstruction model of the human body and the voice information. The server 408 can send the fused human body reconstruction model and voice information to the viewer client 409 watching the live program for display and playback, wherein the displayed human body reconstruction model is shown as model 4091 in the figure. Through the above method, the live broadcast screen of the virtual anchor can be displayed on the viewer client 409 .

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

As shown in FIG. 5 , the present disclosure also provides a three-dimensional reconstruction device, which includes:

The first three-dimensional reconstruction module 501 is configured to perform three-dimensional reconstruction on the target object in the image through a three-dimensional reconstruction network to obtain an initial value of a parameter of the target object, and the initial value of the parameter is used to establish a three-dimensional model of the target object ;

An optimization module 502, configured to optimize the initial value of the parameter based on the pre-acquired supervisory information used to represent the characteristics of the target object, to obtain the optimized value of the parameter;

The second three-dimensional reconstruction module 503 is configured to perform bone skinning processing based on the optimized values of the parameters, and establish a three-dimensional model of the target object.

In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the methods described in the method embodiments above, and its specific implementation can refer to the description of the method embodiments above. For brevity, here No longer.

As shown in FIG. 6, the present disclosure also provides a three-dimensional reconstruction system, which includes:

An image acquisition device 601, configured to acquire an image of a target object; and

The processing unit 602 communicated with the image acquisition device 601 is configured to perform three-dimensional reconstruction on the target object in the image through the three-dimensional reconstruction network to obtain the initial value of the parameter of the target object, and the initial value of the parameter is used To establish a three-dimensional model of the target object; optimize the initial value of the parameter based on the pre-acquired supervisory information used to represent the characteristics of the target object to obtain the optimized value of the parameter; Skeletal skinning processing to establish a 3D model of the target object.

The image acquisition device 601 in the embodiment of the present disclosure may be a device with an image acquisition function such as a camera or a camera, and the images collected by the image acquisition device 601 may be transmitted to the processing unit 602 in real time, or stored, and transmitted from the storage space when needed to processing unit 602. The processing unit 602 may be a single server or a server cluster composed of multiple servers. For the method executed by the processing unit 602, refer to the above-mentioned embodiment of the three-dimensional reconstruction method for details, and details are not repeated here.

The embodiment of this specification also provides a computer device, which at least includes a memory, a processor, and a computer program stored on the memory and operable on the processor, wherein, when the processor executes the program, the computer program described in any of the preceding embodiments is implemented. described method.

FIG. 7 shows a schematic diagram of a more specific hardware structure of a computing device provided by the embodiment of this specification. The device may include: a processor 701 , a memory 702 , an input/output interface 703 , a communication interface 704 and a bus 705 . The processor 701 , the memory 702 , the input/output interface 703 and the communication interface 704 are connected to each other within the device through the bus 705 .

The processor 701 may be implemented by a general-purpose CPU (Central Processing Unit, central processing unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute related programs to realize the technical solutions provided by the embodiments of this specification. The processor 701 may also include a graphics card, and the graphics card may be an Nvidia titan X graphics card or a 1080Ti graphics card.

The memory 702 can be implemented in the form of ROM (Read Only Memory, read-only memory), RAM (Random Access Memory, random access memory), static storage device, dynamic storage device, etc. The memory 702 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 702 and invoked by the processor 701 for execution.

The input/output interface 703 is used to connect the input/output module to realize information input and output. The input/output/module can be configured in the device as a component (not shown in the figure), or can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, and the like.

The communication interface 704 is used to connect with a communication module (not shown in the figure), so as to realize communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.), and can also realize communication through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

Bus 705 includes a path for transferring information between the various components of the device (eg, processor 701, memory 702, input/output interface 703, and communication interface 704).

It should be noted that although the above device only shows the processor 701, the memory 702, the input/output interface 703, the communication interface 704, and the bus 705, in the specific implementation process, the device may also include other components. In addition, those skilled in the art can understand that the above-mentioned device may only include components necessary to implement the solutions of the embodiments of this specification, and does not necessarily include all the components shown in the figure.

An embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the method described in any one of the foregoing embodiments is implemented.

Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can be implemented by any method or technology for storage of information. Information may be computer readable instructions, data structures, modules of a program, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Flash memory or other memory technology, Compact Disc Read-Only Memory (CD-ROM), Digital Versatile Disc (DVD) or other optical storage, Magnetic tape cartridge, tape magnetic disk storage or other magnetic storage device or any other non-transmission medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media excludes transitory computer-readable media, such as modulated data signals and carrier waves.

It can be known from the above description of the implementation manners that those skilled in the art can clearly understand that the embodiments of this specification can be implemented by means of software plus a necessary general hardware platform. Based on this understanding, the essence of the technical solutions of the embodiments of this specification or the part that contributes to the prior art can be embodied in the form of software products, and the computer software products can be stored in storage media, such as ROM/RAM, A magnetic disk, an optical disk, etc., include several instructions to enable a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of this specification.

The systems, devices, modules, or units described in the above embodiments can be specifically implemented by computer chips or entities, or by products with certain functions. A typical implementing device is a computer, which may take the form of a personal computer, laptop computer, cellular phone, camera phone, smart phone, personal digital assistant, media player, navigation device, e-mail device, game control device, etc. desktops, tablets, wearables, or any combination of these.

Each embodiment in this specification is described in a progressive manner, the same and similar parts of each embodiment can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, as for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for relevant parts, please refer to part of the description of the method embodiment. The device embodiments described above are only illustrative, and the modules described as separate components may or may not be physically separated, and the functions of each module may be integrated in the same or multiple software and/or hardware implementations. Part or all of the modules can also be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without creative effort.

Claims

A three-dimensional reconstruction method, said method comprising:

Performing three-dimensional reconstruction on the target object in the image through a three-dimensional reconstruction network to obtain an initial value of a parameter of the target object, wherein the initial value of the parameter is used to establish a three-dimensional model of the target object;

Optimizing the initial value of the parameter based on the pre-acquired supervision information used to represent the characteristics of the target object to obtain the optimized value of the parameter;

Skeleton skinning processing is performed based on the optimized values of the parameters, and a three-dimensional model of the target object is established.
The method according to claim 1, wherein the supervision information comprises first supervision information, or the supervision information comprises first supervision information and second supervision information;

The first supervisory information includes at least one of the following: initial two-dimensional key points of the target object, semantic information of multiple pixels on the target object in the image;

The second supervisory information includes an initial three-dimensional point cloud of the surface of the target object.
The method according to claim 2, further comprising:

The initial two-dimensional key point information of the target object is extracted from the image through a key point extraction network.
The method according to claim 2 or 3, wherein the image comprises a depth image of the target object; the method further comprises:

extracting depth information of the plurality of pixel points on the target object from the depth image;

Back-projecting the plurality of pixel points on the target object in the depth image to a three-dimensional space based on the depth information to obtain the initial three-dimensional point cloud of the surface of the target object.
The method according to claim 4, wherein the image further comprises an RGB image of the target object; extracting the depth information of the plurality of pixels on the target object from the depth image comprises:

Carry out image segmentation to described RGB image;

Determining the image area where the target object is located in the RGB image based on the image segmentation result;

determining the image area where the target object is located in the depth image based on the image area where the target object is located in the RGB image;

Depth information of the plurality of pixels in the image area where the target object is located in the depth image is acquired.
The method according to any one of claims 2 to 5, wherein the method further comprises:

Filter out outliers from the initial three-dimensional point cloud, and use the filtered initial three-dimensional point cloud as the second supervisory information.
The method according to any one of claims 1 to 6, wherein the image of the target object is acquired by an image acquisition device, and the parameters include: the global rotation parameter of the target object, each Key point rotation parameters of key points, body posture parameters of the target object, and displacement parameters of the image acquisition device;

Optimizing initial values of the parameters based on pre-acquired supervisory information representing features of the target object includes:

Under the condition that the initial value of the posture parameter and the initial value of the key point rotation parameter remain unchanged, based on the supervision information and the initial value of the displacement parameter, the displacement parameter of the image acquisition device is adjusted. Optimizing the current value of the current value and the initial value of the global rotation parameter to obtain the optimized value of the displacement parameter and the optimized value of the global rotation parameter;

Based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, optimize the initial value of the key point rotation parameter and the initial value of the body posture parameter, and obtain the optimized value of the key point rotation parameter and the optimal value of the key point rotation parameter. Optimum values for the body parameters.
The method according to claim 7, wherein the supervisory information includes initial two-dimensional key points of the target object;

Optimizing the current value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter based on the supervisory information and the initial value of the displacement parameter, including:

Acquiring the target two-dimensional projection key points corresponding to the two-dimensional projection key points corresponding to the three-dimensional key points of the target object, which belong to the preset part of the target object; wherein, the three-dimensional key points of the target object are based on the global rotation parameter The initial value of the initial value of the key point rotation parameter and the initial value of the body shape parameter are obtained, and the key point of the two-dimensional projection is based on the current value of the displacement parameter and the initial value of the global rotation parameter. The three-dimensional key points of the object are obtained by projection;

obtaining a first loss between the target 2D projected keypoint and the initial 2D keypoint;

obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter;

The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss and the second loss.
The method according to claim 7 or 8, wherein the supervisory information includes the initial two-dimensional key points of the target object; based on the optimized value of the displacement parameter and the optimized value of the global rotation parameter, the The initial value of the key point rotation parameter and the initial value of the body shape parameter are optimized, including:

Obtaining a third loss between the optimized two-dimensional projection keypoint of the target object and the initial two-dimensional keypoint, wherein the optimized two-dimensional projection keypoint is based on the optimized value of the displacement parameter and the global rotation The optimized value of the parameter is obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the initial value of the key point rotation parameter and the initial value of the body shape parameter worth it;

Obtaining a fourth loss, the fourth loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the initial value of the key point rotation parameter, and the initial value of the posture parameter;

Optimizing the initial value of the key point rotation parameter and the initial value of the body shape parameter based on the third loss and the fourth loss.
The method according to any one of claims 7 to 9, wherein the initial value of the key point rotation parameter and the body posture are adjusted based on the optimal value of the displacement parameter and the optimal value of the global rotation parameter. After the initial values of the parameters are optimized, the method further includes:

Joint optimization is performed on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter, and the optimized value of the displacement parameter.
The method according to claim 10, wherein the supervision information includes initial two-dimensional key points of the target object and an initial three-dimensional point cloud of the surface of the target object; based on the supervision information and the displacement parameter The initial value of the initial value of the displacement parameter of the image acquisition device and the initial value of the global rotation parameter are optimized, including:

Acquiring the target two-dimensional projection key points corresponding to the two-dimensional projection key points corresponding to the three-dimensional key points of the target object, which belong to the preset part of the target object; wherein, the three-dimensional key points of the target object are based on the global rotation parameter The initial value of the initial value of the key point rotation parameter and the initial value of the body shape parameter are obtained, and the key point of the two-dimensional projection is based on the current value of the displacement parameter and the initial value of the global rotation parameter. The 3D key points of the target object are obtained by projection;

obtaining a first loss between the target 2D projected keypoint and the initial 2D keypoint;

obtaining a second loss between an initial value of the displacement parameter and a current value of the displacement parameter;

Obtain a fifth loss between the first 3D point cloud of the surface of the target object and the initial 3D point cloud; wherein, the first 3D point cloud is based on the initial value of the global rotation parameter, the key point rotation The initial value of the parameter and the initial value of the posture parameter are obtained;

The current value of the displacement parameter and the initial value of the global rotation parameter are optimized based on the first loss, the second loss and the fifth loss.
The method according to claim 10 or 11, wherein the optimization value of the global rotation parameter, the optimization value of the key point rotation parameter, the optimization value of the body shape parameter and the optimization value of the displacement parameter Perform joint optimization, including:

Obtaining a sixth loss between the optimized two-dimensional projection keypoint of the target object and the initial two-dimensional keypoint, wherein the optimized two-dimensional projection keypoint is based on the optimized value of the displacement parameter and the global rotation The optimized value of the parameter is obtained by projecting the optimized 3D key point of the target object, and the optimized 3D key point is obtained based on the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter and the optimized body shape parameter worth it;

Obtaining a seventh loss, the seventh loss is used to characterize the rationality of the posture corresponding to the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, and the optimized value of the posture parameter;

Obtain an eighth loss between the second 3D point cloud of the surface of the target object and the initial 3D point cloud; the second 3D point cloud is based on the optimized value of the global rotation parameter, the key point rotation parameter Optimal value and the optimal value of described posture parameter are obtained;

Based on the sixth loss, the seventh loss and the eighth loss, the optimized value of the global rotation parameter, the optimized value of the key point rotation parameter, the optimized value of the body shape parameter and the displacement parameter The optimized value is jointly optimized.
A three-dimensional reconstruction device, said device comprising:

The first three-dimensional reconstruction module is configured to perform three-dimensional reconstruction on the target object in the image through a three-dimensional reconstruction network to obtain an initial value of a parameter of the target object, wherein the initial value of the parameter is used to establish a three-dimensional image of the target object Model;

An optimization module, configured to optimize the initial value of the parameter based on the pre-acquired supervisory information representing the characteristics of the target object, to obtain the optimized value of the parameter;

The second three-dimensional reconstruction module is configured to perform bone skinning processing based on the optimized values of the parameters, and establish a three-dimensional model of the target object.
A three-dimensional reconstruction system, the system comprising:

an image capture device for capturing an image of the target object; and

A processing unit communicatively connected to the image acquisition device, configured to perform three-dimensional reconstruction on the target object in the image through a three-dimensional reconstruction network to obtain an initial value of a parameter of the target object, wherein the initial value of the parameter The value is used to establish the three-dimensional model of the target object; the initial value of the parameter is optimized based on the pre-acquired supervisory information representing the characteristics of the target object to obtain the optimized value of the parameter; based on the parameter The optimal value is used for bone skinning processing to establish a 3D model of the target object.
A computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented.
A computer device, comprising a memory, a processor, and a computer program stored in the memory and operable on the processor, the processor implementing the method according to any one of claims 1 to 12 when executing the program.