WO2022110790A1

WO2022110790A1 - Face reconstruction method and apparatus, computer device, and storage medium

Info

Publication number: WO2022110790A1
Application number: PCT/CN2021/102404
Authority: WO
Inventors: 徐胜伟; 王权; 钱晨
Original assignee: 北京市商汤科技开发有限公司
Priority date: 2020-11-25
Filing date: 2021-06-25
Publication date: 2022-06-02
Also published as: JP2023507862A; TW202221652A; TWI778723B; KR20230110607A; CN112419485A; CN112419485B

Abstract

Provided in the present disclosure are a face reconstruction method and apparatus, a computer device, and a storage medium, the method comprising: on the basis of a target image, generating a first real face model; using a pre-generated plurality of second real face models to perform fitting processing on the first real face model to obtain a fitting coefficient corresponding respectively to the plurality of second real face models; and, on the basis of the fitting coefficient corresponding respectively to the plurality of second real face models and a virtual face model with a preset style corresponding respectively to the plurality of second real face models, generating a target virtual face model corresponding to the target image.

Description

Method, device, computer equipment and storage medium for reconstructing human face

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application claims the priority of the Chinese patent application filed on November 25, 2020, with the application number of 202011342169.7 and the invention titled "A method, device, computer equipment and storage medium for face reconstruction", which is incorporated by reference manner is incorporated into this article.

technical field

The present disclosure relates to the technical field of image processing, and in particular, to a method, an apparatus, a computer device and a storage medium for reconstructing a human face.

Background technique

Usually, a three-dimensional model of a virtual face can be established according to a real face or one's own preferences, so as to realize the reconstruction of the face, which has a wide range of applications in the fields of games, animation, and virtual social interaction. For example, in a game, the player can use the face reconstruction system provided by the game program to generate a virtual face 3D model according to the real face included in the image provided by the player, and use the generated virtual face 3D model to have a more sense of substitution participation in the game.

At present, when performing face reconstruction based on real faces in portrait images, face contour features are usually extracted based on face images, and then the extracted face contour features are matched and fused with the pre-generated virtual three-dimensional model. In order to generate a virtual face 3D model corresponding to the real face; however, due to the low matching degree between the face contour features and the pre-generated virtual 3D model, the generated virtual face 3D model and the real face image have a low matching degree. Similarity is low.

SUMMARY OF THE INVENTION

The embodiments of the present disclosure provide at least a method, an apparatus, a computer device, and a storage medium for reconstructing a human face.

In a first aspect, an embodiment of the present disclosure provides a method for reconstructing a face, including: generating a first real face model based on a target image; performing a fitting process on the face model to obtain fitting coefficients corresponding to a plurality of second real face models respectively; based on the fitting coefficients corresponding to the plurality of second real face models respectively, and The face models respectively correspond to virtual face models with preset styles, and a target virtual face model corresponding to the target image is generated.

In this embodiment, the fitting coefficient is used as a medium to establish a plurality of associations between the second real face models and the first real face The association between the virtual face model and the target virtual face model established based on the first real face model enables the target virtual face model determined based on the fitting coefficient to have both the preset style and the first real face The features of the original face corresponding to the model, the generated target virtual face model, and the original face corresponding to the first real face model have a higher degree of similarity.

In an optional embodiment, the fitting coefficients corresponding to the plurality of second real face models and the virtual simulations with preset styles corresponding to the plurality of second real face models respectively. face model, generating a target virtual face model corresponding to the target image, including: based on the fitting coefficients corresponding to the plurality of second real face models respectively, and the corresponding corresponding virtual face models The skeleton data is used to determine target skeleton data; based on the target skeleton data, the target virtual face model is generated.

In an optional embodiment, generating the target virtual face model based on the target skeleton data includes: based on the target skeleton data, and standard skeleton data and standard skin in the standard virtual face model. According to the relationship between the data, the standard skin data is subjected to position transformation processing to generate target skin data; based on the target bone data and the target skin data, the target virtual face model is generated.

In this embodiment, the target skin data and the correlation between the standard bone data and the standard skin data in the standard virtual face model can be used, so that the obtained target virtual face model can better combine the target bone data with the standard skin data. The target skin data is fitted, so that in the target virtual face model formed by using the target bone data, the target virtual face model caused by the change of the bone data shows less abnormal convexity or concave.

In an optional embodiment, the bone data corresponding to the virtual face model includes at least one of the following data: bone rotation data corresponding to each face bone in the multiple face bones of the virtual face model, Bone position data, and bone scaling data; the target bone data includes at least one of the following data: target bone position data, target bone scaling data, and target bone rotation data.

In this embodiment, the bone data corresponding to each of the multiple face bones can be more accurately represented by using the bone data, and the target virtual face model can be determined more accurately by using the target bone data.

In an optional implementation manner, the target skeleton data is determined based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively, including: : Based on the fitting coefficients corresponding to the plurality of second real face models respectively, perform interpolation processing on the bone position data corresponding to the plurality of virtual face models to obtain the target bone position data.

In an optional implementation manner, the target skeleton data is determined based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively, including: : based on the fitting coefficients corresponding to the plurality of second real face models respectively, perform interpolation processing on the bone scaling data corresponding to the plurality of virtual face models to obtain the target bone scaling data.

In an optional implementation manner, the target skeleton data is determined based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively, including: : converting the skeleton rotation data corresponding to the plurality of virtual face models into quaternions, and performing regularization processing on the quaternions corresponding to the plurality of virtual face models respectively, to obtain a regularized quaternion; Based on the fitting coefficients corresponding to the plurality of second real face models, interpolation processing is performed on the regularization quaternions corresponding to the plurality of virtual face models to obtain target bone rotation data.

In this embodiment, the target skeleton data can be used to adjust the virtual face model more accurately, so that the skeleton details in the obtained target virtual face model are more detailed and more similar to the skeleton details of the original face, so that the target The virtual face model has a higher similarity with the original face.

In an optional implementation manner, the generating the first real face model based on the target image includes: acquiring a target image including an original face; performing a three-dimensional face analysis on the original face included in the target image. Reconstruction to obtain the first real face model.

In this embodiment, the face features of the original face in the target image can be more accurately and comprehensively represented by using the first real face model obtained by reconstructing the original face in three dimensions.

In an optional implementation manner, for each second real face model in the plurality of second real face models, the second real face model is generated according to the following manner: obtaining a multiple reference images; for each reference image in the multiple reference images, perform three-dimensional face reconstruction on the reference face included in the reference image to obtain the second real face corresponding to the reference image face model.

In this embodiment, by using multiple reference images, it is possible to cover a wider range of face shape features as much as possible. Therefore, the second real face model obtained by performing 3D face reconstruction based on each reference image in the multiple reference images is also the same as It can cover as wide a range of facial features as possible.

In an optional implementation manner, the first real face model is fitted by using a plurality of pre-generated second real face models, and the corresponding simulations of the plurality of second real face models are obtained. The fitting coefficients include: performing least squares processing on a plurality of the second real face models and the first real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively.

In this embodiment, by using the fitting coefficient, the fitting situation when the first real face model is fitted by using a plurality of second real face models can be accurately represented.

In a second aspect, an embodiment of the present disclosure further provides an apparatus for reconstructing a human face, including:

a first generation module for generating a first real face model based on the target image;

a processing module, configured to perform fitting processing on the first real face model by using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively;

a second generation module, configured to be based on the fitting coefficients corresponding to the plurality of second real face models and the virtual face models with preset styles corresponding to the plurality of second real face models respectively, A target virtual face model corresponding to the target image is generated.

In an optional embodiment, the second generation module is based on the fitting coefficients corresponding to the plurality of second real face models and the The virtual face model of the preset style, when generating the target virtual face model corresponding to the target image, is used for: based on the fitting coefficients corresponding to the plurality of second real face models respectively, and a plurality of the The skeleton data corresponding to the virtual face models are determined to determine target skeleton data; based on the target skeleton data, the target virtual face model is generated.

In an optional embodiment, when the second generation module generates the target virtual face model based on the target skeleton data, it is used for: based on the target skeleton data and the standard virtual face model. The relationship between the standard bone data and the standard skin data, the standard skin data is subjected to position transformation processing, and the target skin data is generated; based on the target bone data and the target skin data, the target virtual data is generated. face model.

In an optional implementation manner, the second generation module determines based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively. When the target skeleton data is used, it is used to: perform interpolation processing on the skeleton position data corresponding to the plurality of virtual face models based on the fitting coefficients corresponding to the plurality of second real face models respectively, to obtain the target skeleton position data .

In an optional implementation manner, the second generation module determines based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively. When the target skeleton data is used, it is used to: perform interpolation processing on the skeleton scaling data corresponding to the plurality of virtual face models based on the fitting coefficients corresponding to the plurality of second real face models respectively, to obtain the target skeleton scaling data .

In an optional implementation manner, the second generation module determines based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively. When the target skeleton data is used, it is used to: convert the skeleton rotation data corresponding to the plurality of virtual face models into quaternions, and perform regularization processing on the quaternions corresponding to the plurality of virtual face models respectively, obtaining a regularized quaternion; based on the fitting coefficients corresponding to the multiple second real face models respectively, perform interpolation processing on the regularized quaternions corresponding to the multiple virtual face models respectively, to obtain the target bone rotation data.

In an optional embodiment, when generating the first real face model based on the target image, the first generation module is used to: acquire a target image including the original face; Three-dimensional face reconstruction is performed on the original face to obtain the first real face model.

In an optional implementation manner, the processing module generates the second real face model for each second real face model in the plurality of second real face models according to the following manner: obtaining the second real face model includes: Multiple reference images of the reference face; for each reference image in the multiple reference images, perform three-dimensional face reconstruction on the reference face included in the reference image, and obtain the first reference image corresponding to the reference image. Two real face models.

In an optional implementation manner, the processing module uses a plurality of pre-generated second real face models to perform fitting processing on the first real face model, and obtains a plurality of second real face models corresponding to When the fitting coefficient is , it is used to: perform least squares processing on a plurality of the second real face models and the first real face models to obtain the corresponding fitting coefficients of the plurality of second real face models respectively. Combined coefficient.

In a third aspect, an optional implementation manner of the present disclosure further provides a computer device, a processor, and a memory, where the memory stores machine-readable instructions executable by the processor, and the processor is configured to execute the instructions stored in the memory. machine-readable instructions, when the machine-readable instructions are executed by the processor, when the machine-readable instructions are executed by the processor, the above-mentioned first aspect, or any possible implementation of the first aspect, is executed steps in the method.

In a fourth aspect, an optional implementation manner of the present disclosure further provides a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and the computer program executes the first aspect, or any of the first aspect, when the computer program is run. steps in one possible implementation.

For a description of the effects of the above-mentioned apparatus for reconstructing a human face, computer equipment, and a computer-readable storage medium, please refer to the description of the above-mentioned method for reconstructing a human face, which will not be repeated here.

In order to make the above-mentioned objects, features and advantages of the present disclosure more obvious and easy to understand, the preferred embodiments are exemplified below, and are described in detail as follows in conjunction with the accompanying drawings.

Description of drawings

In order to explain the technical solutions of the embodiments of the present disclosure more clearly, the following briefly introduces the accompanying drawings required in the embodiments, which are incorporated into the specification and constitute a part of the specification. The drawings illustrate embodiments consistent with the present disclosure, and together with the description serve to explain the technical solutions of the present disclosure. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as limiting the scope. Other related figures are obtained from these figures.

FIG. 1 shows a flowchart of a method for reconstructing a human face provided by an embodiment of the present disclosure;

2 shows a flowchart of a method for generating a target virtual face model corresponding to a target image provided by an embodiment of the present disclosure;

3 shows a flowchart of a specific method for generating a target virtual face model corresponding to a first real face model based on target skeleton data provided by an embodiment of the present disclosure;

FIG. 4 shows an example of multiple faces and face models involved in the method for reconstructing a face provided by an embodiment of the present disclosure;

FIG. 5 shows a schematic diagram of an apparatus for reconstructing a human face provided by an embodiment of the present disclosure;

FIG. 6 shows a schematic diagram of a computer device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purposes, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only These are some, but not all, embodiments of the present disclosure. The components of the disclosed embodiments generally described and illustrated herein may be arranged and designed in a variety of different configurations. Thus, the following detailed description of the embodiments of the present disclosure is not intended to limit the scope of the disclosure as claimed, but is merely representative of selected embodiments of the disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the protection scope of the present disclosure.

After research, it is found that the method of face reconstruction can establish a three-dimensional model of virtual face according to the real face or one's own preferences. Among them, when performing face reconstruction based on the real face in the portrait image, the feature extraction of the real face in the portrait image is usually performed first to obtain the face contour feature, and then the face contour feature is combined with the pre-generated virtual three-dimensional feature. The features in the model are matched, and based on the matching results, the face contour features are fused with the virtual three-dimensional model to obtain a virtual three-dimensional model of the face corresponding to the real face in the portrait image. Since the matching accuracy is low when the face contour features are matched with the features in the pre-generated virtual three-dimensional model, the matching error between the virtual three-dimensional model and the face contour features is relatively large, and it is easy to cause the matching results based on the matching results. The problem of low similarity between the virtual face 3D model obtained by fusing the face contour feature and the face virtual 3D model and the face in the portrait image.

In view of the defects of the above solutions, the embodiments of the present disclosure provide a method for reconstructing a human face, which can generate a target virtual face model with a preset style, and the target virtual face model has a relatively high difference between the target virtual face model and the real face. high similarity.

In order to facilitate the understanding of this embodiment, a method for reconstructing a face disclosed by an embodiment of the present disclosure is first introduced in detail. The execution subject of the method for reconstructing a face provided by the embodiment of the present disclosure is generally a computer with a certain computing capability. equipment, the computer equipment for example includes: terminal equipment or server or other processing equipment, the terminal equipment can be user equipment (User Equipment, UE), mobile equipment, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant) Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. In some possible implementations, the method for reconstructing a human face may be implemented by the processor calling computer-readable instructions stored in the memory.

The following describes the method for reconstructing a human face provided by the embodiments of the present disclosure.

FIG. 1 is a flowchart of a method for reconstructing a face provided by an embodiment of the present disclosure. As shown in FIG. 1 , the method includes steps S101 to S103, wherein:

S101: Generate a first real face model based on the target image;

S102: Perform fitting processing on the first real face model by using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively;

S103: Generate a target virtual person corresponding to the target image based on the fitting coefficients corresponding to the plurality of second real face models and the virtual face models with preset styles corresponding to the plurality of second real face models respectively face model.

In the embodiment of the present disclosure, a process of fitting a first real face model with multiple pre-generated real face models is used to obtain fitting coefficients corresponding to a second real face model, and the fitting coefficients are used as a medium to establish multiple The relationship between a second real face model and the first real face model, and then use the fitting coefficient and the virtual faces with preset styles corresponding to the plurality of second real face models respectively. model, and generate a target virtual face model corresponding to the target image. This method makes the target virtual face model determined based on the fitting coefficient and the virtual face model not only has a preset style, but also has the characteristics of the original face corresponding to the first real face model, that is, the generated target There is a high degree of similarity between the virtual face model and the original face corresponding to the first real face model.

The above steps S101 to S103 will be described in detail below.

For the above-mentioned step S101, the target image is, for example, an acquired image including a human face. All faces can be used as original faces.

When the method for reconstructing a face provided by the embodiment of the present disclosure is applied to different scenarios, the method for acquiring the target image is also different.

For example, when the method for reconstructing a human face is applied to a game, an image containing the face of the game player can be acquired through an image acquisition device installed in the game device, or an image containing the game player's face can be selected from an album of the game device The image of the face of the game player, and the acquired image containing the face of the game player is used as the target image.

For another example, when the method for reconstructing a human face is applied to a terminal device such as a mobile phone, an image including the user's face can be collected by the camera of the terminal device, or an image including the user's face can be selected from an album of the terminal device, Or receive images containing the user's face from other applications installed in the terminal device.

For another example, when the method for reconstructing a human face is applied to a live broadcast scene, a video frame image containing a human face can be obtained from multiple frames of video frame images included in the video stream of the live broadcast device; image as the target image. Here, for example, the target image may have multiple frames; for example, the multiple-frame target image may be obtained by sampling a video stream.

When generating the first real face model based on the target image, for example, the following methods may be adopted: obtaining a target image including the original face; performing three-dimensional face reconstruction on the original face included in the target image to obtain the first real face Model.

Here, when performing three-dimensional face reconstruction on the original face included in the target image, for example, a three-dimensional deformable face model (3 Dimensions Morphable Models, 3DMM) can be used to obtain the first real face model corresponding to the original face. Wherein, the first real face model includes, for example, the position information of each key point in the preset camera coordinate system among the multiple key points of the original face in the target image.

For the above step S102, the second real face model is generated based on the reference image including the reference face. The reference faces in different reference images may be different; exemplarily, multiple people with different at least one of gender, age, skin color, degree of fatness and thinness, etc. may be selected, and the person of each person may be obtained for each of the multiple people face image, and use the acquired face image as a reference image. In this way, the plurality of second real face models obtained based on the plurality of reference images can cover a relatively wide range of face shape features as much as possible.

The reference face includes, for example, N faces corresponding to N different individual objects, (N is an integer greater than 1). In the case of acquiring multiple reference images including the reference face, for example, N photos corresponding to the N different individual objects can be obtained by separately photographing N different individual objects, and each photo is Corresponding to a reference face, the N photos are used as reference images; or, N reference images are determined from a plurality of images including different faces that have been photographed in advance.

Exemplarily, for each second real face model in the plurality of second real face models, the method for generating the second real face model includes: acquiring a plurality of reference images including a reference face; For each reference image in the reference images, three-dimensional face reconstruction is performed on the reference face included in the reference image to obtain a second real face model corresponding to the reference image.

Wherein, the method for performing 3D face reconstruction on the reference face is similar to the above-mentioned method for performing 3D face reconstruction on the original face, and will not be repeated here. The obtained second real face model includes position information of each key point in the preset camera coordinate system among the multiple key points of the reference face in the reference image. Here, the coordinate system of the second real face model and the coordinate system of the first real face model may be the same coordinate system.

When using multiple pre-generated second real face models to perform fitting processing on the first real face model, to obtain the fitting coefficients corresponding to the multiple second real face models, for example, the following method may be used: The second real face model and the first real face model are subjected to least squares processing to obtain fitting coefficients corresponding to the plurality of second real face models respectively.

Exemplarily, in the case of generating N second real face models in advance, the model data corresponding to the first real face model can be represented as D _a , and the model data corresponding to the second real face model can be represented as D _bi (i∈[1, N]), where D _bi represents the i-th second real face model among the N second real face models.

Using D _a to perform least squares processing on each of D _b1 to D _bN , N fitting values can be obtained, and the fitting values are expressed as α _i (i∈[1,N]). Among them, α _i represents the fitting value corresponding to the ith second real face model. Using the N fitted values, the fitting coefficient Alpha can be determined, for example, it can be represented by a coefficient matrix, that is,

Alpha=[α ₁ ,α ₂ ,...,α _N ].

Here, in the process of fitting the first real face model through multiple second real face models, it is necessary to make the data obtained after weighted summation of multiple second real face models through multiple fitting coefficients as close as possible to the data of the first real face model.

The fitting coefficient can also be regarded as an expression coefficient of each second real face model when the first real face model is expressed by using a plurality of second real face models. That is, by using the fitting values corresponding to the expression coefficients of the plurality of second real face models respectively, the second real face model can be transformed and fitted to the first real face model.

For the above step S103, the preset style may be, for example, a cartoon style, an ancient style, or an abstract style, and may be specifically set according to actual needs. Exemplarily, for the case where the preset style is a cartoon style, the virtual face model with the preset style may be, for example, a virtual face model with a certain cartoon style.

Wherein, the method for acquiring multiple virtual face models with preset styles corresponding to multiple second real face models respectively includes, for example, at least one of the following (a1) and (a2).

(a1) Taking obtaining a virtual face model corresponding to a second real face model as an example, a virtual face image with reference face features and a preset style can be designed and produced based on the reference image, and the virtual face image can be designed and produced based on the reference image. The virtual face in the image is subjected to three-dimensional modeling, and the skeleton data and skin data of the virtual face model in the virtual face image are obtained.

The skeleton data includes skeleton rotation data, skeleton scaling data, and skeleton position data of a plurality of bones preset for the virtual face in a preset coordinate system. Here, for example, multiple bones can be divided into multiple levels; for example, they include root (root) bones, facial bones, and facial detail bones; where facial bones can include eyebrow bones, nasal bones, cheekbones, mandibles, and mouth bones, etc. ; For example, the detailed bones of the facial features can further divide the bones of different facial features in detail. Specific settings can be made according to different styles of virtual image requirements, which are not limited here.

The skinned data includes position information of multiple position points on the surface of the virtual face in a preset coordinate system and information about the association relationship between each position point and at least one of the multiple bones.

The virtual model obtained by performing three-dimensional modeling on the virtual face in the virtual face image is used as the virtual face model corresponding to the second real face model.

(a2) Pre-generate standard virtual face models with preset styles. The standard virtual face model also includes standard bone data, standard skin data, and an association relationship between the standard bone data and the standard skin data. Based on the reference face, design and modify the standard skeleton data in the standard virtual face model, so that the modified standard virtual face model has the preset style and also includes the features of the reference face in the reference image. ; Then, based on the relationship between the standard bone data and the standard skin data, adjust the standard skin data, and add the feature information of the reference face to the standard skin data, based on the modified standard bones The data and the modified standard skinning data are used to generate a virtual face model corresponding to the second real face model.

Here, for the specific data representation of the virtual face model, reference may be made to the description in (a1) above, which will not be repeated here.

Referring to FIG. 2 , an embodiment of the present disclosure further provides a fitting coefficient corresponding to a plurality of second real face models, and a virtual simulation with a preset style corresponding to the plurality of second real face models respectively. The skeleton data of the face model, and the method for generating the target virtual face model corresponding to the target image, including:

S201: Determine target skeleton data based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively.

The bone data includes at least one of the following data: bone rotation data, bone position data, and bone scaling data corresponding to each face bone in the multiple face bones of the virtual face model.

In a possible implementation manner, based on the fitting coefficients corresponding to the plurality of second real faces, interpolation processing may be performed on the skeleton data corresponding to the plurality of virtual face models to obtain the target skeleton data. The obtained target bone data includes at least one of the following: target bone position data, target bone scaling data, and target bone rotation data.

Wherein, the target bone position data includes, for example, the three-dimensional coordinate value of the center point of the bone in the model coordinate system; the target bone scaling data includes, for example, the scaling ratio of the target bone relative to the bone in the standard virtual face model; the target bone rotation data includes, for example, the bone The rotation angle of the axis in the model coordinate system.

Exemplarily, the target skeleton data is determined based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models, for example, the following (b1) to (b3) can be used. to achieve at least one of them.

(b1) Based on the fitting coefficients corresponding to the plurality of second real face models respectively, perform interpolation processing on the bone position data corresponding to the plurality of virtual face models respectively to obtain target bone position data.

(b2) Based on the fitting coefficients corresponding to the plurality of second real face models respectively, perform interpolation processing on the skeleton scaling data corresponding to the plurality of virtual face models respectively to obtain the target skeleton scaling data.

(b3) converting the skeleton rotation data corresponding to the plurality of virtual face models respectively into quaternion data, and performing regularization processing on the quaternions corresponding to the plurality of virtual face models respectively, to obtain Regularization quaternion: Based on the fitting coefficients corresponding to the multiple second real face models, the regularization quaternion corresponding to the multiple virtual face models is interpolated to obtain the target bone rotation data.

In a specific implementation, for the above-mentioned methods (b1) and (b2), in the case of acquiring bone position data and bone scaling data, the method further includes determining each level of bones and the corresponding local coordinate system. Among them, when the face model is layered at the bone level, for example, the bone level can be determined directly according to the biological bone layering method, or the bone level can be determined according to the requirements of face reconstruction. The specific layering method can be determined according to the actual situation. The situation is determined and will not be repeated here.

After each bone level is determined, a bone coordinate system corresponding to each bone level can be established based on each bone level. Illustratively, each level of bone can be represented as Bone _i .

At this time, the bone position data may include the three-dimensional coordinate values of each level of bone Bone _i in the virtual face model in the corresponding bone coordinate system; the bone scaling data may include the corresponding bone Bone _i of each level in the virtual face model. The percentage used to represent the scale of the bone in the bone coordinate system, such as 80%, 90% or 100%.

In a possible implementation manner, the bone position data corresponding to the ith virtual face model is represented as Pos _i , and the bone scaling data corresponding to the ith virtual face model is represented as Scaling _i . At this time, the bone position data Pos _i includes the bone position data corresponding to the plurality of hierarchical bones respectively, and the bone scaling data Scaling _i includes the bone scaling data corresponding to the plurality of hierarchical bones respectively.

The corresponding fitting coefficient at this time is a _i . Based on the fitting coefficients corresponding to the M second real face models, interpolation processing is performed on the positional bone data Pos _i corresponding to the M virtual face models to obtain target bone position data.

Exemplarily, for example, the fitting coefficient may be used as the weight corresponding to each virtual face model, and the weighted summation processing is performed on the bone position data Pos _i corresponding to the virtual face model to implement the interpolation processing process. At this time, the target bone position data Pos _new satisfies the following formula (1):

Similarly, based on the fitting coefficients corresponding to the M second real face models, interpolation processing is performed on the bone scaling data corresponding to the M virtual face models to obtain target bone scaling data, wherein the i-th virtual face model is The corresponding bone scaling data is represented as Scaling _i , and the fitting coefficients corresponding to the M second real face models can be used as the weights of the corresponding virtual face models, and the bone scaling data corresponding to the M virtual face models can be performed. Weighted summation processing to realize interpolation processing on M virtual face models; in this case, the target bone scaling data Scaling _new satisfies the following formula (2):

For the above method (b3), the bone rotation data may include vector values used to represent the degree of rotation coordinate transformation of each bone in the virtual face model in the corresponding bone coordinate system, for example, including a rotation axis and a rotation angle. In a possible implementation manner, the bone rotation data corresponding to the ith virtual face model is represented as Trans _i . Since the rotation angle contained in the bone rotation data has the problem of gimbal deadlock, the bone rotation data is converted into a quaternion, and the quaternion is regularized to obtain the regularized quaternion data, which is expressed as Trans' _i , in order to prevent overfitting when the quaternion is directly weighted and summed.

When performing interpolation processing on the regularization quaternion Trans' _i corresponding to the M second real face models based on the fitting coefficients corresponding to the M second real face models, the M second real face models may also correspond to The fitting coefficient of is used as a weight, and the regularization quaternion corresponding to the M virtual face models is weighted and summed; in this case, the target bone rotation data Trans _new satisfies the following formula (3):

Based on the target bone position data Pos _new , the target bone scaling data Scaling _new and the target bone rotation data Trans _new obtained in the above (b1), (b2) and (b3), the target bone data can be determined, which is represented as Bone _new . Exemplarily, the target bone data can be represented as (Pos _new , Scaling _new , Trans _new ) in the form of a vector.

Following the above S201, the method for generating the target virtual face model corresponding to the target image further includes:

S202: Based on the target skeleton data, generate a target virtual face model.

Referring to FIG. 3 , a specific method for generating a target virtual face model corresponding to a first real face model based on target skeleton data provided by an embodiment of the present disclosure includes:

S301: Based on the target skeleton data and the relationship between the standard skeleton data and the standard skin data in the standard virtual face model, perform position transformation processing on the standard skin data to generate target skin data;

S302: Generate a target virtual face model based on the target bone data and the target skin data.

Wherein, the association relationship between the standard skeleton data and the standard skin data in the standard virtual face model is, for example, the association relationship between the standard skeleton data corresponding to each level of bones and the standard skin data. Based on this association, the skin can be bound to the bones in the virtual face model.

Using the target skeleton data and the relationship between the standard skeleton data and the standard skin data in the standard virtual face model, the skin data at the corresponding positions of multiple levels of bones can be subjected to position transformation, so that the generated target skin data The position of the corresponding level bone in the data can be consistent with the position in the corresponding target bone data.

Here, the relationship between the bone data in the standard virtual face model and the standard skin data includes: the coordinate value of each position point in the standard skin deformation data in the model coordinate system, the bone position data of the bone, the bone scaling The relationship between the data and at least one item of the bone rotation data.

When using the target skeleton data and the relationship between the standard skeleton data and the standard skin data in the standard virtual face model to perform position transformation processing on the skin data corresponding to the positions of multiple levels of bones, when the target skeleton data is determined In other words, in the case where at least one of the target bone position data, target bone scaling data and target bone rotation data of the target bone is determined, the above association relationship can be used to determine that after the bone is transformed from the standard bone data to the target bone data , the new coordinate values of each position point in the standard skinning data under the model coordinate system, so as to obtain the target virtual face model based on the new coordinate value of each position point in the standard skinning data under the model coordinate system Target skin data.

Using the target bone data, it is possible to determine the bones of each level used to construct the target virtual face model; and using the target skin data, it is possible to determine the skin that binds the model to the bones, thereby forming the target virtual face model.

The method of determining the target virtual face model may be to directly establish the target virtual face model based on the target skeleton data and the target skin data; Corresponding skeleton data at each level, and then use the target skin data to build the target virtual face model. The specific method for establishing the target virtual face model can be determined according to the actual situation, and will not be repeated here.

The embodiment of the present disclosure also provides an explanation of the specific process of obtaining the target virtual face model Mod _Aim corresponding to the original face A in the target image Pic _A by using the method for reconstructing a face provided by the embodiment of the present disclosure.

Determining the target virtual face model Mod _Aim includes the following steps (c1) to (c5):

(c1) Preparing materials; including: preparing materials for standard virtual face models; and preparing materials for virtual pictures.

When preparing the material of the standard virtual face model, take the cartoon style as the default style as an example, first set a cartoon style standard virtual face model Mod _Base .

When preparing the material of the virtual picture, collect 24 virtual pictures Pic ₁ to Pic ₂₄ ; the number of boys and girls corresponding to the virtual faces B ₁ to B ₂₄ in the collected 24 virtual pictures is balanced and includes as wide a range as possible. distribution of facial features.

(c2) face model reconstruction; including: using the original face A in the target image Pic _A to generate a first real face model Mod _fst ; and using the virtual faces B ₁ to B ₂₄ in the virtual picture to generate a second real face model Face model Mod _snd-1 ~ Mod _snd-24 .

When it is determined that the original face A generates the first real face model Mod _fst , firstly, the face in the target image is straightened and cropped, and then the pre-trained RGB is used to reconstruct the neural network to generate the first real face corresponding to the original face A. Face model Mod _fst . Similarly, by using the pre-trained RGB reconstruction neural network, the second real face models Mod _snd-1 to Mod _snd-24 corresponding to the virtual faces B ₁ to B ₂₄ respectively can be determined.

After determining the second real face models Mod _snd-1 to Mod _snd-24 , the method further includes: determining the second real face models Mod snd- ₁ to Mod _snd -24 by using a preset style and manual adjustment The corresponding virtual face models Mod _fic-1 to Mod _fic-24 with preset styles respectively.

(c3) fitting processing; wherein, including: using a plurality of second real face models to perform fitting processing on the first real face model to obtain the fitting coefficients corresponding to the plurality of second real face models respectively alph=[ alpha _snd-1 , alpha _snd-2 , …, alpha _snd-24 ].

When using a plurality of second real face models to fit the first real face model, the method of least squares is selected for fitting, and a 24-dimensional coefficient alpha is obtained.

(c4) Determining target bone data; wherein, when determining the target bone data, the following steps (c4-1) and (c4-2) are also included.

(c4-1) Read the skeleton data; wherein, the skeleton data includes: the skeleton position data Pos _i , respectively corresponding to the virtual face models Mod _fic-1 to Mod _fic-24 with preset styles under the bones Bone _i of each level. Bone scaling data Scaling _i and bone rotation data Trans _i .

(c4-2) Using the fitting coefficient alpha to perform differential processing on the skeleton data corresponding to the virtual face models Mod _fic-1 to Mod _fic-24 of the preset style, and generate target skeleton data Bone _new , where the target skeleton data includes the target Bone position data Pos _new , target bone scaling data Scaling _new and target bone rotation data Trans _new .

(c5) Generate the target virtual face model.

Use the target bone data to determine the bones of each level used to construct the target virtual face model, replace the target bone data with the standard virtual face model Mod _Base , and use the target skin data to determine the skeleton to bind the model to the bone. skin, and then use the pre-determined relationship between the standard skeleton data and the standard skin data in the standard virtual face model to generate a target virtual face model corresponding to the first real face model.

Referring to FIG. 4 , an example of specific data used in a plurality of processes included in the above-mentioned specific example provided by an embodiment of the present disclosure. Wherein, in Figure 4 a represents the target image, 41 represents the original face A; in Figure 4 b represents a schematic diagram of a cartoon-style standard virtual face model; in Figure 4 c represents the generated target corresponding to the first real face model Schematic diagram of the virtual face model.

Here, it is worth noting that the above steps (c1) to (c5) are only a specific example of a method for reconstructing a human face, and do not limit the method for reconstructing a human face provided by the embodiments of the present disclosure.

Those skilled in the art can understand that in the above method of the specific implementation, the writing order of each step does not mean a strict execution order but constitutes any limitation on the implementation process, and the specific execution order of each step should be based on its function and possible Internal logic is determined.

Based on the same inventive concept, an apparatus for reconstructing a face corresponding to the method for reconstructing a face is also provided in the embodiment of the present disclosure, because the principle of solving the problem by the apparatus in the embodiment of the present disclosure is the same as the above-mentioned method for reconstructing a face in the embodiment of the present disclosure. Similar, therefore, the implementation of the apparatus may refer to the implementation of the method, and repeated descriptions will not be repeated.

Referring to FIG. 5 , an embodiment of the present disclosure provides an apparatus for reconstructing a human face. The apparatus includes a first generating module 51 , a processing module 52 and a second generating module 53 .

The first generating module 51 is configured to generate a first real face model based on the target image.

The processing module 52 is configured to perform fitting processing on the first real face model by using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively.

The second generating module 53 is configured to be based on the fitting coefficients corresponding to the plurality of second real face models and a plurality of virtual people with preset styles corresponding to the plurality of second real face models respectively face model, generating a target virtual face model corresponding to the target image.

In an optional implementation manner, the second generation module 53 is based on the fitting coefficients corresponding to the plurality of second real face models and the corresponding fitting coefficients of the plurality of second real face models respectively. A virtual face model with a preset style, when generating a target virtual face model corresponding to the target image, is used for: based on the fitting coefficients corresponding to the plurality of second real face models respectively, and a plurality of According to the skeleton data corresponding to the virtual face models, target skeleton data is determined; based on the target skeleton data, the target virtual face model is generated.

In an optional embodiment, when the second generation module 53 generates the target virtual face model based on the target skeleton data, it is used to: based on the target skeleton data and the standard virtual face model The relationship between the standard bone data and the standard skin data, the standard skin data is subjected to position transformation processing, and the target skin data is generated; based on the target bone data and the target skin data, the target skin data is generated. Virtual face model.

In an optional embodiment, the skeleton data of the virtual face model includes at least one of the following data: skeleton rotation data corresponding to each face skeleton in the multiple face skeletons of the virtual face model, skeleton Position data and bone scaling data; the target bone data includes at least one of the following data: target bone position data, target bone scaling data, and target bone rotation data.

In an optional embodiment, the target skeleton data includes the target skeleton position data, and the second generation module 53 is based on the fitting coefficients corresponding to the plurality of second real face models, and the The skeleton data corresponding to each of the virtual face models, when determining the target skeleton data, is used for: based on the fitting coefficients corresponding to the plurality of second real face models respectively, for the corresponding skeleton data of the plurality of virtual face models The bone position data is subjected to interpolation processing to obtain the target bone position data.

In an optional embodiment, the target skeleton data includes the target skeleton scaling data, and the second generation module 53 is based on the fitting coefficients corresponding to the plurality of second real face models, and the The skeleton data corresponding to each of the virtual face models, when determining the target skeleton data, is used for: based on the fitting coefficients corresponding to the plurality of second real face models respectively, for the corresponding skeleton data of the plurality of virtual face models The bone scaling data is subjected to interpolation processing to obtain the target bone scaling data.

In an optional embodiment, the target skeleton data includes the target skeleton rotation data, and the second generation module 53 is based on the fitting coefficients corresponding to the plurality of second real face models, and the The skeleton data corresponding to each of the virtual face models, when determining the target skeleton data, is used for: converting the skeleton rotation data corresponding to the plurality of virtual face models respectively into quaternions, and performing analysis on the plurality of virtual face models. Perform regularization processing on the quaternions corresponding to the face models to obtain a regularized quaternion; based on the fitting coefficients corresponding to the plurality of second real face models, respectively The quaternion is used for interpolation processing to obtain the rotation data of the target bone.

In an optional embodiment, when generating the first real face model based on the target image, the first generation module 51 is used to: acquire a target image including the original face; 3D face reconstruction is performed on the original face to obtain the first real face model.

In an optional implementation manner, the processing module 52 generates the second real face model for each of the plurality of second real face models according to the following manner: obtaining the second real face model. including multiple reference images of the reference face; for each reference image in the multiple reference images, perform three-dimensional face reconstruction on the reference face included in the reference image to obtain the reference image corresponding to the reference image. The second real face model.

In an optional implementation manner, the processing module 52 performs fitting processing on the first real face model by using a plurality of pre-generated second real face models, and obtains a plurality of second real face models respectively. When the corresponding fitting coefficient is used, it is used to: perform least squares processing on a plurality of the second real face models and the first real face models, so as to obtain the corresponding fitting coefficients.

For the description of the processing flow of each module in the apparatus and the interaction flow between the modules, reference may be made to the relevant descriptions in the foregoing method embodiments, which will not be described in detail here.

As shown in FIG. 6 , an embodiment of the present disclosure further provides a computer device including a processor 61 and a memory 62 .

The memory 62 stores machine-readable instructions executable by the processor 61, and the processor 61 is used to execute the machine-readable instructions stored in the memory 62. When the machine-readable instructions are executed by the processor 61, the processor 61 executes the following steps : generating a first real face model based on the target image; using a plurality of pre-generated second real face models to perform fitting processing on the first real face model, to obtain a plurality of second real face models corresponding respectively fitting coefficients; based on the fitting coefficients corresponding to the plurality of second real face models respectively, and the virtual face models with preset styles corresponding to the plurality of second real face models respectively, generating The target virtual face model corresponding to the target image.

The above-mentioned memory 62 includes a memory 621 and an external memory 622; the memory 621 here is also called an internal memory, and is used to temporarily store the operation data in the processor 61 and the data exchanged with the external memory 622 such as the hard disk. The external memory 622 performs data exchange.

For the specific execution process of the above instruction, reference may be made to the method for reconstructing a human face described in the embodiments of the present disclosure, and details are not described herein again.

Embodiments of the present disclosure further provide a computer-readable storage medium, where a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the method for reconstructing a face described in the foregoing method embodiments is executed. Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

Embodiments of the present disclosure further provide a computer program product, where the computer program product carries program codes, and the instructions included in the program codes can be used to execute the methods for reconstructing a face described in the foregoing method embodiments. For details, please refer to the foregoing methods. The embodiments are not repeated here.

Wherein, the above-mentioned computer program product can be specifically implemented by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK), etc. Wait.

Those skilled in the art can clearly understand that, for the convenience and brevity of description, for the specific working process of the system and device described above, reference may be made to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided by the present disclosure, it should be understood that the disclosed system, apparatus and method may be implemented in other manners. The apparatus embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored, or not implemented. On the other hand, the shown or discussed mutual coupling or direct coupling or communication connection may be through some communication interfaces, indirect coupling or communication connection of devices or units, which may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution in this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically alone, or two or more units may be integrated into one unit.

The functions, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a processor-executable non-volatile computer-readable storage medium. Based on this understanding, the technical solutions of the present disclosure can be embodied in the form of software products in essence, or the parts that make contributions to the prior art or the parts of the technical solutions. The computer software products are stored in a storage medium, including Several instructions are used to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk and other media that can store program codes .

Finally, it should be noted that the above-mentioned embodiments are only specific implementations of the present disclosure, and are used to illustrate the technical solutions of the present disclosure, but not to limit them. The protection scope of the present disclosure is not limited to this, although the aforementioned The embodiments describe the present disclosure in detail, and those skilled in the art should understand that: any person skilled in the art can still modify the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure. Or can easily think of changes, or equivalently replace some of the technical features; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered in the present disclosure. within the scope of protection. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

A method of reconstructing a human face, comprising:

generating a first real face model based on the target image;

Perform fitting processing on the first real face model by using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively;

Based on the fitting coefficients corresponding to the plurality of second real face models, and the virtual face models with preset styles corresponding to the plurality of second real face models, respectively, generating a corresponding image corresponding to the target image The target virtual face model.
The method for reconstructing a human face according to claim 1, wherein the fitting coefficients corresponding to the plurality of second real face models and the corresponding fitting coefficients of the plurality of second real face models respectively The corresponding virtual face model with preset style generates a target virtual face model corresponding to the target image, including:

Determine target skeleton data based on the fitting coefficients corresponding to the plurality of second real face models and the skeleton data corresponding to the plurality of virtual face models respectively;

Based on the target skeleton data, the target virtual face model is generated.
The method for reconstructing a human face according to claim 2, wherein the generating the target virtual face model based on the target skeleton data comprises:

Based on the target skeleton data and the association relationship between the standard skeleton data and the standard skin data in the standard virtual face model, the standard skin data is subjected to position transformation processing to generate target skin data;

The target virtual face model is generated based on the target bone data and the target skin data.
The method for reconstructing a human face according to claim 2 or 3, wherein,

The skeleton data corresponding to the virtual face model includes at least one of the following data:

Bone rotation data corresponding to each face skeleton in the multiple face skeletons of the virtual face model,

bone position data, and

Bone scaling data;

The target bone data includes at least one of the following data:

target bone position data,

target bone scaling data, and

Target bone rotation data.
The method for reconstructing a human face according to claim 4, wherein the fitting coefficients corresponding to the plurality of second real face models and the bones corresponding to the plurality of virtual face models respectively data, determine the target bone data, including:

Based on the fitting coefficients corresponding to the plurality of second real face models, interpolation processing is performed on the bone position data corresponding to the plurality of virtual face models to obtain the target bone position data.
The method for reconstructing a human face according to claim 4 or 5, wherein the fitting coefficients corresponding to the plurality of second real face models and the corresponding corresponding virtual face models respectively , determine the target bone data, including:

Based on the respective fitting coefficients corresponding to the multiple second real face models, interpolation processing is performed on the bone scaling data corresponding to the multiple virtual face models, to obtain the target bone scaling data.
The method for reconstructing a face according to any one of claims 4 to 6, wherein the fitting coefficients corresponding to the plurality of second real face models and a plurality of the virtual faces The skeleton data corresponding to the model respectively determines the target skeleton data, including:

Convert the bone rotation data corresponding to the multiple virtual face models into quaternions, and

performing regularization processing on the quaternions corresponding to the plurality of virtual face models respectively to obtain a regularized quaternion;

Based on the fitting coefficients corresponding to the plurality of second real face models, interpolation processing is performed on the regularization quaternions corresponding to the plurality of virtual face models to obtain the target bone rotation data.
The method for reconstructing a human face according to any one of claims 1 to 7, wherein the generating the first real face model based on the target image comprises:

Get the target image including the original face;

Performing three-dimensional face reconstruction on the original face included in the target image to obtain the first real face model.
The method for reconstructing a human face according to any one of claims 1 to 8, wherein for each second real face model in the plurality of second real face models, the first real face model is generated according to the following manner Two real face models:

Obtain multiple reference images including reference faces;

For each reference image in the plurality of reference images, three-dimensional face reconstruction is performed on the reference face included in the reference image to obtain the second real face model corresponding to the reference image.
The method according to any one of claims 1-9, wherein the first real face model is fitted by using a plurality of pre-generated second real face models to obtain a plurality of first real face models. The fitting coefficients corresponding to the two real face models, including:

Least squares processing is performed on the plurality of second real face models and the first real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively.
A device for reconstructing a human face, comprising:

a first generation module for generating a first real face model based on the target image;

a processing module, configured to perform fitting processing on the first real face model by using a plurality of pre-generated second real face models to obtain fitting coefficients corresponding to the plurality of second real face models respectively;

a second generation module, configured to be based on the fitting coefficients corresponding to the plurality of second real face models and the virtual face models with preset styles corresponding to the plurality of second real face models respectively, A target virtual face model corresponding to the target image is generated.
A computer device comprising a processor and a memory, the memory storing machine-readable instructions executable by the processor, the processor for executing the machine-readable instructions stored in the memory, the machine-readable instructions When the instructions are executed by the processor, the processor executes the method for reconstructing a human face according to any one of claims 1 to 10.
A computer-readable storage medium on which a computer program is stored, and when the computer program is run by a computer device, the computer device executes the method for reconstructing a human face according to any one of claims 1 to 10.