CN106023288A - Image-based dynamic substitute construction method - Google Patents

Abstract

The invention discloses an image-based dynamic avatar construction method. The method firstly carries out data collection and preprocessing: using a common network camera to collect a series of user face images with set action expressions, and segment these images , feature point calibration and other preprocessing work; then based on the processed image, generate the user's face fusion model and hair deformation model, and then obtain the user's image-based avatar expression; in the real-time face animation driving process, according to the tracking obtained Face action and expression parameters, driving avatar expression to generate corresponding face and hair geometry; finally, based on the obtained face and hair geometry, map and collect images, and fuse the mapped images according to the image confidence to generate a real face Animated images. The facial animation result generated by the present invention has the characteristics of high sense of reality, strong expressive force, rich details, high degree of restoration, and the like.

Description

An Image-Based Dynamic Stand-In Construction Method

technical field

The present invention relates to the field of performance-based face animation technology and face expression action redirection technology, and in particular to a face animation method based on image dynamic avatar.

Background technique

Compared with facial muscle models (such as VENKATARAMANA, K., LODHAA, S., AND RAGHAVAN, R. 2005. A kinematic-variational model for animating skin with wrinkles. Computer & Graphics 29, 5 (Oct), 756–770.) and procedural parametric models (such as JIMENEZ, J., ECHEVARRIA, J.I., OAT, C., AND GUTIERREZ, D.2011. GPU Pro 2. AK Peters Ltd., ch. Practical and Realistic Facial Wrinkles Animation.), data The driving method is more common in the creation of dynamic avatars, which benefits from the fact that this method can obtain real facial movements at a very low computational cost. For example, the multidimensional linear face model (VLASIC, D., BRAND, M., PFISTER, H., AND POPOVI′C, J.2005.Facetransfer with multilinear models. ACM Trans.Graph.24,3 (July), 426 –433.) Use a unified model to represent user individual coefficients and expression coefficients in face shape in different dimensions. Multidimensional linear face models are often used to create user-specific face fusion models by fitting input depth maps or video sequences. The dynamic geometric changes in the face fusion model can be linearly modeled for real-time face tracking and animation systems, such as real-time face animation methods based on depth cameras. These linear models are widely used in real-time face tracking and animation methods because of their computational efficiency, but cannot reveal the details of human faces, such as wrinkles.

In some high-end products such as film production, some special hardware devices (such as LightStage system) are used to produce high-realistic dynamic avatars, which contain rich facial details, such as skin folds. But these special hardware devices cannot be used in applications for ordinary users.

There are also techniques for creating animatronics based on a single frame of image. Input a face image of the user and the target avatar, such as (SARAGIH, J., LUCEY, S., AND COHN, J.2011. Real-time avatar animation from asingle image. In AFGR, 213–220.) The method learns the expression mapping function between the input user and the target avatar user in the preprocessing stage. In the real-time operation stage, the user's face motion in the input video is tracked by fitting a deformable face model, and then based on the mapping function learned in the preprocessing stage, the user's motion is transferred to the avatar, and the avatar corresponding action shape. (CAO,C.,WENG,Y.,ZHOU,S.,TONG,Y.,AND ZHOU,K.2014.Facewarehouse: A 3d facial expression database for visualcomputing.IEEE Transactions on Visualization and Computer Graphics 20,3(Mar .), 413–425.) introduce a technique that allows any user to drive a static face image to make various facial animation effects by acting. The technology first uses a multi-dimensional linear face model to fit a face fusion model for the face in the static image, and extracts the texture of the face model from the image. During real-time operation, the tracked head rigid motion parameters and non-rigid expression coefficients are transferred to the face fusion model to generate a face animation from a new perspective. For hair, the technology uses a single-view-based hair modeling technique to create a 3D model of hair strands that are moved and rendered with the head in real-time to produce the final result. Because only one image is used, the obtained stand-in animation effect is not very expressive, especially the details of expression wrinkles cannot be produced. In addition, large head rotation and exaggerated facial expressions are also difficult problems that this series of methods cannot solve.

Recently, some dynamic avatar methods based on multiple images of users have been proposed, which are dedicated to helping ordinary users generate complete and detailed facial animation avatars. (ICHIM,A.E.,BOUAZIZ,S.,AND PAULY,M.2015.Dynamic 3d avatar creation from hand-held video input.ACMTrans.Graph.34,4(July),45:1–45:14.) proposed A method for 3D dynamic stand-ins expressed at two scales based on handheld device video. Through a series of natural expression images input by the user, the method first uses the motion structure algorithm to obtain the point cloud of the face shape, and then fits the user's natural expression model to the point cloud data. Afterwards, a medium-scale user-specific face fusion model is generated by fitting the natural expression model and facial expressions in the captured video. Afterwards, fine-scale facial details, such as facial wrinkles, are obtained through shading algorithms, and expressed using normal maps and ambient light occlusion maps. (GARRIDO, P., ZOLLHOFER, M., CASAS, D., VALGAERTS, L., VARANASI, K., PEREZ, P., AND THEOBALT, C. 2016. Reconstruction of personalized 3dface rigs from monocular video. ACM Trans. Graph.) proposed a fully automatic method to automatically construct 3D face dynamic avatars from monocular video face video data such as traditional movies. The stand-in of this method is based on three scales of geometric hierarchical expression, from basic facial movements described by sparse geometric scales to facial details such as skin wrinkles described by precise geometric scales. The above two methods respectively use multi-layer geometric expressions to represent avatars, and the calculation is relatively complicated. In addition, and more importantly, the motion effect of hair cannot be effectively handled in previous work.

Contents of the invention

The purpose of the present invention is to address the deficiencies in the prior art, and propose a new image-based method for constructing a dynamic substitute for a human face. Driven by a facial expression tracking and animation system, the substitute expression can give realistic, Expressive facial animation results. The stand-in model proposed in the present invention includes the user's complete head expression, such as human face, hair, teeth, eyes, and even the headgear on the hair. Compared with other stand-in expressions, the expression of the present invention is more complete, so the obtained The animation effect is more realistic and convincing. In the present invention, the data required in the preprocessing process can be obtained by ordinary users using ordinary equipment, such as network cameras, home computers, etc., and in the running process, the avatar can be driven by the performance of any user, which is very suitable for various For a variety of virtual reality applications, such as online games, video chat, distance education and so on.

The purpose of the present invention is achieved through the following technical solutions:

An image-based method for constructing dynamic substitutes for human faces, which mainly includes the following steps:

1. Data collection and preprocessing: Use an ordinary webcam to collect a series of facial images of users with set action expressions, and perform preprocessing work such as segmentation and feature point calibration on these images.

2. Image-based avatar construction: Based on the processed image, the user's face fusion model and hair deformation model are generated, and then the user's image-based avatar expression is obtained.

3. Real-time face and hair geometry generation: In the real-time face animation driving process, according to the tracked facial movement and expression parameters, the avatar expression is driven to generate the corresponding face and hair geometry.

4. Real-time facial animation synthesis: Based on the obtained face and hair geometry, map and collect images, and fuse the mapped images according to image confidence to generate real facial animation images.

The beneficial effect of the present invention is that the avatar expression of the present invention is complete. Compared with the previous dynamic avatar expression method, the image-based dynamic avatar proposed by the present invention includes not only human faces, eyes, teeth and other organs, but also hair that is difficult to model. and the headgear on the hair, so the stand-in expression is more complete; in addition, the details of facial expression changes, such as skin folds and wrinkles, do not need to reconstruct the corresponding 3D geometry in this stand-in expression, but are completely implicitly included in the captured image. To sum up, compared with other face substitute animation methods, the present invention proposes that the face animation based on the substitute is more authentic and expressive, and because of low equipment requirements, easy to use, and suitable for any user, etc., Therefore, it can be used in various applications such as online games, video chats, etc., and has broad application prospects.

Description of drawings

Fig. 1 is an example diagram of data acquisition and preprocessing of the present invention, from left to right are: acquisition image, image segmentation level, hair layer transparent channel and marked feature points.

Fig. 2 is an example figure of the face fusion model and the hair deformation model constructed by the present invention, in which an example of three facial expressions of a user is shown, the first line in the figure is the input image, and the second line is the reconstructed face and hair mesh grid model.

Fig. 3 is an example diagram of an animation result of an image-based dynamic avatar in the present invention, in which 7 examples of different users are shown, each line from left to right is respectively: the frontal image in the collected image, the reconstructed face, Hair mesh model, synthetic face animation results of 5 different poses and expressions.

Fig. 4 is a comparison example diagram between the animation result of the image-based dynamic avatar and the real image in the present invention, the first row in the figure is the facial animation generated by the present invention, and the second row is the real image driven to generate the facial animation.

detailed description

In the process of constructing dynamic avatars, firstly, the present invention collects a set of images of specific actions and expressions of the user, and then needs to preprocess these images, including layering and feature point calibration; based on these processed images, establish A 3D face and hair model. Among them, the face model can fit the face and model by matching the feature points in the image; for the hair geometric model, there is no general hair model, because the shape of the hair is very different for different users , it is difficult to use a global model to represent the shape of hair. Therefore, the present invention can only rely on the input image to build a three-dimensional model of the hair. In addition, for hair, especially long hair, under different head poses, due to factors such as gravity and interaction with the body, it will produce non-rigid deformation. In order to deal with these complicated problems, the present invention establishes a deformable hair model, which can approximate the dynamic hair changes in head rotation. In order to build this deformable hair model, the present invention first estimates the depth of the hair region separately in each captured image; then combines all images to perform a joint optimization of the hair depth; finally constructs a global model based on these optimized depths Deformable hair model. In addition, on the other hand of image avatar construction, other parts of the human body, such as eyes, teeth, and body, are modeled separately and represented by a flat panel, respectively.

In the process of real-time animation driving of the system, for each input frame image, the present invention first uses the ready-made real-time face tracking method to obtain the face movement parameters in the image, including head rigid transformation parameters and facial non-rigid expression coefficients, These parameters are then transferred to the dynamic image avatar to generate the 3D mesh corresponding to the avatar. Then, with the help of this 3D mesh, the system maps and fuses the input captured images to generate an animated image of the avatar's face from a new perspective. Among them, in the fusion process, the present invention relies on three-dimensional geometry to calculate the weight of each pixel in the final image of each mapped image in the fusion process, so as to ensure that different regions in the final final result are smooth enough and seamlessly connected. The invention was used on different users with different hairstyles with convincing results.

1. Data collection and preprocessing

1.1 Data collection

For each user, the present invention uses a common webcam to collect 32 images: including 15 different head poses and 17 different expressions. The first set of 15 images records different head poses of the user maintaining a natural expression, and these head poses include different rotations. These head poses are represented by the Euler angles of rotation, which are: the yaw direction is from -60° to 60°, and the sampling interval is 20° (keep the pitch and roll directions at 0°); the pitch direction is from -30° to 30° °, sampling at intervals of 15° but removing 0° (keeping other directions as 0°); sampling in the roll direction is the same as the pitch direction. These head rotations made by the user do not need to meet the set angle exactly, only an approximate angle is required.

Next, two steps of preprocessing are required for the collected images: image segmentation and feature point calibration.

1.2 Image Segmentation

The first step of preprocessing is to segment the acquired images, segmenting each image into different layers: face, hair (including headgear on the hair), glasses, teeth, body and background. The present invention uses a Lazy Snapping tool to segment each image on the basis of a small amount of manual interaction. In the present invention, human interaction only needs to simply draw a few strokes on the image to complete the segmentation of each level. In addition, hair has complexity at the boundary due to its translucent characteristics, so the present invention further executes an image extraction algorithm on the hair layer, further processes the hair layer, and adds a transparent channel (alpha channel) to the hair layer.

1.3 Feature point calibration

In the second step of preprocessing, the present invention needs to semi-automatically mark the feature points S _{i of each collected image I i .} These feature points describe the two-dimensional position of a series of features of the face in the image, these features include the mouth, the outline of the eyes, the outline of the face, etc. The present invention first uses (CAO, C., HOU, Q., AND ZHOU, K.2014. Displaced dynamic expression regression for real-time facial tracking and animation. ACMTrans. Graph. 33, 4 (July), 43: 1–43 The real-time face tracking algorithm described in :10.) automatically marks the feature points in the image, and then manually corrects them using a drag-and-drop tool.

2. Image-based stand-in construction

2.1 Construction of face fusion model

Based on the calibrated collected images {(I _i , S _i )}, the present invention constructs a face fusion model to represent its low-resolution, dynamic face geometry. First, based on the FaceWarehouse face database, the present invention generates an initial face grid for each image fitting The present invention then uses mesh deformation for each Make corrections to obtain F _i ; finally, the present invention calculates the user's facial expression fusion model {B _j } based on {F _i }.

First, for each image (I _i , S _i ), the present invention uses a bilinear model in the face warehouse, a 3rd-order data tensor C, to calculate the user's global individual coefficient w ^id and in each image expression coefficient to fit the initial mesh of the face The fitting method follows (CAO, C., WENG, Y., LIN, S., AND ZHOU, K.2013.3d shape regression for real-time facial animation. ACM Trans.Graph.32, 4 (July), 41: 1–41:10.), first generate the user-specific fusion model based on the face data tensor C, and then generate a grid matching each image based on the fusion model

The initial mesh obtained by fitting the face warehouse It is only a rough approximation. In order to match the grid with the image more accurately, the present invention uses a grid deformation algorithm to further correct the grid. The purpose of grid correction is to make the corresponding vertices of the grid F _i corresponding to each image I _i coincide with the two-dimensional feature points of the image, and the matching energy described in the present invention is:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; \&Pi;</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; f</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}^{{\mathrm{<span\; class="notranslate">\; init</span>}}^{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; the\; s</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Where Π(·) is the projection operator of the camera, which projects a 3D point in the camera coordinate system to obtain the 2D point position in the image, is the initial face grid fitted in the previous step, s _i,k is the two-dimensional position of the kth feature point in S _i , and v _k is the corresponding vertex number of the feature point in the grid.

In order to ensure the smoothness of the grid in the grid deformation, the present invention adds a Laplace regular energy term in the grid deformation:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}\mathrm{<span\; class="notranslate">\; a</span>}\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Delta;v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Delta;v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; \&Delta;v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

where Δ is the discrete Laplacian operator on the grid based on the cosine formula, and δ _k is the initial grid The length of the Laplacian vector at the kth vertex. After grid deformation correction, each image gets an exact matching grid F _i .

Based on these modified grids {F _i }, combined with the expression coefficients of each image obtained in fitting the initial grid Using the example-based facial rigging described in (LI, H., WEISE, T., AND PAULY, M. 2010. Example-based facial rigging. ACMTrans. Graph. 29, 4 (July), 32:1–32:6.) The face skeleton algorithm of the user can get the corrected face fusion model {B _j }.

2.2 Construction of hair deformation model

Modeling hair for an avatar is much more difficult than modeling a human face. First of all, the hairstyles of different people are very different, it is difficult to use a global hair model to represent various hairstyles, and only the hair model can be completely reconstructed from the image. Also, for hair, especially long hair, as the head rotates, there will be non-rigid motion due to gravity, interaction with the body, etc. Therefore, if the hair is regarded as a rigid object and moves with the head, it cannot match the collected image, and the generated facial animation effect will also lack realism. Therefore, in the present invention, a deformed hair model is created for the image stand-in to approximate the dynamic changes of the hair.

The present invention only requires building a low-resolution hair model to help map and fuse images in real-time. In order to construct such a hair model, the present invention first estimates the depth value in the hair region of each image; all depth maps are then jointly optimized, and in order to jointly optimize the long hair with non-rigid motion, the present invention needs to find hair pixels in different images The correspondence between them; finally, based on these depth maps, a topologically consistent hair mesh is generated for each image.

The present invention uses (CHAI, M., WANG, L., WENG, Y., YU, Y., GUO, B., AND ZHOU, K.2012.Single-view hair modeling for portrait manipulation.ACMTrans.Graph.31 , 4 (July), 116:1–116:8.) for single-view hair modeling to estimate the depth of hair in each image. This method combines both boundary and smooth energy terms for optimization. First, the present invention calculates the contour of the hair based on the segmented hair region Ω _h And the depth value of the pixels on the contour can be directly initialized according to the face mesh F _i generated in the previous step. The setting method of the initial depth ^D0 is as follows: for the inner contour pixel point (which coincides with the face on the image), the depth of the pixel point is directly set as the depth on the face grid F _i ; and for the outer contour pixel point , and their depths are set as the average of the depths of the outer contour points of the face mesh F _i . In this way, the boundary energy term can be described as:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; l</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}^{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span><span\; class="notranslate">\; \&dtri;</span>{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

Wherein, D _p is the depth value of each pixel that the present invention needs to solve, is the initial depth, n _p is the normal direction of the pixel, and Represents the image gradient along a hair contour on a 2D image.

For the smooth energy of the second term, it is used to ensure that the obtained hair depth and normal direction are as smooth as possible, which can be described as:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; the\; s</span>}\mathrm{<span\; class="notranslate">\; m</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; \&Omega;</span>}}_{\mathrm{<span\; class="notranslate">\; h</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}\underset{\mathrm{<span\; class="notranslate">\; q</span>}<span\; class="notranslate">\; \&Element;</span>\mathrm{<span\; class="notranslate">\; N</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; p</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; d</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; D.</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; q</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; )</span>$

Among them, p is a pixel in the hair region Ω _h , N(p) is the four-neighborhood pixel set of pixel p, q is one of the neighbor pixels, D _p and D _q are the depths of p and q respectively, n _p and n _q are the normals of p and q, respectively, and ω _d and ω _n are the weights used to control depth and normal smoothing, respectively. By jointly optimizing the energy E _sil +E _sm , the depth D _i of each image I _i can be obtained.

The above-mentioned depth calculation method processes each image separately, without considering the consistency of hair depth in different images. The resulting hair model cannot match every image. Therefore, the present invention needs to consider the hair depth consistency between different images, and use a global optimization method to jointly solve the hair depth in all images. The joint optimization process is iteratively executed in an alternating manner. In each iteration, each depth map is processed in turn; when a depth map D _i is corrected, other depth maps will be fixed, and other depth maps will be fixed. The graph acts as a constraint to optimize D _i . Specifically, the present invention first transforms other depth maps {D _j } _j≠i into the camera coordinate system of D _i , expressed as Then the consistency of depth between different images can be expressed as D _i and The sum of pixel differences between , that is:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; c</span>}\mathrm{<span\; class="notranslate">\; o</span>}\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}\underset{\mathrm{<span\; class="notranslate">\; p</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; d</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Among them, d _{i, p} and are D _i and The depth value at pixel p, D _i and are the depth map of the i-th image and the transformed depth map, respectively. Combining this joint constraint energy with the above-mentioned boundary energy item E _sil and smooth energy item E _sm for joint optimization, the jointly optimized depth map D _i can be obtained.

Next, the present invention will describe how to transform a depth map D _j into the camera coordinate system of D _i to generate For the case of the user with short hair, it can be assumed that the hair moves rigidly with the head, so D _{j can be converted to the camera coordinate system of D i through} rigid transformation through the rigid transformation parameters of the head mesh model fitted in the two images , the specific formula is:

$\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; d</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span><span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Center\; Dot;</span>{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; T</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span>}^{<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; 1</span>}}<span\; class="notranslate">\; \&Center\; Dot;</span>\mathrm{<span\; class="notranslate">\; P</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; )</span>$

Among them, P( ) means to convert a pixel point in the depth map to a three-dimensional point position in the camera coordinate system, and R and T are to convert the grid from the object coordinate system to the camera when fitting the face grid The rotation and translation parameters of the coordinate system, d _{i, p} and are D _i and Depth value at pixel p.

But in the case of long hair, the movement of the hair cannot simply be described as a rigid movement with the head, because the hair will undergo non-rigid changes due to factors such as gravity or contact with the body. Therefore, the present invention needs to calculate the corresponding relationship C _ij between D _i and D _j . In the following content, the present invention introduces in detail how to calculate the corresponding relationship of hair in two images. Based on this correspondence, the present invention performs grid deformation on D _j to obtain The energy description for deformation optimization is as follows:

${\mathrm{<span\; class="notranslate">\; E.</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; =</span>\underset{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; \&Element;</span>{\mathrm{<span\; class="notranslate">\; C</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}\mathrm{<span\; class="notranslate">\; j</span>}}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; d</span>}}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; d</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; c</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}<span\; class="notranslate">\; +</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; l</span>}}\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Delta;v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; -</span>\frac{{\mathrm{<span\; class="notranslate">\; \&delta;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}}{<span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; \&Delta;v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>}{\mathrm{<span\; class="notranslate">\; \&Delta;v</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}$

Among them, (c _i ,c _j ) is a set of corresponding relations in C _ij , and are D _i and The depth value of the corresponding pixel on the vertex v _k is based on the depth map The kth vertex on the constructed grid, in the Laplacian energy term, Δ is the discrete Laplacian operator based on the cosine formula on the grid, and δ _k is the initial grid The length of the Laplacian vector on the kth vertex, ω _l , is used to control the weight of the Laplacian energy, and is set to 10 in the present invention.

According to the previous description, the hair may contain some non-rigid motion when moving with the head rotation. Therefore, before performing joint depth optimization, the present invention needs to calculate the corresponding relationship between the depth map D _i and another depth map D _j . The algorithm for finding correspondence includes three steps: image space correspondence calculation, rough matching construction and correspondence correction.

In the first step, the present invention first utilizes the PatchMatch algorithm (BARNES, C., SHECHTMAN, E., FINKELSTEIN, A., AND GOLDMAN, DB2009.Patchmatch: A randomized correspondence algorithm for structural image editing. ACM Trans.Graph.28 , 3 (July), 24:1–24:11.) Compute the correspondence of hair regions in images _Ii and _Ij . But the corresponding relationship generated in this way is not accurate enough on some pixels, so in the second step, the present invention uses a grid deformation algorithm to calculate a rough matching relationship. The present invention firstly constructs a regular grid P _i in the hair area of image I _i , where each pixel, combined with the depth in D _i , constructs a vertex in the grid. Afterwards, for each vertex in P _i , if the error of all pixels in the 3×3 neighborhood around the vertex in the PatchMatch algorithm calculation is less than the given threshold 0.05, and the offsets of these pixels after PatchMatch calculation are similar, this The invention calculates the average value of pixel offset in these areas, applies it to the original vertex, and moves to a new position; and for the vertex that does not meet the above, the offset obtained by the PatchMatch algorithm is considered unreasonable. Taking finding a reasonable offset in P _i as a position constraint, the present invention uses a Laplacian grid deformation algorithm to deform P _i . The deformed grid P _i ' is rendered into the image space of I _j , so that according to the rendering projections of P _i and P _i ' in the two images, a rough distance between the pixels of I _i and I _j can be obtained match. In the last step, based on such a rough match, for each pixel in the hair region of the I _i image, in the 9×9 neighborhood of the corresponding pixel in I _j , further use the PatchMatch algorithm to find the most matching pixel. If after this step of correction, the error given by PatchMatch is still greater than the threshold, the corresponding relationship of this pixel is calibrated as illegal, and the constraint of this pixel point can be removed in the joint constraint energy.

Based on these obtained hair depth maps, a deformable hair model of the user can be constructed. Specifically, the present invention first converts and warps the depth maps of all images into the same camera coordinate system, that is, the camera space of the first image I ₁ . For any depth map D _j (j≠1), a regular grid is established based on the depth of pixels, and then the regular grid is deformed by using the same algorithm when calculating the corresponding relationship. Taking each vertex of the deformed mesh as a 3D point, the 3D point cloud of the hair model is obtained. In the point cloud, the present invention uses the following rule to remove the abnormal points in the point cloud: if the value of the normal direction of a certain point in the z direction is less than a given threshold of 0.5, then mark the point as an abnormal point. Based on the 3D point cloud, the present invention uses a Poisson plane reconstruction algorithm to generate a hair mesh H _{1 corresponding to the image I 1 .}

Since the rigid rotation and corresponding relationship between other images I _j (j≠1) and I ₁ are known in previous calculations, the present invention finally performs rigid transformation and non-rigid grid deformation on H ₁ , and transforms it into other 14 From the images of different head poses, the grid shape {H _i } _i=1,2,...,15 of the hair in the images of different head poses is obtained. The present invention uses these 15 hair grid sets as the user's deformable hair model, which expresses the non-rigid movement space of the hair under different head poses. It should be noted that the present invention assumes that facial expressions have no effect on hair shape.

2.3 Construction of eyes, teeth and body models

To make the avatar more complete, the invention goes on to describe constructing models of other parts, including the eyes, teeth, and body. Unlike the face and hair, the animations of these parts are relatively simple and do not vary much with different expressions. Therefore, the present invention constructs eye and body models based on the natural facial expression image (i.e., I ₁ ), and constructs a tooth model based on the tooth-showing expression image.

In this invention, two plates are used to express the eyes: one is used to express the iris, and the other is used to express the white of the eye. The present invention first calculates the bounding box of the corresponding eye vertices on the grid based on the fitted face grid in each image, as the rectangular area of the eye. The position and size of the iris is determined by automatically detecting the largest ellipse in the rectangle, and this detected iris is then copied to the tablet model of the iris. As for the white plate, the present invention first copies the eye image into the white plate, and removes the area of the iris. For the missing pixels after removing the iris, the present invention uses the PatchMatch algorithm to synthesize the colors of these missing pixels using the white area of the eye as a source.

The present invention uses two plates representing the upper and lower teeth in the stand-in, respectively. In the tooth-showing expression image, similar to the eye model, the present invention first calculates the bounding box according to the corresponding vertex positions in the face grid to determine the tooth area. The size of the tooth plate is determined by the corresponding vertices in the face mesh. In tooth construction, the present invention also provides a drag tool for manually correcting inaccurate plate positions.

The present invention approximates the model of the upper body with a flat plate whose color is directly filled by the image of the body layer in _I1 . The depth of the body plate is defined as the average depth of the outline points of the face grid.

3. Real-time face and hair geometry generation

During real-time operation, the present invention uses (CAO, C., HOU, Q., AND ZHOU, K.2014.Displaceddynamic expression regression for real-time facial tracking and animation.ACMTrans.Graph.33,4(July), 43:1–43:10.) for a monocular camera-based face tracking system to capture user facial motions and drive image avatars. Specifically, during the running process, for each input video frame, the face tracking system obtains the parameters of the face movement, including the rigid head transformation R, T and the facial expression coefficient e. Based on these parameters, the present invention generates a geometric model of the face and hair of the current frame stand-in.

3.1 Face geometry generation

Based on the pre-calculated stand-in face fusion model {B _j }, the present invention uses the following formula to generate the face geometric grid F of the current frame:

$\mathrm{<span\; class="notranslate">\; f</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; R</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; 0</span>}}<span\; class="notranslate">\; +</span>\underset{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 46</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}{\mathrm{<span\; class="notranslate">\; B</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\mathrm{<span\; class="notranslate">\; T</span>}$

Among them, R and T are the rigid rotation and translation parameters of the current frame face given by the face tracking system, B ₀ is the natural expression grid, e _j is the value of the jth item of the expression coefficient e, and B _j is the fusion model Emoticon grid.

In order to make the head and neck seamlessly connected, the present invention needs to fix the position of the neck position point in the head model, fix it with the body plate, and then update the positions of other points through the grid deformation algorithm based on Laplace . The updated head geometry mesh is denoted as Finally the system needs to pass F and 3D registration between to recalculate The rigid transformation parameter of

3.2 Hair Geometry Generation

Based on the face geometric mesh generated above The present invention proceeds to generate the geometric mesh H of the hair. Similar to the face mesh construction method, the hair geometric mesh is also obtained by interpolating in the hair mesh {H _i } of the pre-acquired image:

$\mathrm{<span\; class="notranslate">\; h</span>}<span\; class="notranslate">\; =</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; \&CenterDot;</span><span\; class="notranslate">\; (</span>\underset{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}{\overset{\mathrm{<span\; class="notranslate">\; 15</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}}{\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}{\mathrm{<span\; class="notranslate">\; h</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; +</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; T</span>}}$

in, and are the face rigid transformation parameters calculated in the previous step, r _i is the weight of the hair grid H _i of the pre-acquired image, and the weight is based on the head rotation of the current frame and the rotation {R _i } of the head in the pre-acquisition image is computed:

${\mathrm{<span\; class="notranslate">\; r</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; =</span>\frac{{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}}{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; 15</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; r</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\stackrel{<span\; class="notranslate">\; ^</span>}{\mathrm{<span\; class="notranslate">\; R</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; R</span>}}_{\mathrm{<span\; class="notranslate">\; j</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}}$

Among them, ω _r is an interpolation parameter, which is set to 10 in the present invention, e is a natural base number, and the head of the current frame is rotated and the head rotation {R _i } in the pre-acquisition image are expressed by quaternions.

4. Real-time facial animation synthesis

4.1 Mapping images

Based on generated current frame geometry mesh and H, combined with the grid face grid {F _i } and hair grid {H _i } in the pre-acquisition image, the present invention maps the pre-acquisition image {I _i } to obtain a series of mapped images, expressed as

The final avatar-driven animation image, each pixel of which is obtained by mapping the weighted average of the corresponding pixels in the image. In order to obtain the weight of each pixel in each mapped image, the present invention first calculates the weight on the corresponding grid vertex, and then interpolates based on the radial basis function to obtain the weight of each pixel.

4.2 Vertex weight calculation

Specifically, for each mapped image The present invention first calculates grid and the weight w(v _k ) of each vertex v _k on H. The core idea is that when the normal direction/expression of the grid in the captured image is similar to that of the current frame, and the grid vertex in the captured image is more towards the normal of the line of sight, then the vertex in the captured image is given a greater weight, expressed mathematically as follows :

$\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}^{\mathrm{<span\; class="notranslate">\; z</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; no</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; \&CenterDot;</span>{\mathrm{<span\; class="notranslate">\; no</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>}<span\; class="notranslate">\; \&Center\; Dot;</span>{\mathrm{<span\; class="notranslate">\; \&alpha;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; e</span>}}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; 1</span>}<span\; class="notranslate">\; -</span>\mathrm{<span\; class="notranslate">\; \&psi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; )</span>}$

where v _k is the mesh A certain vertex in , v _{i, k} is the corresponding vertex in the corresponding grid F _i /H _i in the collected image I _i , n _{i, k} / n _k is the normal direction corresponding to the vertex v _{i, k} / v _k , is the component of the normal n _{i, k} in the z direction, ω _z , ω _n and ω _e are the relative importance of the control components, which are set to 5, 10 and 30 in the present invention. α _i,k is a binary template on a grid manually calibrated for a specific expression, set to 1 in regions related to expression semantics and 0 in the rest. It should be noted that the process of manually calibrating the binary template is a one-time process and has nothing to do with the avatar itself. It only needs to calibrate the corresponding template on a general expression fusion model, and it can be used in all image avatars. And ψ(e _i ,e) is calculated as follows:

$\mathrm{<span\; class="notranslate">\; \&psi;</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\frac{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; \&CenterDot;</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; )</span>}{<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>\mathrm{<span\; class="notranslate">\; e</span>}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>}$

Among them, (·) represents the dot product of two vectors, e _i and e are the expression coefficients of the captured image I _i and the current frame I respectively.

4.3 Pixel weight calculation and image synthesis

After obtaining the weight of each grid vertex in each image, the system can calculate the weight w _i,p on a single pixel p in each image I _i through radial basis function interpolation:

${\mathrm{<span\; class="notranslate">\; w</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; =</span>\underset{\mathrm{<span\; class="notranslate">\; k</span>}}{<span\; class="notranslate">\; \&Sigma;</span>}{\mathrm{<span\; class="notranslate">\; e</span>}}^{<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; \&omega;</span>}}_{\mathrm{<span\; class="notranslate">\; u</span>}}<span\; class="notranslate">\; |</span><span\; class="notranslate">\; |</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; p</span>}}<span\; class="notranslate">\; -</span>{\mathrm{<span\; class="notranslate">\; u</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; |</span>{<span\; class="notranslate">\; |</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}{\mathrm{<span\; class="notranslate">\; \&beta;</span>}}_{\mathrm{<span\; class="notranslate">\; k</span>}}\mathrm{<span\; class="notranslate">\; w</span>}<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; v</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; ,</span>\mathrm{<span\; class="notranslate">\; k</span>}}<span\; class="notranslate">\; )</span>$

Among them, u _p is the two-dimensional coordinates of pixel p, u _i,k is the projected position of vertex v _i,k on the image, w _u is used to control the image influence area size of each vertex v _i,k , β _k is Visibility item, if the vertex v _k in the grid of the current frame is visible in the current image, it is regarded as 1, otherwise it is set to 0. After calculating the single-pixel weights of all images, these weights are standardized so that the sum of the weights of the same pixel of all images is 1, that is, ∑ _i w _i,p =1.

It should be noted that in practice, the present invention does not use all the vertices of the grid, but performs radial basis function interpolation by sampling vertices uniformly on the grid to obtain 1/10 of the original number of vertices. Through experiments, it is found that using these small number of points can not only speed up the calculation speed, but also help the fusion between different images to be smoother, and can get satisfactory results.

In addition to the face and hair, in the process of face animation synthesis, it is also necessary to generate images of other parts of the avatar, including eyes, teeth and body, and combine them with the face and hair naturally to generate the final result .

For eyes, the present invention is first described in (CAO, C., HOU, Q., AND ZHOU, K.2014.Displaced dynamic expression regression for real-time facial tracking and animation.ACMTrans.Graph.33,4(July),43: 1–43:10.), adding two feature points indicating the position of the iris to all the training data, so that the position of the iris can be tracked during real-time operation, and the rotation of the eyeball can be described. In the process of generating the eye image of the stand-in, for the white plate of the eye, the present invention directly makes it perform rigid motion together with the head, that is, directly applies rigid transformation to the white plate of the eye switch to the corresponding position. And for the iris slab, in addition to the rigid transformation In addition, the present invention needs to calculate the translation of the iris according to the positions of iris feature points obtained by tracking, and add the translation to the iris plate to realize the rotation of the eyeball.

For the teeth, the present invention directly connects the flat plate of the upper jaw teeth with the head, and performs rigid movement together with the head, and utilizes rigid transformation parameters Direct alignment is done with a rigid transformation; the lower teeth move with the jaw.

For the body, the present invention makes it translate together with the head, that is, the translation parameter Applies to the body slab, transforms it into position and renders it in image space as a background for other parts (face, hair, teeth and eyes).

In the present invention, only a low-resolution grid is used to express the avatar, and detailed information on the human face, such as skin folds and wrinkles, is implicitly expressed in the image-based avatar expression in the present invention. In addition, more importantly, the image-based stand-in representation in the present invention can effectively deal with hair and headgear on the hair, and these effects cannot be dealt with in the previous work.

Implementation example

The method described in the present invention is realized on an ordinary desktop computer (Intel i7 3.6GHz CPU, 32GB memory, NVidia Gtx 760 graphics card). All captured images were captured with a common webcam, which can provide images with 1280×720 resolution. In order to construct an image animation stand-in, it usually takes 10 minutes for image acquisition, 40 minutes for image preprocessing and 15 minutes for calculating the face fusion model and hair deformation model. When running in real time, the present invention combines (CAO, C., HOU, Q., AND ZHOU, K.2014.Displaced dynamic expression regression for real-time facial tracking and animation.ACM Trans.Graph.33,4(July), 43:1–43:10.) A real-time face tracking system that takes about 30 milliseconds for each frame of the input image to drive the image avatar to generate animation effects. In general, the present invention can achieve a speed of 25 frames per second on a common desktop computer.

It can be seen from the practical results that the present invention can create real dynamic avatars for different users with different hairstyles and headgears. Different from generating geometric layers for facial details such as wrinkles in previous work, thanks to the image-based stand-in expression method, the result of the present invention can naturally contain various wrinkle details of the user. And the present invention can also deal with large-scale rotation and the hair changes brought about by it, which fully embodies the functions of the human face fusion model and the hair deformation model in the present invention.

Claims (5)

1. A dynamic avatar construction method based on images is characterized by comprising the following steps:

(1) data acquisition and preprocessing: the method comprises the steps of collecting facial images of a series of action expressions of a user by using a network camera, and carrying out preprocessing work such as segmentation, feature point calibration and the like on the images.

(2) Virtual avatar construction based on images: and generating a human face fusion model and a hair deformation model of the user based on the processed image, and then obtaining the avatar expression of the user based on the image.

(3) Real-time face and hair geometry generation: in the real-time human face animation driving process, the substitute expression is driven to generate corresponding human face and hair geometry according to the human face action expression parameters obtained by tracking.

(4) And (3) real-time face animation synthesis: and mapping the acquired images based on the obtained face and hair geometry, and fusing the mapped images according to image confidence coefficients to generate a real face animation image.

2. An image-based dynamic avatar construction method as claimed in claim 1, wherein said step 1 mainly comprises the following sub-steps:

(1.1) requiring the user to perform a series of action expressions and acquiring corresponding facial images.

(1.2) segmenting the acquired image by using a Lazy Snaping algorithm to segment different layers: face, hair, teeth, eyes, body, and background.

And (1.3) calibrating two-dimensional feature points of each image by using a two-dimensional feature point regressor, and carrying out simple manual repair on the unsatisfied part in the automatic calibration result by using a dragging tool.

3. An image-based dynamic avatar construction method as claimed in claim 1, wherein said step 2 mainly comprises the following sub-steps:

(2.1) fitting a unified user individual coefficient and rigid transformation parameters and non-rigid expression coefficients of the face in each image based on the existing three-dimensional face expression database to obtain a face grid matched with each image; then, further deformation correction is carried out on the grids, so that the face grids are more consistent with the image; and generating a face fusion model of the user based on the corrected grids.

(2.2) solving the depth value of each pixel in the head area of each image based on the face mesh of each image; combining the face grids and the hair areas of all the images, and jointly optimizing the depth value of the hair area in the image; and finally, based on the depth information of the hair in all the images, a deformation model of the hair of the user is created.

And (2.3) constructing corresponding eye, tooth and body models for the substitute expression based on the face fusion model generated by fitting.

4. An image-based dynamic avatar construction method according to claim 1, wherein said step 3 mainly comprises the following sub-steps:

and (3.1) acquiring the face action parameters of the current frame from the input face video based on the existing real-time face tracking system, wherein the face action parameters comprise rigid rotation and translation parameters and non-rigid expression coefficients, and generating the face geometry of the user by using the parameters.

And (3.2) generating the hair geometry of the user by using the rigid rotation and translation parameters and the non-rigid expression coefficients.

5. An image-based dynamic avatar construction method according to claim 1, wherein said step 4 mainly comprises the following sub-steps:

and (4.1) mapping the collected images to obtain a series of mapped images based on the geometry of the human face and the hair generated by the current frame.

And (4.2) calculating the weight of each vertex in each mapping image according to the difference between the human face and hair models generated by the current frame and the corresponding grids in the acquired image.

(4.3) in each mapping image, based on the weight of each vertex, calculating the corresponding weight on each pixel by using a radial basis function; and fusing the mapping images based on the weights to obtain a final human face animation image.