CN102262788A - Method and device for processing interactive makeup information data of personal three-dimensional (3D) image - Google Patents

Abstract

The invention relates to a method and a device for processing the interactive makeup information data of a personal three-dimensional (3D) image. The device comprises a client, a server and a communication network, wherein the client comprises an internet terminal, a mobile terminal, a retail terminal and a display unit; the server comprises a face characteristic positioning unit, a 3D image rebuilding unit, a makeup processing unit, a 3D face characteristic database and a makeup article database; and the client is connected with the server through the communication network. The working flow of the device comprises the following seven steps that: the client acquires the picture information of a user and transmits the picture information of the user to the server, and the like. Compared with the prior art, the invention has the advantages that: cross-media application service can be realized on touch screens of the internet terminal, the mobile terminal and the retail terminal at the same time; industry can be more intelligently developed; an application service range is wider; individual experience is more real; and the method and the device are free from limitation of time, places and terminal conditions.

Description

Information data disposal route and device are made up in the interactive examination of individual's three-dimensional image

Technical field

The present invention relates to a kind of examination and make up information data disposal route and device, especially relate to the interactive examination of a kind of individual three-dimensional image and make up information data disposal route and device.

Background technology

Along with Internet development and universal, e-commerce website becomes the important channel of merchandise sales day by day, and the mode of shopping online also is that numerous netizens accept day by day.Yet, the commodity closely bound up with personal image, the commodity such as cosmetics, the clothes etc. that need the user to undergo just to make purchase decision still need the support of technology to online experience.Under this demand, in recent years, various online experience type e-commerce websites emerge in an endless stream, and the key of this type of technology success is the fidelity to the user experience process simulation.

Whether the sensation that virtual image is experienced is true, and primary key point is whether image is the image of oneself.Can not satisfy user's demand with model's vivid experience effect, the user need know oneself later effect on probation.

In the prior art, have can catch people's face dynamic image make up simulation technology (such as, can " CN 101371272A " be disclosed with reference to Japanese Patent Application Publication " spy opens 2003-44837 " and Chinese patent application).But these methods are that calculated amount is big to the face tracking features of conducting oneself of each hardwood in the dynamic image, how to use with terminating machine or terminal software form, can't be applied to the network terminal with the B/S form of software, are of limited application.And in a single day leave the camera zone when user takes in real time in terminal applies or posture will lose efficacy in the front inadequately.

Summary of the invention

Purpose of the present invention be exactly provide a kind of in order to overcome the defective that above-mentioned prior art exists and promote industry development, can be applicable to the network terminal, range of application is wider, it is true more to experience, be not subjected to the time simultaneously, information data disposal route and device are made up in the interactive examination of individual three-dimensional image of place and terminal condition restriction.

Purpose of the present invention can be achieved through the following technical solutions:

The information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image, it is characterized in that, may further comprise the steps:

1) client is obtained user picture information, and it is transferred to server;

2) the face characteristic positioning unit in the server detects people's face position to user picture information, and judges whether to detect successfully, if yes, and execution in step 3), if do not return step 1);

3) the face characteristic positioning unit in the server adopts shape-variable model (ASM) detection algorithm to locate and extract the face characteristic dot information;

4) image of the 3D in server reconstruction unit adopts based on the three-dimensional face property data base and calculates the three-dimensional feature point, and adopts and set up three-dimensional face model based on the surf deform algorithm of master pattern;

5) server with photo as texture in three-dimensional face model;

6) client is played up the 3D model by flash, and drives 3D modeling true man action by the model deformation method, exports true man by the display unit of client and simulates vivid information;

7) user selects toiletware in client, cosmetic treatment unit in the server is according to the toiletware information of selecting and the positional information of human face characteristic point, toiletware accurately is added on the model texture picture, upgrade the 3D image and play up, and by the image of the 3D after display unit display process information.

Shape-variable model (ASM) detection algorithm in the described step 3) is as follows:

31) model initial alignment:

For facial image, adopt people's face detection algorithm to determine whether to have in the photo position of people's face and people's face, according to initial position that obtains and the translation by angle θ rotation, yardstick s convergent-divergent and position t the average shape model is put in the photo, obtains initial shape

$X=M(s,\mathrm{\θ})\stackrel{\‾}{X}+t$

${{\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="HSA00000133534500021.png"\; state="\{\{state\}\}">X</figure-callout>}}_2}^{\&prime;}=M(\mathrm{\&theta;},s)\left[X_2\right]+t={\left(\begin{array}{cccc}{x}_{20}& -y_{20}& 1& 0\\ y_{20}& x_{20}& 0& 1\\ x_{2(n-1)}& -y_{2(n-1)}& 1& 0\\ y_{2(n-1)}& x_{2(n-1)}& 0& 1\end{array}\right)}_{2n\&times;4}\left[\begin{array}{c}s\cos \mathrm{\&theta;}\\ s\sin \mathrm{\&theta;}\\ t_x\\ t_y\end{array}\right]$

32) search on the unique point normal direction:

The local gray level model that utilizes training to obtain, to each unique point it is searched within the specific limits along normal direction, its normalization gray scale derivative vector of each some calculating in this scope and training obtain the mahalanobis distance of the average normalization gray scale derivative vector of this point, therefrom the candidate point of selected distance minimum is as optimal match point, each unique point of model is all carried out the search of optimal match point, obtained a new shape vector X ';

33) with shape to carrying out approximate expression, because the shape that obtains after the search of each unique point can't direct representation under new orthogonal basis, can only obtain the approximate representation on the least error meaning, X is obtained four parameter (1+ds of affined transformation to X ' do alignment computing, d θ, dtx dty), further obtains the changing value of form parameter

$\mathrm{db}=P^{T}P{P}^T\left(M({\left(s(1+\mathrm{ds})\right)}^{-1},-(\mathrm{\&theta;}+\mathrm{d\&theta;}))\right[M(s,\mathrm{\&theta;})\left[\stackrel{\&OverBar;}{X}\right]+\mathrm{dX}-\mathrm{dt}]-\stackrel{\&OverBar;}{X})$

34) the rational constraint of shape, the variation range of form parameter bi should

Just can have reasonably character in the scope, therefore will judge, it be transformed to for bi not in this scope to b+db

In the scope, utilize the form parameter b+db that finally obtains to calculate the shape of reconstruct;

35) repeat 32)～34) step circulate, when the Euclidean distance of the shape vector of twice adjacent circulation during, think its convergence smaller or equal to preset threshold, iteration finishes.

Human face characteristic point information in the described step 3) comprises facial contour dot information, eye contour dot information, nose contour point information, face contour point information, eyebrow contour point information.

The surf deform algorithm based on master pattern in the described step 4) is as follows:

Through the processing of step 3), obtain general faceform's three-dimensional feature point S _MfThree-dimensional feature point S with photo people face _Obj, by their corresponding relation with general faceform S _{Mod el}Elastic deformation is specific faceform, selects thin-plate spline interpolation algorithm (TPS), and this algorithm is a kind of radially basic interpolating function,

The basis function of TPS is U (r)=r ²Logr ², r=‖ P wherein _i-(x, y, z) ‖, (x, y z) are the three-dimensional coordinate of interpolation point, and Pi is a unique point, according to S _MfWith S _ObjCorresponding relation, calculate the radially coefficient of basic interpolating function, obtain the TPS interpolating function and be:

$f(x,y,z)=a_1+a_{x}x+a_{y}y+a_{z}z+\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{w}_{i}U(P_i-(x,y,z))$

(x, y is z) with general faceform S by f _{Mod el}Become specific people's face.

Model deformation method in the described step 6) is specific as follows:

Set up 44 elemental motion unit based on facial movement coded system FACS, each one or several human face characteristic point of elemental motion unit controls is in three-dimensional displacement, with different elemental motion unit combination, produce various expressions, and use TPS that the three-dimensional feature point is carried out the interpolation distortion, realize expression shape change.

Toiletware in the described step 7) comprises glasses, hair style, jewellery, cosmetics, the picture of described glasses, hair style, jewellery loads the back and directly is added on the model texture picture according to the position of human face characteristic point, and described cosmetics are added on the model texture picture according to the position of human face characteristic point after according to range of influence and the synthetic color diagram layer of color.

The information data treating apparatus is made up in the interactive examination of a kind of individual three-dimensional image, it is characterized in that, comprise client, server, communication network, described client comprises display unit, described server comprises the face characteristic positioning unit, 3D image reconstruction unit, the cosmetic treatment unit, the three-dimensional face property data base, the toiletware database, described client is connected with server by communication network, described face characteristic positioning unit, 3D image reconstruction unit, the cosmetic treatment unit connects successively, described 3D image reconstruction unit is connected with the three-dimensional face property data base, and described cosmetic treatment unit is connected with the toiletware database.

Described communication network is internet, communication bus or wireless network.

Compared with prior art, the present invention has the following advantages:

1, promote industry development: a difficult problem is experienced in the shopping that utilizes people's face three-dimensional reconstruction technology and client 3D display technique well to solve user's online cosmetics, allow the user can before buying commodity, experience result behind the cosmetic sampler really, promoted the ecommerce process of cosmetics commodity greatly;

2, applied range: can be applicable in internet, desktop software, terminal all-in-one, the portable terminal;

3, experience is truer: drive 3D modeling true man action by the model deformation method, export true man by the display unit of client and simulate image, make user experience truer.

4, have more the advantage that is not subjected to time, place and terminal condition restriction, the mode that satisfying the modern can be at any time and any place, like according to the individual is used the hope of service.

Description of drawings

Fig. 1 sets up process flow diagram for three-dimensional face model of the present invention;

Fig. 2 is that process flow diagram flow chart is made up in examination of the present invention;

Fig. 3 is a hardware configuration synoptic diagram of the present invention.

Embodiment

The present invention is described in detail below in conjunction with the drawings and specific embodiments.

Embodiment

As Fig. 1, Fig. 2, shown in Figure 3, the information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image, may further comprise the steps:

1) client 1 is obtained user picture, and it is transferred to server 3;

2) 21 pairs of user picture of the face characteristic positioning unit in the server 2 detect people's face position, and judge whether to detect successfully, if yes, and execution in step 3), if do not return step 1);

3) the face characteristic positioning unit in the server 2 21 adopts shape-variable model (ASM) detection algorithm to locate and extract human face characteristic point;

4) image of the 3D in the server 2 reconstruction unit 22 adopts based on three-dimensional face property data base 24 and calculates the three-dimensional feature point, and adopts and set up three-dimensional face model based on the surf deform algorithm of master pattern;

5) server 2 with photo as texture in three-dimensional face model;

6) client 1 is played up the 3D model by flash, and drives 3D modeling true man action by the model deformation method, exports true man by the display unit 11 of client 1 and simulates image;

7) user selects toiletware in client 1, cosmetic treatment unit 23 in the server 2 is according to the toiletware of selecting and the position of human face characteristic point, toiletware accurately is added on the model texture picture, upgrades the 3D image and play up, and by the 3D image after display unit 1 display process.

Shape-variable model (ASM) detection algorithm in the described step 3) is as follows:

31) model initial alignment:

For facial image, adopt people's face detection algorithm to determine whether to have in the photo position of people's face and people's face, according to initial position that obtains and the translation by angle θ rotation, yardstick s convergent-divergent and position t the average shape model is put in the photo, obtains initial shape

$X=M(s,\mathrm{\&theta;})\stackrel{\&OverBar;}{X}+t$

${{\mathrm{<figure-callout\; id="2"\; label="X"\; filenames="HSA00000133534500021.png"\; state="\{\{state\}\}">X</figure-callout>}}_2}^{\&prime;}=M(\mathrm{\&theta;},s)\left[X_2\right]+t={\left(\begin{array}{cccc}{x}_{20}& -y_{20}& 1& 0\\ y_{20}& x_{20}& 0& 1\\ x_{2(n-1)}& -y_{2(n-1)}& 1& 0\\ y_{2(n-1)}& x_{2(n-1)}& 0& 1\end{array}\right)}_{2n\&times;4}\left[\begin{array}{c}s\cos \mathrm{\&theta;}\\ s\sin \mathrm{\&theta;}\\ t_x\\ t_y\end{array}\right]$

32) search on the unique point normal direction:

The local gray level model that utilizes training to obtain, to each unique point it is searched within the specific limits along normal direction, its normalization gray scale derivative vector of each some calculating in this scope and training obtain the mahalanobis distance of the average normalization gray scale derivative vector of this point, therefrom the candidate point of selected distance minimum is as optimal match point, each unique point of model is all carried out the search of optimal match point, obtained a new shape vector X ';

33) with shape to carrying out approximate expression, because the shape that obtains after the search of each unique point can't direct representation under new orthogonal basis, can only obtain the approximate representation on the least error meaning, X is obtained four parameter (1+ds of affined transformation to X ' do alignment computing, d θ, dtx dty), further obtains the changing value of form parameter

$\mathrm{db}=P^{T}P{P}^T\left(M({\left(s(1+\mathrm{ds})\right)}^{-1},-(\mathrm{\&theta;}+\mathrm{d\&theta;}))\right[M(s,\mathrm{\&theta;})\left[\stackrel{\&OverBar;}{X}\right]+\mathrm{dX}-\mathrm{dt}]-\stackrel{\&OverBar;}{X})$

34) the rational constraint of shape, the variation range of form parameter bi should

Just can have reasonably character in the scope, therefore will judge, it be transformed to for bi not in this scope to b+db

In the scope, utilize the form parameter b+db that finally obtains to calculate the shape of reconstruct;

35) repeat 32)～34) step circulate, when the Euclidean distance of the shape vector of twice adjacent circulation during, think its convergence smaller or equal to preset threshold, iteration finishes.

Human face characteristic point information in the described step 3) comprises facial contour dot information, eye contour dot information, nose contour point information, face contour point information, eyebrow contour point information.

The surf deform algorithm based on master pattern in the described step 4) is as follows:

Through the processing of step 3), obtain general faceform's three-dimensional feature point S _MfThree-dimensional feature point S with photo people face _Obj, by their corresponding relation with general faceform S _{Mod el}Elastic deformation is specific faceform, selects thin-plate spline interpolation algorithm (TPS), and this algorithm is a kind of radially basic interpolating function,

The basis function of TPS is U (r)=r ²Logr ², r=‖ P wherein _i-(x, y, z) ‖, (x, y z) are the three-dimensional coordinate of interpolation point, and Pi is a unique point, according to S _MfWith S _ObjCorresponding relation, calculate the radially coefficient of basic interpolating function, obtain the TPS interpolating function and be:

$f(x,y,z)=a_1+a_{x}x+a_{y}y+a_{z}z+\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{w}_{i}U(P_i-(x,y,z))$

(x, y is z) with general faceform S by f _ModelBecome specific people's face.

Model deformation method in the described step 6) is specific as follows:

Set up 44 elemental motion unit based on facial movement coded system FACS, each one or several human face characteristic point of elemental motion unit controls is in three-dimensional displacement, with different elemental motion unit combination, produce various expressions, and use TPS that the three-dimensional feature point is carried out the interpolation distortion, realize expression shape change.

Toiletware in the described step 7) comprises glasses, hair style, jewellery, cosmetics, the picture of described glasses, hair style, jewellery loads the back and directly is added on the model texture picture according to the position of human face characteristic point, and described cosmetics are added on the model texture picture according to the position of human face characteristic point after according to range of influence and the synthetic color diagram layer of color.

Hardware device of the present invention comprises client 1, server 2, communication network 3, described client 1 comprises display unit 11, described server 2 comprises face characteristic positioning unit 21,3D image reconstruction unit 22, cosmetic treatment unit 23, three-dimensional face property data base 24, toiletware database 25, described client 1 is connected with server 2 by communication network 3, described face characteristic positioning unit 21,3D image reconstruction unit 22, cosmetic treatment unit 23 connects successively, described 3D image reconstruction unit 22 is connected with three-dimensional face property data base 24, and described cosmetic treatment unit 23 is connected with toiletware database 25.Described communication network 3 is internet, communication bus or wireless network.

If communication network 3 is Internet, can use on the internet, if communication network 3 is a communication bus, can be applied on the terminal all-in-one, if communication network 3 is a wireless network, can be applicable in the portable terminal.This product will be put on display on 2010 Shanghai World's Fair.

The 3D simulation image (action that is virtually reality like reality) that the present invention makes the user can use the photo of oneself to generate is experienced the effect after on probation such as extensive stock such as wig, eye shadow, lipstick, glasses, can be applied in based on this in the network electronic commerce, as the virtual 3D of the wig in the wig e-commerce website experience, cosmetics virtual image in the cosmetics e-commerce website is experienced and the virtual experience of e-commerce websites such as glasses, jewellery.

According to the position of user picture detected characteristics point, comprise the profile and the colour of skin information of face, eyes, nose, face, eyebrow, and serve as according to the trial effect of dissimilar cosmetics is made color change with these information, realize trial effect.

By a user picture,, use three-dimensional rebuilding method to rebuild with the three-dimensional model of this photo as texture by detected characteristics point position.And the three-dimensional motion when being implemented in model 3D and playing up by the model deformation information that prestores and algorithm, as face action such as blinking, smile, speak, and the wobbling action that is virtually reality like reality of head.

The 3D that realizes three-dimensional image at networking client plays up, and realizes that by changing model texture photo the enforcement of 3D rendering content changes, and makes experience truer in render process.

Claims (8)

1. the information data disposal route is made up in the interactive examination of individual three-dimensional image, it is characterized in that, may further comprise the steps:

1) client is obtained user picture information, and it is transferred to server;

2) the face characteristic positioning unit in the server detects people's face position to user picture information, and judges whether to detect successfully, if yes, and execution in step 3), if do not return step 1);

3) the face characteristic positioning unit in the server adopts shape-variable model (ASM) detection algorithm to locate and extract the face characteristic dot information;

4) image of the 3D in server reconstruction unit adopts based on the three-dimensional face property data base and calculates the three-dimensional feature point, and adopts and set up three-dimensional face model based on the surf deform algorithm of master pattern;

5) server with photo as texture in three-dimensional face model;

6) client is played up the 3D model by flash, and drives 3D modeling true man action by the model deformation method, exports true man by the display unit of client and simulates vivid information;

7) user selects toiletware in client, cosmetic treatment unit in the server is according to the toiletware information of selecting and the positional information of human face characteristic point, toiletware accurately is added on the model texture picture, upgrade the 3D image and play up, and by the image of the 3D after display unit display process information.

2. the information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image according to claim 1, it is characterized in that shape-variable model (ASM) detection algorithm in the described step 3) is as follows:

31) model initial alignment:

For facial image, adopt people's face detection algorithm to determine whether to have in the photo position of people's face and people's face, according to initial position that obtains and the translation by angle θ rotation, yardstick s convergent-divergent and position t the average shape model is put in the photo, obtains initial shape

$X=M(s,\mathrm{\&theta;})\stackrel{\&OverBar;}{X}+t$

${X_2}^{\&prime;}=M(\mathrm{\&theta;},s)\left[X_2\right]+t={\left(\begin{array}{cccc}{x}_{20}& -y_{20}& 1& 0\\ y_{20}& x_{20}& 0& 1\\ x_{2(n-1)}& -y_{2(n-1)}& 1& 0\\ y_{2(n-1)}& x_{2(n-1)}& 0& 1\end{array}\right)}_{2n\&times;4}\left[\begin{array}{c}s\cos \mathrm{\&theta;}\\ s\sin \mathrm{\&theta;}\\ t_x\\ t_y\end{array}\right]$

32) search on the unique point normal direction:

The local gray level model that utilizes training to obtain, to each unique point it is searched within the specific limits along normal direction, its normalization gray scale derivative vector of each some calculating in this scope and training obtain the mahalanobis distance of the average normalization gray scale derivative vector of this point, therefrom the candidate point of selected distance minimum is as optimal match point, each unique point of model is all carried out the search of optimal match point, obtained a new shape vector X ';

33) with shape to carrying out approximate expression, because the shape that obtains after the search of each unique point can't direct representation under new orthogonal basis, can only obtain the approximate representation on the least error meaning, X is obtained four parameter (1+ds of affined transformation to X ' do alignment computing, d θ, dtx dty), further obtains the changing value of form parameter

$\mathrm{db}=P^{T}P{P}^T(M{\left((s(1+\mathrm{ds})\right)}^{-1},-(\mathrm{\&theta;}+\mathrm{d\&theta;}))\left[M(s,\mathrm{\&theta;})\right[\stackrel{\&OverBar;}{X}]+\mathrm{dX}-\mathrm{dt}]-\stackrel{\&OverBar;}{X})$

34) the rational constraint of shape, the variation range of form parameter bi should

Just can have reasonably character in the scope, therefore will judge, it be transformed to for bi not in this scope to b+db

In the scope, utilize the form parameter b+db that finally obtains to calculate the shape of reconstruct;

35) repeat 32)～34) step circulate, when the Euclidean distance of the shape vector of twice adjacent circulation during, think its convergence smaller or equal to preset threshold, iteration finishes.

3. the information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image according to claim 1, it is characterized in that the human face characteristic point information in the described step 3) comprises facial contour dot information, eye contour dot information, nose contour point information, face contour point information, eyebrow contour point information.

4. the information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image according to claim 1, it is characterized in that the surf deform algorithm based on master pattern in the described step 4) is as follows:

Through the processing of step 3), obtain general faceform's three-dimensional feature point S _MfThree-dimensional feature point S with photo people face _Obj, by their corresponding relation with general faceform S _ModelElastic deformation is specific faceform, selects thin-plate spline interpolation algorithm (TPS), and this algorithm is a kind of radially basic interpolating function,

The basis function of TPS is U (r)=r ²Logr ², r=‖ P wherein _i-(x, y, z) ‖, (x, y z) are the three-dimensional coordinate of interpolation point, and Pi is a unique point, according to S _MfWith S _ObjCorresponding relation, calculate the radially coefficient of basic interpolating function, obtain the TPS interpolating function and be:

$f(x,y,z)=a_1+a_{x}x+a_{y}y+a_{z}z+\underset{i=1}{\overset{n}{\mathrm{\&Sigma;}}}{w}_{i}U(P_i-(x,y,z))$

(x, y is z) with general faceform S by f _ModelBecome specific people's face.

5. the information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image according to claim 1, it is characterized in that the model deformation method in the described step 6) is specific as follows:

Set up 44 elemental motion unit based on facial movement coded system FACS, each one or several human face characteristic point of elemental motion unit controls is in three-dimensional displacement, with different elemental motion unit combination, produce various expressions, and use TPS that the three-dimensional feature point is carried out the interpolation distortion, realize expression shape change.

6. the information data disposal route is made up in the interactive examination of a kind of individual three-dimensional image according to claim 1, it is characterized in that, toiletware in the described step 7) comprises glasses, hair style, jewellery, cosmetics, the picture of described glasses, hair style, jewellery loads the back and directly is added on the model texture picture according to the position of human face characteristic point, and described cosmetics are added on the model texture picture according to the position of human face characteristic point after according to range of influence and the synthetic color diagram layer of color.

7. the information data treating apparatus is made up in the interactive examination of individual three-dimensional image, it is characterized in that, comprise client, server, communication network, described client comprises internet terminal, portable terminal, retail terminal, and be equipped with display unit, described server comprises the face characteristic positioning unit, 3D image reconstruction unit, the cosmetic treatment unit, the three-dimensional face property data base, the toiletware database, described client is connected with server by communication network, described face characteristic positioning unit, 3D image reconstruction unit, the cosmetic treatment unit connects successively, described 3D image reconstruction unit is connected with the three-dimensional face property data base, and described cosmetic treatment unit is connected with the toiletware database.

8. the information data treating apparatus is made up in the interactive examination of a kind of individual three-dimensional image according to claim 7, it is characterized in that described communication network is internet, communication bus or wireless network.

Images