CN104616005A - Domain-self-adaptive facial expression analysis method - Google Patents

Abstract

The invention relates to a domain-self-adaptive facial expression analysis method and belongs to the field of computer vision and emotion computing research. The method aims to solve the problem that the prediction precision is hindered by training and test data domain differences in automatic expression analysis, and is more aligned with the actual needs. The invention provides the domain-adaptive expression analysis method based on an object domain. The method comprises the following steps: defining a data domain for each tested object; defining the distance between object domains in a way of establishing an auxiliary prediction problem; selecting a group of objects similar to the data character of the tested object from a source data set to form a training set; on the training set, directly using part of tested object data in model training in a way of weighted cooperative training, and thus enabling a prediction model to be closer to the tested object domain. The method has the advantages that the isolation problem of training and testing data is solved, and the prediction model is adaptive to the testing data domain; the method has robustness for the domain differences and is wide in range of application.

Description

A kind of facial Expression Analysis method of domain-adaptive

Technical field

The invention belongs to computer vision and affection computation research field, specifically a kind of facial Expression Analysis method automatically.

Background technology

Automatic facial Expression Analysis is computer vision field long-standing studying a question.The target of the automatic Expression analysis of main flow is from image or video, extract a series of Facial action unit with semantic class information.Definition in usual employing FACS handbook.FACS (Facial Action Coding System) is the segmentation labeling system of a set of research facial expression that behaviorist proposes.Face action is decomposed into a series of expression motor unit (Action Unit, AU) by FACS system.Each motor unit is relevant to one or more facial muscle movements.

The Expression analysis research of Most current all supposes that training (source) data and test (target) data are from same Data distribution8.Usually following steps are adopted: first on the training be collected in advance (source) data set, train forecast model, then forecast model is applied to test (target) data set when analyzing.Video data for training requires band espressiove label, and label of expressing one's feelings obtained by manually marked by the mark person of professional training usually.

The video data that can be used at present training collects substantially under laboratory controlled imaging circumstances.Practical application then requires to test on the image under true environment.Appearance in true environment facial image, the change of the factor such as attitude and illumination often far exceeds the scope of training data.Now, data fields (domain) difference between training and testing data just can not be left in the basket.The forecast model that this species diversity causes traditional algorithm to be trained and draws can not obtain the performance consistent with in training set on test video.

For this reason, the present invention proposes a kind of Expression analysis method with domain-adaptive ability.The method supposes training data from test data from different data fields, and designs respective algorithms step, makes forecast model adaptive testing data fields.

Summary of the invention

The object of the invention is to solve training and test data field difference in Expression analysis and hinder the problem of precision of prediction, thus make Expression analysis system be more suitable for actual application environment.The present invention proposes a kind of Expression analysis method with domain-adaptive ability.Be conceived to individual mobile terminal applied environment, we define a data fields to each study subject.Object data field is made up of the video data of all collections from this object.First this method carrys out the distance between defining objects field by a kind of mode building auxiliary forecasting problem.This distance definition reflects the correlativity in object domain between geometric properties and appearance features.Based on the spacing of object domain, we choose the object close with tested object data character in source data set, composition training set.On training set, we adopt the mode of weighting coorinated training, and partial test object video data are directly used in model training, make forecast model further close to test data field.

Compared with prior art, advantage of the present invention is: solve training data and the isolated problem of test data, makes expression action prediction model be adapted to test data.Propose Expression analysis algorithm, to test and training field difference, there is better robustness, expand the practical ranges of Expression analysis technology.

Accompanying drawing explanation

Fig. 1: facial feature points detection result is illustrated

Fig. 2: reference man's face shape

Fig. 3: image alignment result is illustrated

Fig. 4: algorithm overall flow figure

Embodiment

The present invention is a kind of automatic facial Expression Analysis method with domain-adaptive ability.The face Facial action unit (Action Unit, AU) that the present invention defines in FACS is as the target of Expression analysis.AU is one group of motor unit being defined on facial muscle movements.Such as AU 12 represents the corners of the mouth and raises up, and is semantically substantially equal to " laughing at " this action.On the basis making full use of between two class facial image features relevant complementary character again, put forward the methods of the present invention can full automatic analytical test object video, provides the label whether specific AU occurs in each frame.

Utilize prior art, we can detect human face characteristic point.We select SDM (SupervisedDescent Machine) technology each frame in face video to detect human face characteristic point.Fig. 1 is shown in the testing result signal of human face characteristic point.

Except the expression information that we are concerned about, also contain head pose in face video, focal length, shooting angle and distance etc. influence factor.In order to eliminate the impact of these disturbing factors on Expression analysis, human face expression video aligns to reference man's face shape by we.Reference man's face shape of our chosen in advance fixed size (200X 200 pixel), as shown in Figure 2.To each facial image, we utilize ProcrustesAnalaysis to calculate optimum yardstick, and (in plane) rotates and translation transformation, make facial image and reference figuration closest.Procrustes Analaysis is a kind of research means in shape analysis field, can be used for shape alignment.We utilize the optimal transformation parameter calculated, and carry out texture to facial image.This process is called as image alignment, which ensures the face images participating in training and testing and compares on unified yardstick, and not by the impact that head pose (in plane) deflects.Facial image after alignment and unique point are as shown in Figure 3.

According in FACS handbook to the description of each AU, we define geometric properties and appearance features two category feature.Geometric properties refers to the dimensional measurements such as a series of angle and distances calculated by human face characteristic point, and comprise the folding angle of the corners of the mouth, nose is to the height etc. of the Distance geometry eyes at canthus.Geometric properties is designated as f by us ₁.Meanwhile, appearance features then depicts texture in facial image, edge, the information such as direction of a curve.Appearance features is designated as f by us ₂.The present invention selects the descriptor of SIFT (Scale Invariant Feature Transform) as appearance features.The extracting position of usual SIFT descriptor and yardstick detect son by SIFT and obtain, and our fixed extraction position of selecting one group to be determined by human face characteristic point in the present invention.The extraction yardstick of we fixing SIFT simultaneously.In conjunction with before the image alignment step that describes, we ensure that the SIFT descriptor in training and testing, face images extracted can compare on the unified platform.

Training objects set selected part based on auxiliary forecasting problem:

The input data of Expression analysis method of the present invention comprise the set of source data D with expression label _s, and tested object data U={X ^t.The target of Expression analysis method is the expression label of prediction tested object x in L ^sfor the human face expression video collected in advance, and Y ^sit is then corresponding expression label.Y ^susually by manually provided by the mark person of professional training, the information whether specific AU occurs at each frame is comprised.X in U ^tfor tested object face face video.Here we suppose only there is a tested object in U, and we think that U forms the data fields of this object.

In the present invention, we propose a kind of strategy choosing training objects in set of source data L newly.From use in prior art all training datas or stochastic sampling different, we choose n the object close with U data characteristics in L, composition set L ', for training forecast model.By this selection, training dataset is changed into the L ' more adapting to U, to reach the object of domain-adaptive from the L had nothing to do with U by us.

We utilize geometric properties f ₁with appearance features f ₂between relevance select the object close with U.Design this strategy mainly based on following 2 considerations.

A. geometric properties and expression motor unit tight association that we will predict, simultaneous altitude is abstract, is relatively not easy the impact being subject to features of the object.In the training objects selection strategy proposed, the effect that geometric properties plays " bridge ".

B. by analyzing the relevance pattern of each object geometric properties and appearance features, we can find the object close with U character.Learn at the data fields close to U.

Particularly, first we utilize geometric properties f ₁set up one group and assist forecasting problem.Suppose f ₁comprise l ₁dimension.To f ₁in each dimension f ₁(i), we set up and solve following auxiliary problem (AP):

A. assisted tag is set up.To each sample, set up assisted tag wherein f ₁(i) mean value in L, and symbolic operation is got in sign () expression.

B. to whole samples of each object j, f is utilized ₂training linear SVM model, predicts assisted tag Y ₁(i).The weight vectors of training gained Linear SVM model is designated as w _j(i).

To each object j, by solving above-mentioned l ₁individual auxiliary problem (AP), we can obtain one group and assist forecast model weight vectors { w _j(i) } _i, wherein each w _ji () is for object j is by f ₁linear SVM weight vectors in i auxiliary forecasting problem that () sets up.By this group weight vectors splicing, we set up an auxiliary forecast model long vector W to each object _j=[w _j(1) ^t, w _j(2) ^t..., w _j(l ₁) ^t] ^t, wherein l ₁f ₁dimension.W _jdimension be l ₁× l ₂, wherein l ₂f ₂dimension.Auxiliary forecast model long vector W _jrepresentative be relevance in corresponding objects data between appearance features and geometric properties, reacted the data characteristic of object.Therefore, object-based auxiliary forecast model matrix, the distance definition between any two object j and k can be corresponding W by we _jand W _kdistance between two matrixes.Particularly, the distance d between object j and k _jkbe defined as: d _jk=Σ | W _j-W _k|-Σ _sign(W _jο W _k), wherein Σ represents all elements in superimposed vector, and ο represents hadamard product of matrices, and namely corresponding element is multiplied.In this new distance definition, Σ | W _j-W _k| the close situation of each corresponding element value is being investigated in the effect of item, and Σ sign (W _jο W _k) Xiang Ze investigates the consistent situation of corresponding element symbol.Consider W _jphysical significance be the weight vectors of Linear SVM, in this application of the close object of searching data characteristic, symbol unanimously even than value close to even more important.Based on the distance between object, we choose n and the immediate object of U in L, composition training set L '.

Coorinated training part based on sample weighting:

Choosing on the training set L ' and tested object U obtained, we extract geometric properties and appearance features, utilize the method for weighting coorinated training, train corresponding geometric properties and appearance features fallout predictor respectively.In the present invention, we propose a set of strategy heavy to coorinated training sample weighting.By weighting, forecast model, by more effective adaptive testing data fields U, has better robustness simultaneously.

Fallout predictor training method selects linear SVM (linear Support Vector Machine, SVM).In order to make the output of different SVM model to compare together, when training, we adopt the method for PlattScaling to carry out the output for the treatment of S VM.Through this step, the output of each SVM model has unified Probabilistic.We use it for the degree of confidence of evaluation and foreca when testing.

Concrete, first we utilize Linear SVM to train geometric properties fallout predictor h on L ' ₁with appearance features fallout predictor h ₂.For convenience of discussing, we are by h ₁and h ₂training set be called L ₁' and L ₂'.On input test object data U, we adopt the method consistent with on L ', detect human face characteristic point, extract geometric properties f ₁with appearance features f ₂.Then, we circulate execution following steps:

A. by h ₁be applied to tested object data U, obtain prediction label by h ₂be applied to U, obtain prediction label

B., in U, h is chosen ₁predict with high confidence, and not at L ₂' middle sample.With as pseudo-label, add h ₂training set L ₂'.Similar, choose h ₂high confidence level new samples, adds L ₁'.

C. for L ₁' and L ₂' in each training sample x, give training weight Q _x.

D. based on training weight Q _x, re-training h ₁and h ₂.

The end condition of this cyclic process does not have new high confidence level sample to add L in U ₁' or L ₂'.So-called high confidence level refers to h ₁or h ₂degree of confidence during prediction is higher than the threshold value of setting in advance.

Training sample weight Q _xaccording to from the positive and negative sample proportion of sample in U and h ₁with h ₂between difference calculate.Particularly:

$Q_x=\left\{\begin{array}{cc}1+a{R}_{\mathrm{skew}}\left(x\right)+b|h_1\left(x\right)-h_2\left(x\right)|& \mathrm{ifx}\∈U\\ 1& \mathrm{ifx}\∈L^{\′}\end{array}\right.$

Wherein a and b be preset be greater than zero constant, by choosing obtain by cross validation (crossvalidation) in a database.And R _skewx () then investigates the degree of all positive and negative imbalanced training sets from sample in U.If the pseudo-label of x is+1, so R _skew(x) for all from the ratio of the number of negative sample number and positive sample in sample in U.Otherwise pseudo-label is-1, R _skewx () is then positive number of samples and the ratio of negative sample number.

By based on R _skewx the mode of () weighted, when training forecast model, we give the less class of number of samples more weight.This strategy equilibrium is from the impact of positive and negative in U two aspect samples.Meanwhile, by giving | h ₁(x)-h ₂(x) | the more weight of larger sample, in fact we be exaggerated the effect of complementary character between geometric properties and appearance features.

Model Fusion part based on superiority prediction:

After weighting coorinated training process completes, we are by h ₁and h ₂merge the final Expression analysis forecast model h of composition.Merge the mode adopting weighted sum.In the present invention, we propose a kind of new weight calculation strategy.Model Weight is based on predicting that the geometric properties obtained is relative to the advantage of appearance features on U.Concrete has: h=v ₁h ₁+ v ₂h ₂, v ₁and v ₂by superiority prediction model h _1v2output on U calculates.

Superiority prediction model h _1v2object be that prediction geometric properties is relative to the advantage of appearance features.Here advantage refers at tested object data U, performs the advantage of performance during Expression analysis task.If h _1v2there is preferably performance, then to h with higher degree of confidence prediction geometric properties by the Expression analysis task of U ₁larger weight v is set ₁.Otherwise, then to h ₂larger weight v is set ₂.

Superiority prediction model h _1v2obtain by doing binaryzation classification based training on L.By carrying out house one cross validation (leave-one-out cross validation) in L, we can obtain on each object j, model of geometrical features and appearance features model performance separately.Model of geometrical features is had the object of performance advantage as positive sample by us, and appearance features model is had the object of performance advantage as negative sample, by { W _j} _jas training data, Linear SVM is utilized to train the forecast model h that gains the upper hand _1v2.We adopt the method for PlattScaling when training, make h _1v2output between [0,1], representative is the probability that geometric properties has performance advantage.

In testing, we are by h _1v2be applied to U.Export based on it, Model Fusion weight is set: V ₁=h _1v2(U)+λ, v ₂=1-h _1v2(U)+λ.Wherein λ is the regular terms pre-set, and object avoids the impact of a certain feature to be covered completely.λ is set to 1 usually.

Finally, h is applied on U by we, obtains Expression analysis result.Algorithm whole description is as follows:

Data: the set of source data L={X of band espressiove label ^s, Y ^s, and tested object data U={X ^t}

Result: tested object prediction expression label

1. according in FACS about the description of selected Facial action unit, definition geometric properties f ₁with appearance features f ₂

2. choose object set close with tested object U in L, composition training set

2.1 couples of f ₁in each dimension f ₁i (), sets up auxiliary forecasting problem, by f ₂training linear SVM forecast model

2.2 couples of each object j, build auxiliary forecast model long vector W _j=[w _j(1) ^t, w _j(2) ^t..., w _j(l ₁) ^t] ^t, wherein w _ji () is for object j is by f ₁linear SVM weight vectors in i auxiliary forecasting problem that () sets up, l ₁f ₁dimension

2.3 calculate the distance in L between all objects and U, and the distance between object j and k is defined as W _jand W _kbetween distance

2.4 select n with U apart from minimum object in L, composition training set L '

3. weighting coorinated training forecast model h ₁and h ₂

On L ', utilize f ₁training forecast model h ₁, utilize f ₂training forecast model h ₂

By h ₁and h ₂training set be designated as L respectively ₁' and L ₂'

Still there is high confidence level new samples in while U and can add training set L ₁' and L ₂' do

{

By h ₁be applied on U, choose high confidence level sample, add L ₂'; Similarly, by h ₂high confidence level sample adds L ₁'

To L ₁' and L ₂' in each training sample x, according to propose weight calculation strategy, give weight Q _x

Based on sample weights Q _x, re-training h ₁and h ₂

}

4. merge final forecast model h, prediction of output expression label

The geometric properties superiority prediction model h that training in advance is obtained _1v2be applied on U, calculate weight v ₁and v ₂

Fusion forecasting model, h=v ₁h ₁+ v ₂h ₂

Defeated s goes out prediction expression label,

Overall flow figure is shown in Fig. 4.Because the method for the invention has field adaptability, no longer require that test data and training data are from same distribution, the more realistic scene of the present invention, has potential application widely in man-machine interaction and affection computation field.

Above-described is a preferred embodiment of the present invention, is noted that under the premise without departing from the principles of the invention, and some improvements and modifications that the technician of this research field makes also should be considered as protection scope of the present invention.

Claims (6)

1. a domain-adaptive facial Expression Analysis method, is characterized in that following steps:

(1) according in FACS handbook about the description of expression motor unit, definition face facial geometric feature f ₁with appearance features f ₂;

(2) auxiliary forecasting problem is built, by geometric properties f ₁calculate label, by appearance features f ₂prediction label, record Prediction Parameters vector;

(3) by the distance between Prediction Parameters Definition of Vector object, distance definition comprises value close to two parts consistent with symbol, chooses with tested object U apart from close training objects in training set L, composition training set

(4) weighting coorinated training is adopted, training forecast model h ₁and h ₂, training sample weight calculation comprises positive and negative number of samples ratio and h ₁with h ₂absolute difference two factors of score value;

(5) the superiority prediction model h of training in advance is utilized _1v2, merge h ₁and h ₂obtain model h, export and finally predict the outcome.

2. method according to claim 1, is characterized in that:

Described geometric properties and appearance features have independently Expression analysis ability, have complementarity simultaneously.

3. method according to claim 1, is characterized in that: in described step (1), characterizing definition comprises further:

(1.1) geometry such as geometric properties is by angle, length tolerance composition, calculates based on human face characteristic point;

(1.2) appearance features selects the SIFT descriptor of fixed position and yardstick.

4. method according to claim 1, is characterized in that: described step (4) weighting coorinated training sample comprises further: in weight calculation, and all samples from tested object U of ration statistics of positive and negative number of samples, are designated as R _skew(x).

5. method according to claim 1, is characterized in that: described step (4) weighting coorinated training

Sample comprises further: the concrete formula of weight calculation is:

$Q_x\text{=}\left\{\begin{array}{ccc}1+a{R}_{\mathrm{skew}}\left(x\right)+b|h_1\left(x\right)-h_2\left(x\right)|& \mathrm{if}& x\∈U\\ 1& \mathrm{if}& x\∈L^{\′}\end{array}\right.$

Wherein a and b be preset be greater than zero constant, choose obtain by carrying out cross validation (crossvalidation) in a database.

6. method according to claim 1, is characterized in that: described step (5) comprises further: utilize training in advance superiority prediction model h _1v2, prediction geometric properties f ₁relative to appearance features f on tested object U ₂the probability of having the advantage, merges model of geometrical features h ₁with appearance features model h ₂.