KR100839536B1 - System and method for facial region/hair information extraction, character generation - Google Patents

Abstract

An apparatus and a method for extracting facial feature, an apparatus and a method for extracting hair, and a system and a method for generating a photographic character are provided to offer the preprocessing of accurate face recognition and face normalizing, thereby improving face recognition performance even in a change of lighting and a pose and extracting physiognomic information for a physiognomic service. A system for generating a photographic character comprises an image input unit(10), a facial region detecting unit(20), a facial feature extracting unit(30), a hair extracting unit(40), and a photographic character generating unit(50). The facial region detecting unit detects a facial region image from an image received from the outside. The facial feature extracting unit applies IPCA(Incremental Principal Component Analysis) to an AAM(Active Appearance Model) for the facial region image detected in the facial region detecting unit to update learned AAM basis and extract facial feature, and normalizes the facial region image based on the position of the extracted facial feature. The hair extracting unit acquires an image patch corresponding a background region and an image patch corresponding to a head region in a face peripheral image received from the outside and removes the background region using the color histogram information of each image patch and a probability analysis scheme to extract a hair image. The photographic character generating unit matches the facial region image normalized in the facial feature extracting unit and the hair image extracted in the hair extracting unit to generate a photographic character.

Description

Facial feature extraction apparatus and method, hair extraction apparatus and method, real character generation system and method {System and method for facial region / hair information extraction, character generation}

1 is an overall configuration diagram of an embodiment of a live action character generation system according to the present invention;

2 is a flowchart illustrating an embodiment of the method for generating a photorealistic character according to the present invention;

3 is a block diagram of an embodiment of a facial feature point extraction apparatus according to the present invention;

4 is a flowchart illustrating an embodiment of a method for extracting facial feature points according to the present invention;

5 is an explanatory diagram illustrating an example of positions of feature points and an example of a face normalized image according to a method for extracting feature points of the present invention;

6 is a diagram illustrating an embodiment of a hair extraction apparatus and method according to the present invention;

7 is an explanatory diagram showing a process of generating a live-action character according to an embodiment of the present invention.

* Explanation of symbols on the main parts of the drawing

10: image input unit 20: face region extraction unit

30: facial feature point extraction unit 31: model fitting unit

32: fitting accuracy determination unit 33: basis update determination unit

34: base updater 35: normalizer

40: hair extraction unit 41: background area removal unit

42: hair region extraction unit 50: live action character generator

The present invention relates to a facial feature extraction apparatus and its method, a hair extraction apparatus and its method, a live-action character generation system and the method, and a computer-readable recording medium recording a program for realizing the above methods. Extracts facial feature points (images of faces such as eyes, nose, mouth, jawline, etc.) that are robust to various environmental changes (face size change, pose change, wearing glasses and jewelry, lighting change, etc.) Facial feature point extraction apparatus and method for extracting hair images reliably regardless of the shape, color, and shape of various hairs and generating and producing photorealistic characters using the obtained facial images and hair images. Extraction apparatus and method, real character generation system and method and realizing the above methods It relates to a computer-readable recording medium recording the program.

In recent years, a character that represents his or her own character by combining a variety of patterns in advance with a favorite character among sites that provide various contents such as portal services, internet chat, internet broadcasting, online games, and characters. In general, a method of completing a method and applying it to a predetermined animation motion is generally used. In addition, a service has been created in which a professional designer can create a character by a part of automation or the whole manual work and provide it as an avatar.

However, the above methods are hardly seen as their own characters. In addition, there is a limit to use because it is impossible to process in real time and the production cost is high.

When generating a character, the feature points of the face image and the hair region are extracted and generated. The above problems are solved by various environmental changes (face size change, pose change, wearing glasses and jewelry, lighting change, etc.). It comes from the fact that it is impossible to extract the feature points of the face image robustly and reliably. In addition, the reason that the hair can not be reliably extracted even in the change of the shape and color of various hair is also the cause.

In general, face images are information that can express personal characteristics well, and feature point extraction and character generation using real face image information provide face recognition, preprocessing for coronary information service, and basic technology as well as realistic elements. By being applied immediately, it can be the most effective way of expressing itself by overcoming problems such as simplicity or difference from the existing virtual character.

However, in a conventional extraction method such as template matching and curve tracking algorithm, in order to extract all the various feature points of the face, the processing speed takes a lot depending on the change in the size of the face, the change of pose, the wearing of glasses and accessories, the change of lighting, etc. There was a problem that performance is greatly reduced.

The present invention has been proposed to solve the above problems, facial features (eye, nose, mouth, jawline, etc.) that are robust to various environmental changes (face size change, pose change, wearing glasses and jewelry, lighting changes, etc.) It is an object of the present invention to provide a facial feature point extracting apparatus and a method thereof, and a computer-readable recording medium having recorded thereon a program for realizing the method.

In addition, the present invention can be read by a computer recording a hair extracting apparatus and method for extracting hair reliably regardless of the shape, color, and shape of various hairs while being robust to light changes. Another purpose is to provide a record carrier.

In addition, the present invention provides a computer-readable recording system for generating a real character using the extracted facial feature point and hair extraction information, a real character character generation system and method thereof, and a program for realizing the method. Another purpose is to provide the medium.

Other objects and advantages of the present invention can be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Also, it will be readily appreciated that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

A live action character generation system of the present invention for achieving the above object comprises: a face area detection means for detecting a face area image from an image input from the outside; The facial feature image is extracted by updating the learned AAM basis by applying IPCA (Incremental Principal Component Analysis) to the AAM (Active Appearance Model) on the face region image detected by the face region detecting means. Face feature point extracting means for normalizing the face region image based on a position of? Color histogram information and probability of each image patch are obtained by obtaining an image patch corresponding to a background region and an image patch corresponding to a head region from an image normalized around the face from the image received from the outside (hereinafter, referred to as a peripheral image). Hair extracting means for extracting a hair image by removing a background area using a conventional analysis method; And live-action character generation means for generating a live-action character by matching the face region image normalized by the facial feature point extraction means and the hair image extracted by the hair extraction means.

Meanwhile, the facial feature extraction apparatus of the present invention is a facial feature extraction apparatus for obtaining an AAM parameter for minimizing a difference between a facial region image composed of an AAM (Active Appearance Model) and a facial region image of an image input from the outside. Model fitting means; Based on the difference between the face region image of the image received from the outside and the face region image reproduced by using the AAM parameter acquired by the model fitting means, the fitting accuracy is determined, the face feature point with high fitting accuracy is output, and the AAM base update is induced. Fitting accuracy determination means for; Normalization means for normalizing the face region image by obtaining position coordinates of a facial feature point output from the fitting accuracy determining means, extracting texture information therein; A basis update determining means for determining whether the AAM basis is updated by measuring an AAM error of a face region image of the image received from the outside when the fitting accuracy determination means determines that the fitting is correct; And a basis updating means for adding a face region image of the image received from the outside to the previously learned AAM basis using an incremental principal component analysis (IPCA).

Meanwhile, the hair extracting apparatus of the present invention, in the hair extracting apparatus, corresponds to a background region obtained from an image obtained by normalizing a face periphery including a head region and a background region from an image received from the outside (hereinafter, referred to as a face peripheral image). The background region is removed by using color histogram information and a probabilistic analysis method on an image patch, and an approximate probability value of each color at all points (pixels) of the image around the face [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face surrounding image] if it is lower than a predetermined threshold value; And extracting a hair region using color histogram information and a probabilistic analysis method on an image patch corresponding to a head region obtained from the surrounding face image, and extracting the hair region from each point (pixel) of the surrounding face image from which the background region is removed. Approximate probability of color [Note; Here, the approximate probability value indicates the degree to which the color value of the histogram is distributed in the hair area in the face image from which the background area is removed] is higher than a predetermined threshold. Means;

On the other hand, the live-action character generation method of the present invention, a live-action character generation method comprising the steps of: detecting a face region image from an image received from the outside; Extracting a facial feature point by updating an learned AAM basis by applying an incremental principal component analysis (IPCA) to an active appearance model (AAM) on the detected face region image; Normalizing the face region image based on the position of the extracted facial feature point; Obtaining an image patch corresponding to a background area and an image patch corresponding to a head area from an image normalized to a face periphery from an image received from the outside (hereinafter, referred to as a peripheral image); Extracting a hair image by removing a background area by using color histogram information and a probabilistic analysis method of the acquired image patches; And generating a live-action character by matching the normalized face region image with the extracted hair image.

Meanwhile, in the facial feature point extraction method, in the facial feature point extraction method, acquiring an AAM parameter that minimizes a difference between a face region image composed of an AAM (Active Appearance Model) and a face region image of an image input from the outside. ; Determining a fitting accuracy based on a difference between a face region image of the image received from the outside and a face region image reproduced using the obtained AAM parameter, outputting a facial feature point having a high fitting accuracy, and inducing an AAM basis update; Normalizing the face region image by obtaining position coordinates of the output facial feature points and extracting texture information therein; Determining whether the AAM basis is updated by measuring an AAM error of a face region image of an image received from the outside when the fitting is determined to be correct; And adding a face region image of the image received from the outside to the previously learned AAM basis using an incremental principal component analysis (IPCA).

On the other hand, the hair extraction method of the present invention, in the hair extraction method, each point at every point (pixel) of the image (hereinafter referred to as a face peripheral image) normalized around the face including the head region and the background region from the image input from the outside Approximate probability of color [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face image, and if the threshold value is lower than the predetermined threshold, determining that the background area is not the background area and removing the background area; And an approximate probability value of each color at all points (pixels) of the image around the face from which the background region is removed. Here, the approximate probability value indicates the degree to which the color value of the histogram is distributed in the hair area in the face image from which the background area is removed] is determined as the hair area when extracting the corresponding hair image. Include.

On the other hand, the present invention, a live-action character generation system having a processor, the function of detecting a face region image from an image received from the outside; Extracting facial feature points by applying an ACA (Incremental Principal Component Analysis) to the AAM (Active Appearance Model) on the detected face region image to update the learned AAM basis; Normalizing the face region image based on the position of the extracted facial feature point; A function of acquiring an image patch corresponding to a background region and an image patch corresponding to a head region from an image normalized around a face (hereinafter, referred to as a peripheral image) from the image received from the outside; Extracting a hair image by removing a background area by using color histogram information and a probabilistic analysis method of each acquired image patch; And a computer-readable recording medium having recorded thereon a program for realizing a function of generating a real character by matching the normalized face region image with the extracted hair image.

On the other hand, the present invention, a facial feature point extraction apparatus having a processor, the function for obtaining an AAM parameter for minimizing the difference between the face region image composed of the AAM (Active Appearance Model) and the image received from the outside; Determining a fitting accuracy based on a difference between the face region image of the image input from the outside and the face region image reproduced using the obtained AAM parameter, outputting a facial feature point having a high fitting accuracy, and inducing an AAM basis update; Obtaining a position coordinate of the output facial feature point and extracting texture information therein to normalize the face region image; Determining whether the AAM basis is updated by measuring an AAM error of a face region image of the image received from the outside when the fitting is determined to be correct; And a computer readable recording medium recording a program for realizing a function of adding a face region image of the image input from the outside to an AAM basis previously learned using an incremental principal component analysis (IPCA).

On the other hand, the present invention, in the hair extraction apparatus provided with a processor, each color at every point (pixel) of the image (hereinafter, the peripheral image of the face) normalized around the face including the head region and the background region from the image input from the outside Approximate probability of [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face surrounding image] is less than a predetermined threshold, and determines that the background area is not the background area, and removes the background area; And an approximate probability value of each color at all points (pixels) of the image around the face from which the background region is removed. Here, the approximate probability value indicates the degree to which the color value of the histogram is distributed in the hair area in the face image from which the background area is removed] is determined as the hair area, and the hair image is extracted. A computer readable recording medium having recorded thereon a program for realization is provided.

The present invention relates to the extraction of facial regions including hair and the creation of live-action characters, which are characterized by facial feature points (eyes, nose, mouth, jawline, etc.) that are robust to face size changes, pose changes, whether glasses and jewelry are worn, lighting changes, etc. Facial feature position information), hair extraction that is robust against changes in illumination, and reliable for changes in the shape and color of various hairs, and real-life characters are generated and produced using the extracted feature points and hair extraction information thus obtained.

To this end, in the present invention, by applying the Incremental Principal Component Analysis (IPCA) to the existing AAM (Active Appearance Model), by updating the learned AAM basis (Basis) to obtain a facial feature point extraction result that is robust to various changes, such as lighting, user It is able to reliably extract the hair area regardless of the color and shape of various hairs while being robust to lighting changes by background modeling and hair modeling, and generate and produce live-action characters by matching them.

According to the present invention, by providing the pre-processing and face normalization of the accurate face recognition, it is possible to improve the face recognition performance that is robust to lighting and pose changes, face modeling, coronary information extraction for coronary services according to the shape of the facial feature point, hair Real-life face modeling and character generation can be realized by synthesizing live-action hair by the extraction module.Real-time character generation module can automatically generate real-time character of customer image in real time, effectively reducing character creation cost and creation time. In addition, it is possible to obtain an effect that can produce a customized product using the character image of the customer.

The above objects, features and advantages will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, whereby those skilled in the art may easily implement the technical idea of the present invention. There will be. In addition, in describing the present invention, when it is determined that the detailed description of the known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description thereof will be omitted. Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

1 is a block diagram of an embodiment of a live action character generation system according to the present invention.

As shown in FIG. 1, the live action character generation system according to the present invention includes an image input unit 10, a face region detection unit 20, a facial feature point extractor 30, a hair extractor 40, and a live action character generation. Part 50 is made. In this case, the face area detection unit 20 detects the position and size of the face area in the image input unit 10. In the present invention, a face area detection method uses a known method.

The live-action character generation unit 50 obtains the final live-action character by matching the face normalized image obtained by the facial feature point extraction unit 30 with the hair extraction image obtained by the hair extraction unit 40.

Specifically, the face region detector 20 for detecting the position and size of the face region in the input image input through the image input unit 10, and the AAM (AAM) within the face region detected by the face region detector 20. Applying Incremental Principal Component Analysis (IPCA) to an Active Appearance Model (IPCA) to update the learned AAM Basis to extract facial feature points with high fitting accuracy, and to normalize faces based on the location of the extracted feature points From the feature point extractor 30 and the normalized image (the image normalized only around the face including the hair in the input image input through the image input unit 10), a small image patch that is assured as the background region and the head region is obtained. Hair extraction unit 40 for removing background area and extracting live hair through histogram information and probabilistic analysis method of patches, and face It includes a live-action character generation unit 50 for generating a live-action character by matching the face normalized image normalized by the feature point extraction unit 30 and the live-action hair image extracted by the hair extraction unit 40.

Looking at the operation of the live-action character generation system according to the present invention having such a configuration as shown in FIG.

As shown in FIG. 2, when an image is first input through the image input unit 10 (201), the face region detection unit 20 determines the position of the face region in the input image input using a known face region detection method. In step 202, the facial feature point extractor 30 applies an incremental principal component analysis (IPCA) to an AAM (Active Appearance Model) within the face region detected by the face region detector 20, thereby learning the AAM. By updating the basis, a facial feature point with high fitting accuracy is extracted, and the face is normalized based on the position of the extracted feature point (203).

In addition, the hair extractor 40 extracts the background region from the normalized input image (the image normalized only around the face including the hair in the input image input through the image input unit 10) through color histogram information and a probabilistic analysis method. Remove and extract live hair (204).

Thereafter, the live action character generator 50 generates a live action character by matching the face normalized image normalized by the facial feature point extractor 30 and the live hair image extracted by the hair extractor 40 (205).

The facial feature point extraction process 203 and the hair extraction process 204 will be described in more detail below.

First, the facial feature point extractor 30 and the facial feature point extraction process 203 will be described.

3 is a block diagram of an embodiment of a facial feature point extraction apparatus according to the present invention.

The facial feature point extractor 30 applies an IPA (Incremental Principal Component Analysis) to an AAM (Active Appearance Model) in the face region detected by the face region detector 20, thereby updating the learned AAM basis. A facial feature point with high fitting accuracy is extracted, and the face is normalized based on the location of the extracted feature point. At this time, in extracting facial feature points, the facial feature points (eyes, noses) that are robust to various environmental changes (face size change, pose change, wearing glasses and accessories, lighting change, etc.) through AAM Basis update Facial feature position information of the face, such as a mouth, a jawline, etc.) can be extracted.

As shown in FIG. 3, the facial feature point extractor 30 may include a model fitting unit 31 for obtaining an AAM parameter for minimizing a difference between a face image composed of AAM and an input face image, and an input image and an AAM parameter. The accuracy of the fitting is judged based on the difference of the images reproduced by using the image, and the fitting accuracy judging unit 32 which outputs the facial feature point with high fitting accuracy and induces the base update is obtained, and the coordinates of the facial feature points existing on the face contour are obtained. A normalization unit 35 for extracting texture information therein, and a base update determination unit 33 for determining whether to update the AAM basis by measuring the AAM error of the current input image when the fitting is correct. And a base updater 34 that adds a current input image to an existing learned basis using an incremental principal component analysis (IPCA).

Here, the main algorithms used are Active Appearance Model (AAM) and Incremental Principal Component Analysis (IPCA).

In AAM, objects (faces in the present invention) are modeled by dividing them into shapes and textures.By using face learning data, an average face is composed and a face area within an image from outside is defined. Similarly, the parameters of the shape and texture of the average face change until they converge. At this time, each of the shape and texture is modeled by PCA (Principal Component Analysis).

Therefore, the AAM has the advantage of being able to flexibly express various deformations of the object, and has been applied to various fields such as face tracking, facial feature point extraction, and facial expression recognition.

However, since each PCA model of shapes and textures in AAM depends on the face data used for training, when a face input image comes in significantly different from the data used for training, the existing trained model expresses a new face input image well. If you can't do that.

Therefore, in the present invention, by applying the IPCA to the existing AAM, by updating the learned AAM Basis, it is robust to various environmental changes (face size change, pose change, wearing glasses and jewelry, lighting change, etc.). The result of extracting facial feature points (feature location information of the face such as eyes, nose, mouth, and jawline) can be obtained.

Looking at the operation of the facial feature point extraction unit 30 according to the present invention having the configuration as described above is as shown in FIG.

First, in the case of a still image or a motion image, the image input unit 10 transmits an image of a frame unit to the face region detector 20 (401).

Thereafter, the face region detector 20 detects the face region from the input image and provides the size and position information of the face detected by the face feature point extractor 30 (402).

Then, the model fitting unit 31 of the facial feature point extractor 30 obtains an AAM parameter for minimizing the difference between the face image composed of the AAM and the input face image (403). At this time, the process of acquiring the AAM parameter is performed while repeatedly changing the AAM parameter until the error value falls below a predetermined value in one image.

The texture information in the face image region represented by the AAM parameter thus obtained is passed to the fitting accuracy determiner 32. This is because when the AAM base update is performed using an incorrect image of the fitting, the existing model may be changed even worse, so it is necessary to determine whether the current input image is pitched correctly.

Therefore, the fitting accuracy determination unit 32 determines the accuracy of the fitting from the difference value between the texture information in the face image and the texture of the image reproduced by the AAM parameter (404).

As a result of the determination, if the fitting is correct, the base update is induced for an image having a good fitting result. At this time, by outputting a facial feature point with high fitting accuracy, the normalization unit 35 obtains the coordinates of the facial feature point existing on the facial contour and extracts texture information therein.

As a result of the determination, if the fitting is not correct, the process proceeds to the next frame without performing the base update on the image of the current frame (401).

Thereafter, the base update determiner 33 measures the AAM error of the current input image so as to perform the base update only when the error is larger than a predetermined value (see Equation 1 below) (405). Here, since the error is small, the input image can be sufficiently represented by the currently learned base value. Therefore, the AAM basis is updated only when the error is large without performing the AAM base update.

Finally, the base updater 34 adds the current input image to the previously learned base using IPCA (406). In this case, when the basis is added, the average, variance, basis vector, and Eigen-value of the training data change. At this time, the basis update process will be described in more detail through the following equation.

First, as a determination condition for the base update, as shown in Equation 1 below, the AAM error of the current input image is measured to perform the base update only when the error is larger than a predetermined value.

delete

In Equation 1, I (x) is an input image, I _re (x) is an image reproduced with an AAM parameter, σ is a variance of pixels, and c ₁ is a threshold value for determining fitting accuracy. That is, if the difference between the input image and the image reproduced with the AAM parameter exceeds the threshold value, it is determined that it is a wrong fit and does not perform the base update, and the base update process is performed only when the threshold value is not exceeded.

는 현재 학습데이터의 평균, x _{n +1} 은 현재 프레임의 입력영상,

는 현재 입력영상이 AAM 학습에 추가되었을 경우의 새로운 학습 데이터의 평균을 나타낸다. In Equation 2, n is the number of learning data currently used for AAM learning.

Is the average of the current learning data, x _{n +1} is the input image of the current frame,

Represents the average of new learning data when the current input image is added to AAM learning.

는 다음 입력영상에서 현재 학습 데이터의 평균으로 사용된다. Where the average of the new training data

Is used as the average of the current learning data in the next input image.

는 현재 학습데이터의 평균, I(x)는 현재 프레임의 입력영상, C는 현재 학습데이터의 Covariance 행렬, C`는 현재 입력영상이 AAM 학습에 추가되었을 경우의 새로운 학습 데이터의 Covariance 행렬을 나타낸다.Equation 3 shows a covariance matrix when a current input image is added. n is the number of learning data currently used for AAM learning.

Is the average of the current training data, I (x) is the input image of the current frame, C is the covariance matrix of the current training data, and C` is the covariance matrix of the new training data when the current input image is added to AAM learning.

는 현재 학습데이터의 평균, I(x)은 현재 입력영상이다. The input image I (x) of the current frame appears as shown above in the learned AAM texture space. U is a set of currently learned basis vectors, a _{n + 1} is a parameter vector when the input image is projected onto the currently learned space,

Is the mean of the current learning data, I (x) is the current input image.

When the parameter vector a _{n + 1} of the input image vector appearing in the AAM texture space is restored to the image space, it is expressed as Equation 5 above. In this case, a difference from the original image is inevitably generated during the restoration process.

In Equation 6, b _{n +1} represents an extra vector that is not represented as a learned basis (Basis). In this case, when the extra vector is small, it means that the input image is well represented by the learned Basis, and thus the process is completed without updating. However, if the extra vector is large, the base vector must be updated to represent the input image well, so the update process continues.

는 정규화한 여분 벡터 b _{n +1} 이며, R은 회전 행렬, U`는 입력영상이 적용된 새로운 기저행렬이다. 즉, 현재 영상에서 기존 Basis로 표현되지 않는 부분만을 새로운 Basis에 추가하여 전체 기저값을 회전한 것이 새로운 기저 벡터의 집합이 되는 것이다.In Equation 7, U is a set of currently learned basis vectors,

Is the normalized extra vector b _{n +1} , R is the rotation matrix, and U` is the new base matrix to which the input image is applied. In other words, a new basis is a set of new basis vectors by adding only a portion of the current image that is not represented by the existing basis to the new basis.

는 eigenvalue 행렬이다.When the parameter vector of the new input image is added to the existing AAM texture model, eigenproblem is expressed as shown in [Equation 8]. C` represents the Covariance Matrix of the new training data when the current input image is added to the AAM learning, U` new base matrix is applied to the input image, and

Is the eigenvalue matrix.

When [Equation 3] and [Equation 7] are substituted into [Equation 8], the above [Equation 9] is derived. Since this is the eigenproblem for R , solving this equation gives us the R matrix. In addition, substituting the obtained R into Equation 7 yields a new base matrix U` to which a new input image is applied.

5 is an explanatory diagram illustrating an example of positions of feature points and an example of a face normalized image according to the method for extracting feature points of the present invention.

The position 5a of the feature points obtained by applying the AMM and the IPCA to the position of the feature points to be obtained from the face used in the present invention and the face region detected in the actual input image, and the face normalization result 5b using the same.

Now, the hair extractor 40 and the hair extraction process 204 will be described.

As shown in FIG. 6, the background region is determined based on an approximate probability value according to the degree of the color value of the histogram being distributed in the background region, and the normalized input image (in the input image input through the image input unit 10). Background region removal unit 41 for removing the background region from the normalized area around the face including the hair, and an approximate probability value according to the degree to which the histogram color value is distributed in the hair region in the image from which the background region is removed. A hair region extracting unit 42 for determining a hair region based on the ground and extracting a live hair image is included.

Here, the approximate probability value used in the background region removal unit 41 is a probability value (H2) obtained by normalizing the histogram of the patch region and normalizing a probability value (H1) representing the frequency of occurrence of each histogram after obtaining a histogram of the entire input image. Divided by), the color value of one histogram is distributed in the background area.

Therefore, the background region removal unit 41 is a small image patch (i.e., the image region reliably belonging to the background region) in the normalized input image (the image normalized only around the face including the hair in the input image input through the image input unit 10). From the normalized input image, we obtain a small image patch that clearly belongs to the background region on the human shoulder and outside the ear, and remove the background region using the color histogram, information, and probabilistic analysis of the patch. If the approximate probability value H1 / H2 of each color is lower than a predetermined threshold at all points of the normalized input image, it is determined that it is not a background area.

In addition, the approximate probability value used in the hair region extraction unit 42 is a probability value obtained by normalizing the histogram of the patch region to obtain a histogram of the entire input image after obtaining a histogram of the entire input image (H2). Divided by), and represents the degree to which the color value of one histogram is distributed in the hair region.

Therefore, the hair region extracting unit 42 is a small image patch (i.e., a small image patch reliably belonging to the hair region) in the normalized input image (the image normalized only around the face including the hair in the input image input through the image input unit 10). In the image from which the background area is removed, a small image patch that is sure to be the head area at the top of the face) is extracted, and the hair area is extracted through the color histogram, information, and probabilistic analysis method of the patch. At every point, the area of the approximate probability (H1 / H2) of each color higher than the predetermined threshold is discriminated as the hair area to extract the live hair.

The hair extraction unit 40 uses color image information, that is, R-G-B value, and both the input image and the final result are also represented as color images.

The background region removing unit 41 is a preprocessing step for extracting a hair region, and has a function of removing a background region from a given input image, which reliably belongs to the background region in the image region outside the ear on the human shoulder in the input image. A small image patch is obtained and implemented through color histogram information and probabilistic interpretation of the patch.

In this regard, first, the histogram of the patch region is normalized to display the frequency of occurrence of each histogram as a probability. Likewise, the histogram of the entire input image is obtained and then normalized to obtain a probability value. In this case, a value obtained by dividing the probability value H1 of the histogram of the patch region by the probability value H2 of the histogram of the entire input image is an approximate probability value of how much the color value of one histogram is distributed in the background region. That is, since the color of one point in the face area that is not the background area does not belong to the background area, the value of (H1 / H2) above will be very low, whereas the color of one point of the background area is (H1 / H2). It will have a relatively high value. Therefore, if it is lower than the preset threshold value by using (H1 / H2) of each color at every point of the input image, it is determined that it is not a background area.

The hair region extracting unit 42 obtains a small image patch that is secured to the head region at the top of the face in the image from which the background region is removed through the background region removing unit 41, and the background region removing unit 41 obtains By applying the proposed method to the color histogram information of the image patch, it is possible to reliably extract hair regardless of the shape, color, and shape of various hairs while being robust to lighting changes by distinguishing only the area above the critical point as the hair area.

7 is an example of applying live-action character extraction, and shows a process of generating a live-action character by extracting facial feature points from the input image and extracting the real hair, matching the normalized face normalized image with the live-hair hair image.

The method of the present invention can also be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all kinds of recording devices that store data that can be read by a computer system. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage, and the like, which are also implemented in the form of a carrier wave (for example, transmission over the Internet). It also includes. The computer readable recording medium can also be distributed over network coupled computer systems so that the computer readable code is stored and executed in a distributed fashion. And functional programs, codes and code segments for implementing the present invention can be easily inferred by programmers in the art to which the present invention belongs.

The present invention described above is capable of various substitutions, modifications, and changes without departing from the technical spirit of the present invention for those skilled in the art to which the present invention pertains. It is not limited by the drawings.

As described above, the present invention is characterized by facial feature points (eye, nose, mouth, jawline, etc.) that are robust to various environmental changes (face size change, pose change, wearing glasses and jewelry, lighting change, etc.). ) Can extract the effect.

In addition, the present invention has the effect of being able to extract hair reliably regardless of the shape, color, and shape of various hairs while being robust to lighting changes.

In addition, the present invention has the effect of generating and producing a live-action character using extracted facial feature points and hair extraction information.

Accordingly, the present invention, by providing the pre-processing and face normalization of the accurate face recognition, it is possible to improve the face recognition performance, face modeling, and coronary information extraction for the coronary service according to the shape of the facial feature point, which is robust to lighting and pose changes, The realistic hair modeling and character generation can be realized by synthesizing the photorealistic hair with the hair extraction module, and the real-time character generation module automatically generates real-time character of the customer image, effectively reducing the character creation cost and creation time. And, there is an advantage that can produce a customized product using the customer's character image.

Claims (31)

In the live-action character generation system, Face region detection means for detecting a face region image from an image received from the outside; The facial feature image is extracted by updating the learned AAM basis by applying IPCA (Incremental Principal Component Analysis) to the AAM (Active Appearance Model) on the face region image detected by the face region detecting means. Face feature point extracting means for normalizing the face region image based on a position of? Color histogram information and probability of each image patch are obtained by obtaining an image patch corresponding to a background region and an image patch corresponding to a head region from an image normalized around the face from the image received from the outside (hereinafter, referred to as a peripheral image). Hair extracting means for extracting a hair image by removing a background area using a conventional analysis method; And Photorealistic character generation means for generating a photorealistic character by matching the facial region image normalized by the facial feature point extraction means and the hair image extracted by the hair extraction means Photorealistic character generation system comprising a. The method of claim 1, The facial feature point extracting means, Model fitting means for obtaining an AAM parameter for minimizing a difference between a face region image composed of the AAM and a face region image of an image input from the outside; Based on the difference between the face region image of the image received from the outside and the face region image reproduced by using the AAM parameter acquired by the model fitting means, the fitting accuracy is determined, the face feature point with high fitting accuracy is output, and the AAM base update is induced. Fitting accuracy determination means for; Normalization means for normalizing the face region image by obtaining position coordinates of a facial feature point output from the fitting accuracy determining means, extracting texture information therein; A basis update determining means for determining whether the AAM basis is updated by measuring an AAM error of a face region image of the image received from the outside when the fitting accuracy determination means determines that the fitting is correct; And Basis update means for adding the face region image of the image received from the outside using the IPCA to the previously learned AAM basis Photorealistic character generation system comprising a. The method of claim 2, The model fitting means, Photorealistic character characterized in that to obtain the AAM parameter by changing the AAM parameter until the difference between the face region image composed of the AAM and the face region image of the image received from the outside falls below a predetermined value Generating system. The method of claim 2, The fitting accuracy determination means, And based on the result of determining the fitting accuracy, the AAM basis update is not induced for a facial feature point having a low fitting accuracy. The method according to any one of claims 1 to 4, The face surrounding image, Photorealistic character generation system, characterized in that the normalized surrounding the face including the head region and the background region from the image received from the outside. The method of claim 5, wherein The hair extracting means, The background area is removed using color histogram information and a probabilistic analysis method for the image patch corresponding to the acquired background area, and approximate probability values of each color at all points (pixels) of the image around the face [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face surrounding image] if it is lower than a predetermined threshold value; And An approximate probability value of each color is extracted from all points (pixels) of the peripheral image of the face from which the background region is removed by using color histogram information and a probabilistic analysis method on an image patch corresponding to the acquired head region. [week; Here, the approximate probability value indicates the degree to which the color value of the histogram is distributed in the hair area in the face image from which the background area is removed] is higher than a predetermined threshold. Way Photorealistic character generation system comprising a. The method of claim 6, The approximate probability value for determining the background area is And a probability value representing a frequency of occurrence of each histogram obtained by normalizing the histogram of the image patch corresponding to the acquired background region, divided by a probability value obtained by normalizing the entire histogram of the image around the face. delete The method of claim 7, wherein The image patch corresponding to the acquired background region, the live action character generation system, characterized in that it comprises the image portion on the shoulder of the person, the outside of the ear in the peripheral face image. The method of claim 6, The approximate probability value for determining the hair region is The probability value indicating the frequency of occurrence of each histogram obtained by normalizing the histogram of the image patch corresponding to the obtained head region is divided by the probability value obtained by normalizing the entire histogram of the image around the face from which the background region is removed. Character generation system. The method of claim 10, The image patch corresponding to the obtained head region, Photorealistic character generation system, characterized in that it comprises a top image portion of the face in the peripheral image is removed from the background area. In the facial feature extraction apparatus, Model fitting means for obtaining an AAM parameter for minimizing a difference between a face region image composed of an AAM (Active Appearance Model) and a face region image of an image input from the outside; Based on the difference between the face region image of the image received from the outside and the face region image reproduced by using the AAM parameter acquired by the model fitting means, the fitting accuracy is determined, the face feature point with high fitting accuracy is output, and the AAM base update is induced. Fitting accuracy determination means for; Normalization means for normalizing the face region image by obtaining position coordinates of a facial feature point output from the fitting accuracy determining means, extracting texture information therein; A basis update determining means for determining whether the AAM basis is updated by measuring an AAM error of a face region image of the image received from the outside when the fitting accuracy determination means determines that the fitting is correct; And Basis update means for adding the face region image of the image received from the outside to the previously learned AAM basis using Incremental Principal Component Analysis (IPCA) Facial feature point extraction apparatus comprising a. The method of claim 12, The model fitting means, The AAM parameter is obtained by changing the AAM parameter until the difference between the face region image composed of the AAM and the face region image of the image input from the outside falls below a predetermined value. Extraction device. The method of claim 12, The fitting accuracy determination means, And the AAM basis update is not induced for a facial feature point having a low fitting accuracy according to the result of determining the fitting accuracy. The method according to any one of claims 12 to 14, The base update means, A basis update is performed by adding a portion of the face region image of the image received from the outside to a new AAM basis by adding a portion to the new AAM basis to set a set of new basis vectors by rotating the entire basis value. Facial feature point extraction device. In the hair extraction device, Color histogram information about the image patch corresponding to the background area obtained from the image normalized around the face including the head region and the background region from the image input from the outside (hereinafter, referred to as the peripheral image), and the background using the probabilistic analysis method. An approximate probability value of each color is removed at all points (pixels) of the surrounding image of the face. Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face surrounding image] if it is lower than a predetermined threshold value; And The hair region is extracted using color histogram information and a probabilistic analysis method on the image patch corresponding to the head region obtained from the surrounding face image, and each color is extracted at every point (pixel) of the surrounding face image from which the background region is removed. Approximate probability of [Note; Here, the approximate probability value indicates the degree to which the color value of the histogram is distributed in the hair area in the face image from which the background area is removed] is higher than a predetermined threshold. Way Hair extraction device comprising a. The method of claim 16, The approximate probability value for determining the background area is And a probability value representing a frequency of occurrence of each histogram obtained by normalizing the histogram of the image patch corresponding to the acquired background region, divided by a probability value obtained by normalizing the entire histogram of the surrounding image. The method of claim 17, The image patch corresponding to the acquired background region, the hair extraction device, characterized in that including the image portion on the shoulder of the person, the outside of the ear in the peripheral face image. The method of claim 16, The approximate probability value for determining the hair region is A probability value representing the frequency of occurrence of each histogram obtained by normalizing the histogram of the image patch corresponding to the acquired head region, and the hair divided by a probability value obtained by normalizing the entire histogram of the face image from which the background region is removed. Extraction device. The method of claim 19, The image patch corresponding to the obtained head region, Hair extraction apparatus, characterized in that it comprises a top image portion of the face in the peripheral face image is removed from the background area. In the live-action character generation method, Detecting a face region image from an image received from the outside; Extracting a facial feature point by updating an learned AAM basis by applying an incremental principal component analysis (IPCA) to an active appearance model (AAM) on the detected face region image; Normalizing the face region image based on the position of the extracted facial feature point; Obtaining an image patch corresponding to a background area and an image patch corresponding to a head area from an image normalized to a face periphery from an image received from the outside (hereinafter, referred to as a peripheral image); Extracting a hair image by removing a background area by using color histogram information and a probabilistic analysis method of the acquired image patches; And Generating a photo-realistic character by matching the normalized face region image with the extracted hair image Photorealistic character generation method comprising a. The method of claim 21, Extracting the facial feature point, Obtaining an AAM parameter for minimizing a difference between a face region image composed of the AAM and a face region image of an image input from the outside; Determining a fitting accuracy based on a difference between a face region image of the image received from the outside and a face region image reproduced using the obtained AAM parameter, outputting a facial feature point having a high fitting accuracy, and inducing an AAM basis update; Normalizing the face region image by obtaining position coordinates of the output facial feature points and extracting texture information therein; Determining whether the AAM basis is updated by measuring an AAM error of a face region image of an image received from the outside when the fitting is determined to be correct; And Adding a face region image of the image received from the outside to the previously learned AAM basis using the IPCA; Photorealistic character generation method comprising a. The method of claim 21 or 22, Extracting the hair image, Approximate probability value of each color at all points (pixels) of the surrounding image of the face [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face image, and if the threshold value is lower than the predetermined threshold, determining that the background area is not the background area and removing the background area; And Approximate probability value of each color at all points (pixels) of the image around the face from which the background region is removed [Note; Where the approximate probability value represents the degree to which the color value of the histogram is distributed in the hair region in the face image from which the background region has been removed] is determined as the hair region, and the hair image is extracted. Photorealistic character generation method comprising a. In the facial feature point extraction method, Obtaining an AAM parameter for minimizing a difference between a face region image composed of an AAM (Active Appearance Model) and a face region image of an image input from the outside; Determining a fitting accuracy based on a difference between a face region image of the image received from the outside and a face region image reproduced using the obtained AAM parameter, outputting a facial feature point having a high fitting accuracy, and inducing an AAM basis update; Normalizing the face region image by obtaining position coordinates of the output facial feature points and extracting texture information therein; Determining whether the AAM basis is updated by measuring an AAM error of a face region image of an image received from the outside when the fitting is determined to be correct; And Adding a face region image of the image received from the outside to the previously learned AAM basis using incremental principal component analysis (IPCA); Facial feature point extraction method comprising a. The method of claim 24, Obtaining the AAM parameter, The AAM parameter is obtained by changing the AAM parameter until the difference between the face region image composed of the AAM and the face region image of the image input from the outside falls below a predetermined value. Extraction method. In the hair extraction method, Approximate probability value of each color at all points (pixels) of the normalized image around the face including the head region and the background region from the image input from the outside (hereinafter, the face peripheral image) [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face image, and if the threshold value is lower than the predetermined threshold, determining that the background area is not the background area and removing the background area; And Approximate probability value of each color at all points (pixels) of the image around the face from which the background region is removed [Note; Where the approximate probability value represents the degree to which the color value of the histogram is distributed in the hair region in the face image from which the background region has been removed] is determined as the hair region, and the hair image is extracted. Hair extraction method comprising a. The method of claim 26, The approximate probability value for discriminating the background area is a probability value representing the frequency of occurrence of each histogram obtained by normalizing the histogram of the image patch corresponding to the obtained background area, divided by the probability value obtained by normalizing the entire histogram of the image around the face. Characterized in that the value, The image patch corresponding to the acquired background region, the hair extraction method, characterized in that it comprises the image portion on the shoulder of the person, the outside of the ear in the peripheral face image. The method of claim 26, The approximate probability value for determining the hair region is a probability value representing the frequency of occurrence of each histogram obtained by normalizing the histogram of the image patch corresponding to the acquired hair region, and normalizing the entire histogram of the surrounding image where the background region is removed. Characterized in that divided by the probability value obtained by The image patch corresponding to the acquired head region, the hair extraction method, characterized in that it comprises a top image portion of the face in the peripheral image from which the background region is removed. In the live-action character generation system provided with a processor, Detecting a face region image from an image received from the outside; Extracting facial feature points by applying an ACA (Incremental Principal Component Analysis) to the AAM (Active Appearance Model) on the detected face region image to update the learned AAM basis; Normalizing the face region image based on the position of the extracted facial feature point; A function of acquiring an image patch corresponding to a background region and an image patch corresponding to a head region from an image normalized around a face (hereinafter, referred to as a peripheral image) from the image received from the outside; Extracting a hair image by removing a background area by using color histogram information and a probabilistic analysis method of each acquired image patch; And A function of generating a live-action character by matching the normalized face region image with the extracted hair image A computer-readable recording medium having recorded thereon a program for realizing this. In a facial feature point extraction device having a processor, Obtaining an AAM parameter for minimizing a difference between a face region image composed of an AAM (Active Appearance Model) and a face region image of an image input from the outside; Determining a fitting accuracy based on a difference between the face region image of the image input from the outside and the face region image reproduced using the obtained AAM parameter, outputting a facial feature point having a high fitting accuracy, and inducing an AAM basis update; Obtaining a position coordinate of the output facial feature point and extracting texture information therein to normalize the face region image; Determining whether the AAM basis is updated by measuring an AAM error of a face region image of the image received from the outside when the fitting is determined to be correct; And A function of adding the face region image of the image input from the outside to the previously learned AAM base using Incremental Principal Component Analysis (IPCA) A computer-readable recording medium having recorded thereon a program for realizing this. In the hair extraction device equipped with a processor, Approximate probability value of each color at all points (pixels) of the normalized image around the face including the head region and the background region from the image input from the outside (hereinafter, the face peripheral image) [Note; Wherein the approximate probability value indicates the degree to which the color value of the histogram is distributed in the background area in the face surrounding image] is less than a predetermined threshold, and determines that the background area is not the background area, and removes the background area; And Approximate probability value of each color at all points (pixels) of the image around the face from which the background region is removed [Note; Here, the approximate probability value indicates the degree to which the color value of the histogram is distributed in the hair region in the face image from which the background region has been removed. A computer-readable recording medium having recorded thereon a program for realizing this.

Images