CN114972053B - Simulated lip gloss cosmetic method - Google Patents

Abstract

The invention discloses a simulated lip gloss cosmetic method, which comprises the following steps: step 1, when an image is shot, taking a human face as a recognition area, and storing the recognition area image; step 2, extracting brightness characteristics of the face recognition image, and calculating a brightness average value; step 3, removing noise in the brightness average value according to the brightness average value, and calculating a brightness true value; step 4, carrying out brightness correction on the face recognition image; step 5, extracting a lip area as a subsequent operation area; step 6, the user selects lip gloss color, and whether lip gloss blending is performed or not is selected; if the lip gloss blending is not carried out, each lip gloss color and each lip area color are blended, and a lip gloss effect graph is generated; if lip gloss blending is carried out, all lip gloss colors and lip area colors are blended, and a lip gloss effect graph is generated. According to the invention, the real brightness value is calculated, so that the make-up effect can be truly simulated; by controlling the melting and mixing mode, various lip priming effects, lip biting make-up and other high-grade dressing effects are realized.

Description

A kind of simulation lip gloss makeup method

Technical Field

The invention relates to the field of simulated beauty makeup, and in particular to a simulated lip gloss beauty makeup method.

Background Art

At present, there are endless applications such as real-time beauty and beauty camera, which use facial recognition functions to provide various virtual try-on effects including beauty, lip gloss, foundation, highlights and shadows. These effects are suitable for various scenarios such as live broadcasts, short videos, and video calls. With the maturity of AI and AR technology, there are more and more scenarios for beauty applications. Among the many beauty effects, lip gloss trial makeup is the most widely used. Through mobile phone apps, consumers can try various lipstick colors without leaving home, and then realize online guided shopping. With the arrival of the epidemic and the strengthening of people's awareness of safety and hygiene, AR beauty functions are also provided in physical stores. Using AI and AR technology, consumers can quickly see the makeup effects of different lipstick colors, providing contactless virtual trial makeup services, which are convenient, fast, clean and hygienic.

At present, the technical solution for lip color testing generally uses standardized tools to extract lipstick color and material to make lip color pictures. Third-party facial recognition libraries are used to detect key points of the lips, and image rendering technologies such as OpenGL are used to map the lips. However, the previous technical solutions mainly have the following problems:

1. When trying on makeup virtually, LUT (Look-Up-Table) is generally used or the RGB value of the lip color is directly extracted for coloring. The user's own lip base color is not taken into consideration, resulting in the lip color not blending naturally with the lips. The stored lip color is simply copied to the user's lips, which will result in different effects when the user tries on makeup and when the user uses it in person after purchase.

2. When trying on lip gloss in real life, the color of the lip gloss will be affected by ambient light, etc., so the color you actually see will change with the change of ambient light. When trying on virtual makeup, the lip gloss effect does not take into account the influence of ambient light and cannot truly restore the actual makeup effect.

Summary of the invention

The present invention provides a method to overcome the above problems.

The present invention comprises the following steps:

Step 1: When a user takes an image, the image area is divided synchronously, the face is used as the recognition area, and the recognition area image, i.e., the face recognition image, is stored;

Step 2: Extract brightness features of each facial feature area in the image of the user's face recognition and calculate the brightness mean;

Step 3: According to the brightness mean, remove the noise in the brightness mean and calculate the true brightness value;

Step 4: Perform brightness correction on the image of the user's face recognition according to the true brightness value;

Step 5: Divide the image of the user's face recognition into regions and extract the lip region as the subsequent operation region;

Step 6. The user selects the lip gloss color and chooses whether to blend the lip gloss. If not, each lip gloss color and lip area color will be blended to generate a lip gloss effect image. If lip gloss is blended, all selected lip gloss colors and lip area colors will be blended to generate a lip gloss effect image.

Furthermore, step 2 performs brightness detection on the face recognition image and calculates the brightness mean:

measurement＝0.299*R+0.587*G+0.114*B (1)

Among them, R represents the red component in the sampled texture; G represents the green component; B represents the blue component; the value range of R, G, and B is [0,1]; measurement represents the brightness mean, and the value range of the brightness mean is [0,1].

Furthermore, step 3 removes the noise in the brightness mean value to obtain the true brightness value, including the following steps:

Step 31, using the covariance prediction equation, calculate the estimated uncertainty in the brightness mean:

p＝p+q (2)

Among them, p is the estimation uncertainty, q is the process noise value, and the process noise value is set based on experience;

Step 312: Calculate the result gain in the brightness mean according to the estimated uncertainty:

k＝p/(p+r) (3)

Among them, k is the result gain, r is the camera sensor noise value, and the camera sensor noise value is set based on experience;

Step 32: Calculate the actual brightness value based on the result gain:

x＝k*measurement+(1-k)*x (4)

Among them, measurement is the mean brightness and x is the true brightness value.

Furthermore, if the user chooses not to blend the lip gloss in step 6, the following steps are performed:

Step 6A: The user selects one or more lip gloss colors, performs brightness correction on each lip gloss color, blends the corrected lip gloss color with the lip base color, and displays lip gloss effect pictures of the corresponding lip gloss colors respectively;

Furthermore, if the user blends the lip gloss in step 6, the following steps are performed:

Step 6B1: The user selects multiple lip gloss colors and sorts the lip gloss colors

Step 6B2: Perform brightness correction on each lip gloss color according to the true brightness value;

Step 6B3: The user chooses to perform self-service lip gloss blending or system lip gloss blending:

Step 6B3a: If the user selects self-service lip gloss blending:

The lip region is divided into several sub-regions, and the lip gloss colors are sent to the user in sequence according to the order in step 6B1. The user selects whether to blend each lip gloss color in each sub-region in turn; after all lip gloss colors are blended, a lip gloss effect picture is displayed;

Step 6B3b, if the user selects system lip gloss blending:

Obtain lip gloss blending style pictures online, and the user selects a lip gloss blending style;

The lip area is divided into several sub-areas, and the lip gloss is blended according to the lip gloss blending style and the lip gloss color selected by the user, and a lip gloss effect picture is displayed.

Furthermore, each time the lip gloss is blended, the following processing strategies are used:

Step a: Blend the lip color with the corrected lip gloss color for the first time:

color=lipColor.rgb*colorCorrection.rgb (5)

Among them, lipColor.rgb represents the lip color, colorCorrection.rgb represents the corrected lip color; color represents the color after the first blending; the output color includes three components: r, g, and b.

Step b: Blend the blended color with the lip color lipColor for a second time:

ret＝(lipColor*(1-lipstickColor.a)+color*lipstickColor.a) (6)

Among them, ret represents the second blending result, that is, the final blending color of the lip gloss, and lipstickColor.a represents the opacity of the lip gloss, and the value range of the lip gloss opacity is [0, 1].

The present invention can calculate the brightness mean and filter out the noise to obtain a real brightness value as the input of the subsequent ambient brightness, so that the lip color is more realistic and can truly simulate the actual makeup effect. Through the multi-layer blending solution, the naturalness of the lip color fusion is improved.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

Fig. 1 is an overall flow chart of the present invention;

FIG2 is an overall flow chart of the present invention;

FIG3 is an upper lip recognition diagram of the present invention;

FIG4 is a lower lip recognition diagram of the present invention;

FIG5 is a face recognition diagram of the present invention;

FIG6 is a first lip gloss blending diagram of the present invention;

FIG. 7 is a second lip gloss blending diagram of the present invention.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

As shown in FIG. 1 and FIG. 2 , the method of this embodiment may include:

Step 1: When a user takes an image, the image area is divided synchronously, the face is used as the recognition area, and the recognition area image, i.e., the face recognition image, is stored;

Specifically, since virtual makeup requires high accuracy in lip recognition, the third-party facial recognition library SenseAR is used for face and lip recognition. SenseAR can detect multiple faces and provide 64 key points of the lips, with accuracy that meets the requirements. As shown in Figures 3 and 4. At the same time, the facial frame needs to be recognized, as shown in Figure 5.

Step 2: Extract brightness features of each facial feature area in the image of the user's face recognition and calculate the brightness mean;

Specifically, for each frame of the camera image, first use the face frame obtained in the first step (if there are multiple faces, take the larger frame) to intercept the part inside the face frame of each frame as the input source for brightness detection. The purpose is to prevent the environmental pixels outside the face from affecting the brightness estimation (for example, a white or black background will cause a large error in the brightness estimation).

Then, the mean of each RGB channel is calculated for the intercepted face frame to obtain a three-element array (R, G, B).

Then convert the obtained RGB mean into a grayscale value using the following formula. This grayscale value is the brightness mean estimated by this scheme, and the range is [0,1].

Step 3: According to the brightness mean, remove the noise in the brightness mean and calculate the true brightness value;

Specifically, the brightness mean obtained in step 2 will have random noise due to the measurement error of the camera sensor, and this noise will affect the final effect. This solution filters the noise to obtain the true brightness value. Here, it is assumed that the noise is the most common Gaussian white noise, so the measured value satisfies the normal distribution. Since white noise has a zero mean, the mean of the normal distribution is the true brightness value that the solution will obtain. Due to the existence of noise, the normal distribution will have a certain variance. This solution uses a one-dimensional Kalman filter to filter out the noise and obtain the true brightness value.

The one-dimensional Kalman filter needs to define the following variables to filter out noise: process noise value q, measurement noise value (camera sensor noise) r, filter output value x, estimated error value p, and Kalman gain value k.

The noisy brightness mean obtained in step 2 is input into the one-dimensional Kalman filter and the true brightness value is output.

Since steps 2 and 3 require averaging and filtering of the image, which is time-consuming, and for situations with high real-time requirements, it is not possible to process every frame, this solution sets the brightness algorithm to an interval of 200ms. The filtered brightness mean will then be used to correct the color.

Step 4: Perform brightness correction on the image of the user's face recognition according to the true brightness value;

Specifically, brightness correction is performed on each area of the face recognition image.

Step 5: Divide the image of the user's face recognition into regions and extract the lip region as the subsequent operation region;

Step 6. The user selects the lip gloss color and chooses whether to blend the lip gloss. If not, each lip gloss color and lip area color will be blended to generate a lip gloss effect image. If lip gloss is blended, all selected lip gloss colors and lip area colors will be blended to generate a lip gloss effect image.

In this embodiment, on the Mac platform, the Qt Creator development tool is used, using C++ and OpenGL languages:

Specifically, the idea behind blending lip gloss in step 6 is as follows:

When blending with the lips, the first layer blends the lip gloss and lip texture according to the preset Multiply method, and then blends with the lip texture for the second time to obtain the final output value ret. After testing, after two fusions, the natural fusion degree of the lip gloss and lips is better. When blending the second layer, just repeat the previous steps and use ret as the input of the second layer, that is, lipstickColor. The ret material obtained after the second layer of blending is the actual desired lip gloss color.

After the lip gloss images that need to be mapped are made, two lip gloss images are made to show the effect of lip gloss base, as shown in Figure 6, as the base lip gloss, that is, the first layer of lip gloss, and as shown in Figure 7, as the surface lip gloss, that is, the second layer of lip gloss. The blending methods of the two layers can be different. The lip gloss resource files and blending methods are added through the script. Figure 6 shows the effect of using MultiplyBlend, and Figure 7 shows the effect of using Normal Blend.

Therefore, this solution dynamically creates a sampler and shader program according to the number of lip gloss layers. The output of each layer is used as the input of the next layer. At the same time, when mapping each layer, the texture needs to be color corrected using the brightness mean obtained in step 3. Here, the following variable values are defined: lipColor is the color of the lip texture of the face, lipstickColor is the color of the lip gloss texture, ambientLum is the brightness mean obtained in step 3, colorOffset is the color offset value, and colorCorrection is the corrected value.

This solution dynamically generates shaders according to the number of lip gloss layers and blending methods to achieve the effect of multi-layer lip gloss. The present invention can not only achieve basic makeup such as single-layer lip gloss, but also achieve multi-layer advanced makeup effects such as lip gloss primer and bitten lip makeup. After brightness correction of the lips under different ambient lights, the presented lip gloss color can be adaptively adjusted according to the brightness. At the same time, after applying multi-layer blending and double fusion technology, the lip gloss can show more layers, with better natural fusion, and more realistically show the actual makeup effect.

Preferably, step 2 performs brightness detection on the face recognition image and calculates the brightness mean:

measurement＝0.299*R+0.587*G+0.114*B (2)

Among them, R represents the red component in the sampled texture; G represents the green component; B represents the blue component; the value range of R, G, and B is [0,1]; measurement represents the brightness mean, and the value range of the brightness mean is [0,1].

Preferably, step 3 removes noise in the brightness mean to obtain the true brightness value, including the following steps:

Step 31, using the covariance prediction equation, calculate the estimated uncertainty in the brightness mean:

p＝p+q (3)

Among them, p is the estimation uncertainty, q is the process noise value, and the process noise value is set based on experience;

Specifically, the initial value of p is not very important. You can initialize a larger p value. With continuous iterations, the estimation error will continue to converge. Because the range of brightness is [0,1], you can just take p as 1. Similarly, the initial value of x is not important. It will continue to approach the true value as the program iterates. You can take x as the brightness mean of the first frame. The values of q and r are indeed important for estimating an accurate true value. After testing, when the value of q is 0.0005 and the value of r is 0.001, good filtering performance and effect can be obtained.

Step 312: Calculate the result gain in the brightness mean according to the estimated uncertainty:

k＝p/(p+r) (4)

Among them, k is the result gain, r is the camera sensor noise value, and the camera sensor noise value is set based on experience;

Step 32: Calculate the actual brightness value based on the result gain:

x＝k*measurement+(1-k)*x (5)

Among them, measurement is the mean brightness and x is the true brightness value.

Specifically, when the user sets the lip and face recognition to be performed multiple times during one use, the covariance update equation can be used to update the parameters and iterate, and the final output gradually converges and obtains the true value of the brightness. Preferably, in step 6, if the user chooses not to perform lip gloss blending, the following steps are performed:

Step 6A: The user selects one or more lip gloss colors, performs brightness correction on each lip gloss color, blends the corrected lip gloss color with the lip base color, and displays lip gloss effect pictures of the corresponding lip gloss colors respectively;

Preferably, if the user blends the lip gloss in step 6, the following steps are performed:

Step 6B1: The user selects multiple lip gloss colors and sorts the lip gloss colors

Step 6B2: Perform brightness correction on each lip gloss color according to the true brightness value;

Specifically, the lip gloss texture is sampled and lipstickLum is calculated.

ColorOffset uses polynomial fitting to calculate the coefficients. The specific formula is as follows:

colorOffset＝0.675*(-0.40862638*lipstickLum+0.81462644*ambientLum2+0.03340745) (1)

The generated colorOffset correction value needs to be corrected for lipstickerColor. After correction, in order to ensure the color is valid, it is limited to [0,1]. This method can be used to correct the brightness of the lipstick color with good ambient brightness.

By using the corrected texture for mapping, the color of the lip gloss can take into account the influence of ambient light. If the face is bright, the brightness of the lip gloss after makeup is brighter than the color of the original texture, and more realistic.

Step 6B3: The user chooses to perform self-service lip gloss blending or system lip gloss blending:

Step 6B3a: If the user selects self-service lip gloss blending:

The lip region is divided into several sub-regions, and the lip gloss colors are sent to the user in sequence according to the order in step 6B1. The user selects whether to blend each lip gloss color in each sub-region in turn; after all lip gloss colors are blended, a lip gloss effect picture is displayed;

Step 6B3b, if the user selects system lip gloss blending:

Obtain lip gloss blending style pictures online, and the user selects a lip gloss blending style;

The lip area is divided into several sub-areas, and the lip gloss is blended according to the lip gloss blending style and the lip gloss color selected by the user, and a lip gloss effect picture is displayed.

Specifically, the present invention can simulate a variety of lip makeup by blending lip glosses of different colors on different areas of the lips. Not only are the colors more realistic, but the makeup effects can also be adjusted according to user needs, increasing the practicality of the present invention in specific use.

Preferably, each time the lip gloss is blended, the following processing strategy is used:

Step a: Blend the lip color with the corrected lip gloss color for the first time:

color=lipColor.rgb*colorCorrection.rgb (6)

Among them, lipColor.rgb represents the lip color, colorCorrection.rgb represents the corrected lip color; color represents the color after the first blending; the output color includes three components: r, g, and b.

Step b: Blend the blended color with the lip color lipColor for a second time:

ret＝(lipColor*(1-lipstickColor.a)+color*lipstickColor.a) (7)

Among them, ret represents the second blending result, that is, the final blending color of the lip gloss, and lipstickColor.a represents the opacity of the lip gloss, and the value range of the lip gloss opacity is [0, 1].

Specifically, the lip color and the corrected lip gloss color are first blended according to the preset multiply method. The formula is different depending on the preset blending method. In this example, the formula is as follows:

color=lipColor.rgb*colorCorrection.rgb (8)

The output color includes three components: r, g, and b.

The blended color is blended with the lip color lipColor for the second time, and the blending weight is the alpha value in the lip gloss texture. The formula for this part is as follows:

ret＝(lipColor*(1-lipstickColor.a)+color*lipstickColor.a,1.0) (9)

The blending result ret includes four components: r, g, b, and a, where a is alpha, indicating opacity, which is fixed at 1.0.

This embodiment uses the third-party facial recognition library SenseAR to identify lip feature points. When applying virtual makeup, the brightness value of the facial range is detected and the lip color is corrected for ambient light. At the same time, by performing multi-layer blending operations on the lips, the effect of lip primer or lip biting makeup can be achieved, and the effect of natural fusion with the lips can be achieved.

Beneficial effects:

The present invention can calculate the brightness mean and filter out the noise to obtain a real brightness value as the input of the subsequent ambient brightness, so that the lip color is more realistic and can truly simulate the actual makeup effect. Through the multi-layer blending solution, not only simple basic makeup is supported, but also a variety of lip primer effects and advanced makeup effects such as bitten lip makeup can be supported by controlling the blending method, and the naturalness of the fusion is improved through the secondary blending operation.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims (6)

1. A method for applying simulated lip gloss, comprising the following steps: Step 1: When a user takes an image, the image area is divided synchronously, the face is used as the recognition area, and the recognition area image, i.e., the face recognition image, is stored; Step 2: Extract brightness features of each facial feature area in the image of the user's face recognition and calculate the brightness mean; Step 3: According to the brightness mean, remove the noise in the brightness mean and calculate the true brightness value; Step 4: Perform brightness correction on the image of the user's face recognition according to the true brightness value; Step 5: Divide the image of the user's face recognition into regions and extract the lip region as the subsequent operation region; Step 6. The user selects the lip gloss color and chooses whether to blend the lip gloss. If not, each lip gloss color and lip area color will be blended to generate a lip gloss effect image. If lip gloss is blended, all selected lip gloss colors and lip area colors will be blended to generate a lip gloss effect image. 2. The method for simulating lip gloss makeup according to claim 1, wherein the step 2 performs brightness detection on the face recognition image and calculates the brightness mean: measurement＝0.299*R+0.587*G+0.114*B (1) Among them, R represents the red component in the sampled texture; G represents the green component; B represents the blue component; the value range of R, G, and B is [0,1]; measurement represents the brightness mean, and the value range of the brightness mean is [0,1]. 3. The method for simulating lip gloss makeup according to claim 1, wherein the step 3 removes noise in the brightness mean value to obtain the true brightness value, comprising the following steps: Step 31, using the covariance prediction equation, calculate the estimated uncertainty in the brightness mean: p＝p+q (2) Among them, p is the estimation uncertainty, q is the process noise value, and the process noise value is set based on experience; Step 312: Calculate the result gain in the brightness mean according to the estimated uncertainty: k＝p/(p+r) (3) Among them, k is the result gain, r is the camera sensor noise value, and the camera sensor noise value is set based on experience; Step 32: Calculate the actual brightness value based on the result gain: x＝k*measurement+(1-k)*x (4) Among them, measurement is the mean brightness and x is the true brightness value. 4. The method for applying simulated lip gloss makeup according to claim 1, wherein if the user chooses not to blend the lip gloss in step 6, the following steps are performed: Step 6A: The user selects one or more lip gloss colors, performs brightness correction on each lip gloss color, blends the corrected lip gloss color with the lip base color, and displays lip gloss effect pictures of the corresponding lip gloss colors. 5. The method for applying simulated lip gloss makeup according to claim 1, wherein if the user blends the lip gloss in step 6, the following steps are performed: Step 6B1: The user selects multiple lip gloss colors and sorts the lip gloss colors Step 6B2: Perform brightness correction on each lip gloss color according to the true brightness value; Step 6B3: The user chooses to perform self-service lip gloss blending or system lip gloss blending: Step 6B3a: If the user selects self-service lip gloss blending: The lip region is divided into several sub-regions, and the lip gloss colors are sent to the user in sequence according to the order in step 6B1. The user selects whether to blend each lip gloss color in each sub-region in turn; after all lip gloss colors are blended, a lip gloss effect picture is displayed; Step 6B3b, if the user selects system lip gloss blending: Obtain lip gloss blending style pictures online, and the user selects a lip gloss blending style; The lip area is divided into several sub-areas, and the lip gloss is blended according to the lip gloss blending style and the lip gloss color selected by the user, and a lip gloss effect picture is displayed. 6. A method for applying simulated lip gloss makeup according to claim 5, characterized in that each time the lip gloss is blended, the following processing strategy is used: Step a: Blend the lip color with the corrected lip gloss color for the first time: color=lipColor.rgb*colorCorrection.rgb (5) Among them, lipColor.rgb represents the lip color, colorCorrection.rgb represents the corrected lip color; color represents the color after the first blending; the output color includes three components: r, g, and b; Step b: Blend the blended color with the lip color lipColor for a second time: ret＝(lipColor*(1-lipstickColor.a)+color*lipstickColor.a) (6) Among them, ret represents the second blending result, that is, the final blending color of the lip gloss, and lipstickColor.a represents the opacity of the lip gloss, and the value range of the lip gloss opacity is [0, 1].