JP3789911B2 - Image processing apparatus and face image processing apparatus - Google Patents

Description

  The present invention relates to a technique for observing skin color, particularly skin color. The present invention diagnoses a customer's skin condition by observing the customer's skin color, outputs the diagnosis result, and outputs the result when basic makeup is performed based on the diagnosis result. The present invention relates to a device for giving advice on whether to apply makeup.

  2. Description of the Related Art A skin color measuring device, a skin color diagnostic device, and a face image processing device for photographing a face image of a subject and performing advice on a beauty method and a makeup method based on an image of a skin region of the face image are known. In this apparatus, a surface reflected light component that becomes a noise component is removed using polarized light, and a pigment component such as a melanin component and a hemoglobin component is obtained for the internal light component (see, for example, Patent Document 1).

When performing skin color measurement or skin color diagnosis using such an apparatus disclosed in Patent Document 1, an image of a skin region in which hair, eyes, lips, and the like are removed from a subject's face image is extracted, and the extracted skin When a skin color change, such as when the area image was cared for with a whitening agent or when administered with a blood circulation promoter, is simulated, and this is displayed again on the face image, the subject visually The effect of care can be recognized.
JP 2002-200050 A

  As described above, when performing skin color measurement or skin color diagnosis using the apparatus disclosed in Patent Document 1, when extracting an image of a skin region from which hair, eyes, lips, and the like are removed from a face image of a subject, Therefore, it is necessary to generate a mask image corresponding to a region where this extraction is desired.

  When the color of an image desired to be extracted is determined, the mask image can be generated by designating an area including the color. For example, there is a technique called `` blue back photography '' in which a blue background is placed and an object other than blue to be extracted is placed in front of the background. In this technique, a mask image consisting of blue pixels placed in the background is used. , It is possible to easily extract an image of a subject that is not blue.

  In the case of a face image, even though it is said to be a skin color, it differs slightly for each subject, so it is difficult to determine the color uniformly. In addition, the face has irregularities, and even if the part including the shadow and the part not including the shadow are recognized as different skin colors even if they are the same skin color, it is not easy to process the shadow part. As described above, generation of a mask used for a face image requires a trial and error operation by an expert.

  On the other hand, skin color measurement or skin color diagnosis is performed while counseling with the subject, and it is desired that the process of extracting the skin region from the face image of the subject can be performed in real time in the process of counseling.

  The present invention has been performed against such a background, and provides an image processing apparatus or method capable of generating a mask image corresponding to a skin region of a subject's body without trial and error operations. Objective. An object of this invention is to provide the face image processing apparatus or method which can perform a skin color measurement or skin color diagnosis in real time based on a test subject's face image.

  The image processing apparatus of the present invention can process an image at high speed by a simple process of comparing a preset threshold value with each pixel value. For example, the processing using the interrelationship of a plurality of pixels such as comparing and averaging pixel values between pixels is not performed, so the processing can be speeded up. Therefore, if the image processing apparatus of the present invention is applied to a face image processing apparatus, it is possible to extract an image of a skin region from the face image of the subject in real time in the process of counseling between the subject and the counselor and to proceed with the counseling.

That is, the first aspect of the present invention is an image processing apparatus, and the feature of the present invention is that a pigment that constitutes a skin color from an internal light image excluding a surface reflected light component of a predetermined part of the subject skin surface Extract the melanin component and hemoglobin component from the components, generate an image using these components, convert the generated image into an image represented by the brightness value, and generate an image by reversing the brightness of the image The first means (procedure # 1) to extract the hemoglobin component from the pigment component constituting the skin color from the internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface, and generate an image using this component Then, a second means (procedure # 5) for converting the generated image into an image represented by a lightness value to generate an image, the image generated by the second means, and the first means In Ri is by synthesizing the generated images to place and a third means for generating (Procedure # 6) image.

  As described above, the first means generates a face image by using the melanin component and the hemoglobin component, so that pixels recognized as outside the skin color range such as hair and background can be easily excluded. Further, by using both the melanin component and the hemoglobin component, it is possible to extract the skin region with higher accuracy by using both pigment components than by extracting the skin region by only one pigment component. Further, by extracting the lips and the like that contain a particularly large amount of hemoglobin component by the second means, and combining the image generated by the first means and the image generated by the second means A mask image can be obtained.

The image processing apparatus of the present invention converts an internal light image excluding the surface reflected light component of a predetermined part of the subject skin surface into an image represented by a saturation value, and is generated by this image and the first means. A fourth means (procedure # 2) for generating an image by combining the image may be provided.

  Thereby, the part which is not skin color, such as the brown hair which has a color close | similar to a skin color, and the brownish eyeball, can be excluded. That is, since hair, eyes, and the like are generally low in saturation, when an image is generated based on the saturation value, it becomes easy to exclude hair, eyes, and the like.

The image processing apparatus of the present invention converts an internal light image obtained by removing a surface reflected light component of a predetermined part of the surface of the subject's skin into an image represented by a brightness value, and this image and an image generated by the fourth means And a fifth means (procedure # 3) for generating an image.

As a result, it is possible to obtain an image in which a “blurring” process is performed around the mask. In other words, “blurring” is to perform a weighted averaging process for the value of a pixel and the values of surrounding pixels for a certain pixel. In the present invention, such a relationship between pixels is not used. The feature is to speed up the processing. Using the internal light image excluding the surface reflected light component of the predetermined part of the subject skin surface where the lightness changes greatly in the whole skin color region of the image generated by the fourth means, particularly in the skin color boundary region, By synthesizing images generated by converting to lightness values, an image that has been subjected to the “blurring” process can be obtained.

In the image processing apparatus of the present invention, when the white pixel value is “1” and the black pixel value is “0”, the pixel values of the pixels constituting the image generated by the fifth means are A sixth means (procedure # 4) for generating an image by performing a predetermined calculation so as to be closer to “1” than the pixel value can be provided.

  This makes it possible to create a mask image in which the white portion is more equal to the skin color area.

  The image processing apparatus of the present invention is a seventh method for generating an image by combining the image generated by the second means (procedure # 5) and the image generated by the sixth means (procedure # 4). Means (procedure # 6).

  As a result, a better mask image can be obtained as compared with the case where the image generated by the second means and the image generated by the first means are combined to generate an image.

  According to a second aspect of the present invention, an image generated by the image processing apparatus of the present invention is used as a mask image, and a means for extracting an image of a part corresponding to the mask image from the face image of the subject, and this extracting means A face image processing apparatus comprising: means for displaying an image of an extracted part.

  According to the image processing apparatus of the present invention, a mask image for extracting a skin region from a face image can be created without trial and error operations. Thereby, when skin color measurement or skin color diagnosis is performed by counseling with a subject, a skin region can be extracted from a face image in real time and an image subjected to a predetermined process can be displayed.

The third aspect of the present invention is to extract a melanin component and a hemoglobin component from the pigment component constituting the skin color from the internal light image excluding the surface reflected light component of a predetermined part of the subject skin surface, and image the component by these components. A first step (procedure # 1) for generating and converting the generated image into an image represented by a brightness value, and further inverting the brightness of the image to generate an image; The hemoglobin component is extracted from the pigment component that constitutes the skin color from the internal light image excluding the surface reflection light component, and an image is generated by this component, and the generated image is converted into an image represented by the brightness value. The second step (procedure # 5) for generating an image and the image generated by the second step and the image generated by the first step are combined to generate an image. An image processing method characterized by performing the third step (Procedure # 6).

The image processing method of the present invention converts an internal light image excluding the surface reflected light component of a predetermined part of the subject skin surface into an image represented by a saturation value, and is generated by this image and the first means. A fourth step (procedure # 2) for generating an image by combining with the image can be executed.

The image processing method of the present invention converts an internal light image excluding the surface reflected light component of a predetermined part of the surface of the subject's skin into an image represented by a brightness value, and this image and the image generated by the fourth step And the fifth step (procedure # 3) for generating an image can be executed.

  In the image processing method of the present invention, when the white pixel value is set to “1” and the black pixel value is set to “0”, the pixel values of the pixels constituting the image generated by the fifth step are A sixth step (procedure # 4) for generating an image by performing a predetermined calculation so as to be closer to “1” than the pixel value can be executed.

  The image processing method of the present invention is a seventh method for generating an image by synthesizing the image generated in the second step (procedure # 5) and the image generated in the sixth step (procedure # 4). Step (procedure # 6) can be executed.

  According to a fourth aspect of the present invention, an image generated by the image processing method of the present invention is used as a mask image, and an image of a part corresponding to the mask image is extracted from the face image of the subject, and this extraction step includes And a step of displaying an image obtained by performing a predetermined process on the extracted part.

  According to a fifth aspect of the present invention, there is provided a program characterized in that, when installed in an information processing apparatus, the information processing apparatus realizes functions corresponding to the respective means of the image processing apparatus of the present invention. Alternatively, the program is characterized in that, when installed in the information processing apparatus, the information processing apparatus realizes equipment corresponding to each means of the face image processing apparatus of the present invention.

  By recording the program of the present invention on the recording medium of the present invention, the information processing apparatus can install the program of the present invention using this recording medium. Alternatively, the program of the present invention can be directly installed in the information processing apparatus via a network from a server holding the program of the present invention.

  Thus, using a general-purpose information processing device, a mask image corresponding to the skin region of the subject's body can be generated without trial and error operations, or skin color measurement or skin color diagnosis based on the face image of the subject Can be realized in real time.

  According to the present invention, an image processing apparatus capable of automatically generating a mask image corresponding to the skin area of the subject's body, or face image processing capable of performing skin color measurement or skin color diagnosis in real time based on the face image of the subject. An apparatus can be realized.

  The face image processing apparatus of this embodiment will be described with reference to FIG. As shown in FIG. 1, the face image processing apparatus according to the present embodiment uses the image generated by the mask generation unit 3 as a mask image, and extracts an image of a part corresponding to the mask image from the face image of the subject. Unit 4 and a display unit 5 for displaying an image of a part extracted by the skin color processing unit 4.

  The polarizing plates 50, 51, 52, the camera device 1, and the analysis unit 2 are described in Patent Document 1. Briefly described here, when the light whose polarization plane is adjusted by passing through the polarizing plate 50 passes through the surface of the skin and repeats scattering inside and out several times, the polarizing plate The light arranged at 50 is internally reflected to be emitted as light having various polarizations, that is, internal light.

  In addition, when the light whose polarization plane is adjusted by being transmitted through the polarizing plate 50 is reflected on the surface of the skin, the polarization direction adjusted by the polarizing plate 50 is preserved. For example, light that has been adjusted to vertically polarized light is reflected on the surface by the polarizing plate 50, so that it is reflected again as light having longitudinally polarized light, that is, surface light.

In this way, using the property that the surface light and the internal light can be distinguished by polarized light, the images that are transmitted through the polarizing plates 51 and 52 are recorded by the camera device 1. An image taken through the polarizing plate 51 is taken as an image a. An image taken through the polarizing plate 52 is taken as an image b. Here, when you want to get an internal light image,
2 x image b
As shown in FIG. That is, since the image b includes only the internal light element, all the light captured as the image b becomes the internal light image as it is. However, since about half of the internal light is shielded by the polarizing plate 52, the light amount of the image b is doubled for the purpose of compensating for the light shielding. This 2 × image b corresponds to the internal light image to be analyzed.

  In this way, the analysis unit 2 analyzes the melanin component and the hemoglobin component with respect to the internal light component from which the skin surface reflection component that becomes the noise component is removed.

  Below, the mask production | generation part 3 which is the characteristics of this invention is demonstrated in detail.

(Procedure # 1)
In the procedure # 1, first, an image of a pigment component of a melanin component and a hemoglobin component is generated from a face image, and after this image is converted into a brightness image, a mask for the face portion is extracted using a threshold value. In the image generated in this manner, pixels that are recognized as outside the skin color range, such as hair and background, are excluded, and a threshold can be set without trial and error operations. In addition, a melanin component and a hemoglobin component are used in combination instead of a single pigment component image. It is more accurate to use both pigment components than to extract a skin region with only one pigment component. This is because it can be extracted.

A flowchart of procedure # 1 is shown in FIG. RGB (red, green, blue) of an image composed of a melanin component and a hemoglobin component is converted from the original image, which is an internal light image obtained by removing the surface reflected light component, from the face image of the subject, and the brightness inversion processing is performed. The pixel value in the image generated in this way is set to MH (step 1). Subsequently, MH is compared with a predetermined threshold value range (step 2).

When MH is within the threshold value range, it is used as the skin region.
MHmin <MH <MHmax
The pixel value is set to “1” (white) (step 3). When MH deviates from the upper limit or lower limit of the threshold value to the allowable value (tol) or more, it is assumed that it is other than the skin area,
MH ≦ MHmin-tol or MH ≧ MHmax + tol
The pixel value is set to “0” (black) (step 4). When MH is outside the threshold value range but within the allowable value,
MHmax + tol> MH ≧ MHmax
If so, the pixel value is set to 1-((MH-MHmax) / tol) as a familiar area around the skin.
(Step 5). Here, in order to reduce the unnaturalness of the image, the “familiarization region” is a boundary where different colors are adjacent to each other, rather than suddenly changing to a different color after passing the boundary. The method of changing to color is used, but it refers to the area used in that method. Also,
MHmin ≧ MH> MHmin-tol
If so, the pixel value is 1-((MHmin-MH) / tol) as a familiar area around the skin.
(Step 6). Note that the range of values taken by the pixel values in steps 5 and 6 is between 0 and 1. Thereby, the mask # 1 is obtained by the procedure # 1.

(Procedure # 2)
In procedure # 2, a portion that is close to the skin color but not the skin is removed. For example, pixels such as brown hair, brown eyes, and brown clothes are removed. Focusing on the fact that brown is generally low in saturation, as shown in FIG. 3, an image S obtained by converting the saturation value from RGB of the original image is generated (step 1). The mask # 1 created in the procedure # 1 is multiplied by the image S (step 2). Thereby, mask # 2 is generated.

(Procedure # 3)
In normal mask creation, in order to reduce the uncomfortable feeling of compositing, a “blurring” process is performed around the mask. In general, a “blurring” process is a weighting between a pixel and the values of surrounding pixels. An averaging process is performed. That is, when performing a general “blur” process, the pixel value cannot be calculated independently of the surrounding pixels, and is not suitable for the present embodiment. Therefore, in the present embodiment, as shown in FIG. 4, an image V obtained by converting the brightness value of the original image is generated (step 1). The mask # 2 created in the procedure # 2 is multiplied by the image V (step 2). As a result, a mask # 3 having the same effect as “blur” is generated.

(Procedure # 4)
In the mask created up to procedure # 3, the range of values is gradually narrowed by multiplying by a decimal value. In procedure # 4, this range is expanded again. Variables used to extend the range are predetermined and given to the system, as are other thresholds. That is, if the pixel value of the mask generated in procedure # 4 is MaskMH, the pixel value of the mask generated in procedure # 3 is Mask # 3, and the default parameter is Sscale, as shown in FIG.
MaskMH = Mask # 3 * Sscale
And However, if MaskMH exceeds the value of “1”, it is rounded to “1”. The “predetermined operation” in claim 4 or 10 is, for example, the operation of procedure # 4. Thereby, an image with clear distinction between black and white close to the pixel value of the mask image that should be originally obtained can be obtained.

(Procedure # 5)
By procedure # 4, hair, eyes, etc. could be excluded. However, lips and the like have not been excluded yet. In procedure # 5, using the hemoglobin pigment component image, lips, redness, and the like, which are portions having a particularly large hemoglobin component, are extracted from the face. That is, as shown in FIG. 6, the RGB values of the hemoglobin component image are converted to lightness values (step 1). As a result, an image having a lightness image is obtained. Let H be the pixel value of a lightness image. This H is compared with a predetermined threshold value range (step 2), and H is smaller than the threshold. Hthresh> H
In this case, the pixel value is set to “0” (black) because it is a lip and red region (step 3). Also, H deviates from the threshold value beyond the allowable value. H ≧ Hthresh + tol
In this case, the pixel value is set to “1” (white) because it is outside the lip and red area (step 4). Further, H is equal to or greater than a threshold value, but is within an allowable value. Hthresh + tol > H ≧ Hthresh
In this case, since it is a familiar area around the lips, the pixel value is set to (H-Hthresh) / tol.
(Step 5). Note that the range of the pixel value in step 5 is between 0 and 1. Thereby, MaskH by the procedure # 5 is generated.

(Procedure # 6)
As the final procedure, as shown in FIG. 7, the final mask FinalMask is generated by multiplying the mask MaskMH created in procedure # 4 and the mask MaskH created in procedure # 5.

(Summary)
FIG. 8 shows a flowchart in which procedure # 1 to procedure # 6 are combined into one. In the flowchart of FIG. 8, the procedure # 2, the procedure # 3, and the procedure # 4 can be omitted, and the mask image can be generated by the procedure # 1, the procedure # 5, and the procedure # 6. According to this, as compared with the mask generation through all procedures, the quality as a mask image is naturally reduced, but the processing procedure can be shortened, which is advantageous in performing mask generation at high speed.

  Using the mask image generated in this way, the skin color processing unit 4 shown in FIG. 1 displays an image obtained by extracting a skin region from the face image and applying a predetermined process to the skin region on the display unit 5. The “predetermined process” as defined in claim 6 or 12 is, for example, a process of artificially changing the melanin / hemoglobin component described in Patent Document 1. The counselor can perform counseling with reference to the skin area displayed on the display unit 5. Further, as an additional function of the face image processing device, a function of converting the skin color into a desired color, a function of cutting out and displaying a predetermined part of the skin region, and a function of displaying the cut out predetermined part in an enlarged manner may be added. it can. The cutout and enlarged display of the predetermined part can be performed also on an image having only a melanin component and a hemoglobin component.

  Further, if a function for generating a mask image in real time in the face image processing apparatus of the present embodiment is used, it is possible to extract a skin region as a moving image from a face image as a moving image.

  The present invention can be implemented as a program that, when installed in a general-purpose information processing apparatus, causes the information processing apparatus to realize functions corresponding to the face image processing apparatus of the present invention. The program is recorded on a recording medium and installed in the information processing apparatus, or installed in the information processing apparatus via a communication line, so that the information processing apparatus includes an analysis unit 2, a mask generation unit 3, and a skin color processing unit. 4 can be realized.

  The present invention can provide an image processing apparatus or method that can automatically generate a mask image corresponding to a skin region of a subject's body. The present invention can provide a face image processing apparatus or method capable of performing skin color measurement or skin color diagnosis in real time based on a face image of a subject.

The block diagram of the face image processing apparatus of this embodiment. Flowchart of procedure # 1. Flowchart of procedure # 2. Flowchart of procedure # 3. Flowchart of procedure # 4. Flowchart of procedure # 5. Flowchart of procedure # 6. The flowchart which put together procedure # 1- # 6.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera apparatus 2 Analysis part 3 Mask generation part 4 Skin color processing part 5 Display part 10 Face image processing apparatus 50, 51, 52 Polarizing plate

Claims (14)

The melanin component and hemoglobin component are extracted from the pigment components constituting the skin color from the internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface, and an image is generated with these components, and the generated image is A first means for generating an image by converting to an image represented by a brightness value and further inverting the brightness of the image;
A hemoglobin component is extracted from the pigment components constituting the skin color from the internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface, and an image is generated by this component, and the generated image is represented by the brightness value. A second means for generating an image by converting into an image to be generated;
An image processing apparatus comprising: third means for generating an image by synthesizing the image generated by the second means and the image generated by the first means. The internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface is converted into an image represented by a saturation value, and this image is combined with the image generated by the first means to obtain an image. The image processing apparatus according to claim 1, further comprising a fourth means for generating. The internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface is converted into an image represented by a brightness value, and this image and the image generated by the fourth means are combined to generate an image The image processing apparatus according to claim 2, further comprising a fifth means. When the white pixel value is “1” and the black pixel value is “0”,
6. A sixth means for generating an image by performing a predetermined calculation so that a pixel value of a pixel constituting the image generated by the fifth means is closer to “1” than the pixel value. 3. The image processing apparatus according to 3.   5. The image processing according to claim 1, further comprising seventh means for generating an image by synthesizing the image generated by the second means and the image generated by the sixth means. 6. apparatus. Means for extracting, as a mask image, an image generated by the image processing apparatus according to claim 1 or 5, and extracting an image of a portion corresponding to the mask image from the face image of the subject;
A face image processing apparatus comprising: means for displaying an image obtained by performing predetermined processing on a part extracted by the extracting means. The melanin component and hemoglobin component are extracted from the pigment components constituting the skin color from the internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface, and an image is generated with these components, and the generated image is A first step of generating an image by converting to an image represented by a brightness value, and further inverting the brightness of the image;
A hemoglobin component is extracted from the pigment components constituting the skin color from the internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface, and an image is generated by this component, and the generated image is represented by the brightness value. A second step of converting the generated image into an image and generating the image;
An image processing method comprising: executing a third step of generating an image by combining the image generated by the second step and the image generated by the first step. The internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface is converted into an image represented by a saturation value, and this image is combined with the image generated by the first step to obtain an image. The image processing method according to claim 7, wherein the fourth step of generating is executed. The internal light image excluding the surface reflected light component of the predetermined part of the subject's skin surface is converted into an image represented by a brightness value, and this image and the image generated by the fourth step are combined to generate an image The image processing method according to claim 8, wherein the fifth step is executed. When the white pixel value is “1” and the black pixel value is “0”,
6. The sixth step of generating an image by performing a predetermined calculation so that the pixel value of a pixel constituting the image generated in the fifth step is closer to “1” than the pixel value is performed. 10. The image processing method according to 9.   The image processing according to any one of claims 7 to 10, wherein a seventh step of generating an image by combining the image generated by the second step and the image generated by the sixth step is executed. Method. Extracting the image generated by the image processing method according to claim 7 or 11 as a mask image, and extracting an image of a part corresponding to the mask image from the face image of the subject;
And a step of displaying an image obtained by performing a predetermined process on the part extracted by the extracting step.   A program that, when installed in an information processing apparatus, causes the information processing apparatus to realize functions corresponding to the respective means of the image processing apparatus according to any one of claims 1 to 5.   A program that, when installed in an information processing apparatus, causes the information processing apparatus to realize functions corresponding to the respective means of the face image processing apparatus according to claim 6.

Images