JPH11168620A - Image processing method, input device and record medium thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily select colors by deciding candidates for a color to be change corresponding to a display image, selecting the color to be changed among the color candidates and changing a decided color change area to the selected color. SOLUTION: A user selects and input threshold TH-edge that detects a flesh- colored area by using a scroll bar B7. A computer detects a flesh colored area that does not include eyes and a mouth by using the threshold. The computer shows the detected flesh colored area. The user selects and inputs a ratio which mixes it with a color before conversion by using a scroll bar B8. Next, color to be changed is selected and inputted, a scroll bar B9 is used as a means which selects and inputs the color to be changed. The bar B9 shows candidates for colors to be changed. The computer performs processing which changes a decided color change area to a selected color, e.g., performs charge in flesh- colored.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention displays, for example, a person or landscape photographed by a digital camera on a display and changes the skin color of the face, the color of the sky, the green color of the plant, etc. to a desired color. It relates to the technology to be performed.

[0002]

2. Description of the Related Art A digital camera is used to photograph a person or a landscape and display it on a display. However, in actual photographing, for example, the exposure light may be natural light or light from a fluorescent lamp. The color of the subject is not always the color desired by the user. Therefore, there is a demand to change the color of the subject to a desired color.

[0003]

Here, taking a case where a person's face is photographed as an example, in order to convert the skin color to a desired color, a skin color region is first extracted, and a candidate skin color to be converted is obtained. Is displayed, and a desired skin color is selected and converted from the display. As for changing the skin color, a technique for automatically detecting the skin color from a human image has been studied, as described in Japanese Patent Application Laid-Open No. 8-153197. However, the skin color of the face differs depending on the scene being photographed, and even in the same face, the skin color may be significantly different between the left and right due to the relationship between light and shade, so it is difficult to improve the extraction accuracy of the skin color region, and the photograph was taken. Complicated calculations were required, for example, depending on the color of the face, fine segmentation was required.

[0004] Further, a color palette displaying skin color candidates is displayed on the display together with the photographed face, and a user designates a desired color from a plurality of displayed colors. There are also conventional techniques for changing skin color. However, in general, unnecessary colors are often included in the color palette, so that there is a problem that it is difficult for the user to select. On the other hand, if the number of color candidates is reduced to facilitate selection, a desired color may not be selected. Furthermore, in the step of selecting and converting a desired skin color, the color before conversion is often mixed at a certain ratio, but there is a problem that the user cannot arbitrarily select this ratio.

Further, according to the prior art, a user often displays a preview image on a dialog box displayed on a display and adjusts parameters while viewing the image. Since the operation is performed on a single dialog box, there are also problems that the number of parameters increases and the usability of the user deteriorates.

An object of the present invention is to provide an image processing method for easily and naturally converting the color of a visualized image to a desired color when the image information is visualized.

[0007]

In order to achieve the above object, an image processing method according to the present invention comprises the steps of: inputting a parameter for specifying a color change area for changing a color to a display image; A step of determining a color change area based on the parameter; a step of determining a color candidate to be changed according to the display image; and a step of selecting a color to be changed from the determined color candidates. And changing the determined color change area to the selected color.

[0008]

According to the image processing method of the present invention, a step of inputting a parameter for specifying a color change area for changing a color in a display image, and a step of determining a color change area based on the parameter Determining a color candidate to be changed according to the display image; selecting a color to be changed from the determined color candidates; and selecting the determined color change area. For example, the user can select the color to be changed from the limited colors, thereby facilitating the selection of the color.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, a first embodiment according to the present invention will be described with reference to the drawings. FIG. 1 is a flowchart illustrating an image processing method for detecting a skin color of a face displayed on a display of a computer and converting the color to a desired skin color. FIG. 2 is a perspective view showing a computer from which a program for executing the image processing method is read.

In FIG. 2, image information of a person photographed by a digital camera (not shown) is stored in a memory built in the computer 1, and based on the image information, a face F of the person is displayed on a display means. A display 1a
Is displayed in. A first dialog box D1 including six scroll bars B1 to B6 is displayed around the displayed face F, and allows a user operation to be described later. That is, the buttons of the scroll bars B1 to B6 can be moved up and down or left and right by the mouse 1b. The floppy disk FD serving as a recording medium stores a program for executing an image processing method described later.
The computer 1 reads the program of
Image processing of changing the skin color becomes possible. Note that such a program may be stored on a CD, a ROM, or the like. Hereinafter, this image processing will be described.

In steps S101 to S108 of FIG. 1, a color change area for changing a color is determined. In step S101, the user first selects a face area. More specifically, in FIG. 2, a rectangular area Af surrounding a face F whose color is to be converted is designated. This designation can be made, for example, by the user dragging the mouse 1b.

In the following, steps S102 to S104 show a case where a part of the face is designated to determine the color change area. In step S102, first, the computer 1
Displays a first dialog box D1 (FIG. 2). Since the positions and areas of the plurality of rectangular areas A1 to A3 can be changed on the display 1a, the user moves these rectangular areas A1 to A3 so as to surround the eyes E and the mouth M of the face F. , Eyes E, mouth M
Is selected (step S103). Please note that
The rectangular areas A1 and A2 surrounding the eye E can be moved left and right by the scroll bar B1 in FIG. 2, the rectangular areas A1 and A2 can be moved up and down by the scroll bar B2, and the rectangular areas A1 and A2 can be moved by the scroll bar B3.
Area can be changed. The rectangular areas A1, A
2 may be configured so that its position and area can be changed independently.

On the other hand, the user can move the rectangular area A3 surrounding the mouth M right and left by the scroll bar B4,
The rectangular area A3 can be moved up and down by the scroll bar B5, and the area of the rectangular area A3 can be changed by the scroll bar B6.

In step S104, the computer 1 detects the eyes E and the mouth M. The manner in which the eyes E and the mouth M are detected in the selected area will be described below. FIG.
Is a flowchart showing a subroutine for detecting an eye E and a mouth M.

First, the computer 1 detects an edge pixel in the area selected in step S103 (FIG. 1) (step S104a). Specifically, first, a linear sum X _i , _j of the RGB values of the target pixel is obtained. here,
i and j are vertical and horizontal positions in the image. This linear sum is the eye,
Each mouth has a unique value, and in the case of eye detection,
For example, X _i , _j = 0.3 * R _i , _j + 0.59 * G _i , _j +
It is calculated with 0.11 * B _i , _j , and in the case of mouth detection, for example, X _i , _j = 0.21 * R _i , _j− 0.52 * G _i , _j +
Calculated as 0.31 * B _i , _j .

The computer 1 obtains a difference between a certain pixel and its surrounding four pixels, and obtains edge amounts E _i , _j . E
_i and _j are represented by the following equations. E _i , _j = | X _{i + 1} , _j −X _i , _j | + | X _i−1 , _j −X _i , _{j
| + | X i, j +} 1 -X i, j | + | X i, j-1 -X i, j |

Further, the computer 1 calculates an average Ave_edge of the edge amount in the target area. If E _i , _j is larger than Ave_edge in the target area, it is determined that the pixel is an edge pixel.

Next, in step S104b, the computer 1 labels the detected edge pixel in the vicinity of the upper, lower, left, and right of the detected edge pixel. More specifically, if there are edge pixels near the periphery 8 of the edge pixel, the same label is determined, and Ai (i = label) is obtained for the area of each label.

The computer 1 further performs step S104
In c, the area to which the label having the largest area is allocated in each area is determined as an eye and mouth area. If the positions of the eyes or the mouth are determined, the outline of the face is statistically estimated to some extent, and the accuracy of detecting the skin color region can be improved by using the estimated outline of the face. The contour of the face may be estimated by determining the position of the eyebrows.

Steps S105 to S107 are steps for changing the color change area obtained in steps S102 to S104. Returning to the flowchart of FIG. 1 again, the computer 1 displays a second dialog box D2 in step S105. FIG. 4 is a diagram showing a second dialog box D2 displayed on the display. In the display 1a, the second as shown in FIG.
Along with the dialog box D2, the target face image is displayed as the preview image Pv. While viewing the preview image Pv, the user displays, in the form of scroll bars B7 to B9, a threshold for detecting a flesh color area, a mixing ratio with the color before conversion, and a candidate for a color to be converted. Each item is changed using this scroll bar. More specifically, the threshold value for detecting the skin color area can be changed by the scroll bar B7 in FIG. 4, the mixing ratio with the color before the change can be changed by the scroll bar B8, and the desired skin color can be changed by the scroll bar B9. You can change the color tone.

First, in step S106, the user uses the scroll bar B7 to select and enter a threshold value TH_edge for detecting a flesh color area. By using this threshold, the computer 1 detects a flesh-colored area that does not include the eyes and the mouth (step S107). The mode is described below.

First, the linear sum X _i , _j of the face area is obtained. Here, i and j are vertical and horizontal positions in the image. This linear sum is calculated by, for example, X _i , _j = R _i , _j + B _i , _j + G _i , _j .

The difference between a certain pixel and its surrounding four pixels is obtained,
The edge amounts E _i and _j are obtained. E _i and _j are represented by the following equations. E _i , _j = | X _{i + 1} , _j −X _i , _j | + | X _i−1 , _j −X _i , _{j
| + | X i, j +} 1 -X i, j | + | X i, j-1 -X i, j |

The computer 1 determines that E _i ,
_{If j} is larger than the input threshold TH_edge and is not the eye or mouth area detected in step S104, it is determined that the area corresponding to the pixel is a skin color area.

Further, in step S108, the computer 1 displays the detected skin color area. More specifically, in the skin color region detected in step S107, the color of the boundary region pixel with the region other than the skin color region is displayed as white (or black) on the preview image Pv of the second dialog box D2 (FIG. 4). The user looks at the boundary area and determines whether the skin color area has been appropriately selected (step S109). If it is determined that the selection is not appropriate, the process returns to step S106, and the user changes the threshold value TH_edge to be input, whereby the skin color area is detected again. If it is determined that the area is appropriate, the user clicks the OK button to determine the area. By the above processing, the area for changing the color is determined.

In step S110, the user uses the scroll bar B8 to select and input a ratio to be mixed with the color before conversion. Regarding the detected skin color area, the RGB values before the change are Rb, Gb, and Bb, respectively, and the favorite skin colors selected by the operator are Ro, Go, and Bo.
Assuming that the input mixing ratio is Rate and the converted RGB values are Ra, Ga, and Ba, the conversion is performed according to the following equation. Ra = Rate * Rb + (1-Rate) * Ro Ga = Rate * Gb + (1-Rate) * Go Ba = Rate * Bb + (1-Rate) * Bo

The value of Rate is a number between 0 and 1, and as this value approaches 1, the amount of change from the original color increases. In general, if the size is too large, the boundary between a region where the color is changed and a region where the color is not changed is conspicuous and unnatural.

In step S111, a color to be changed is selected and input. The scroll bar B9 is used as means for selecting and inputting the color to be changed. Scroll bar B
9 displays a candidate for a color to be changed. Here, the distribution of the flesh color of the Caucasian and yellow races on the u * v * space of CIEL * u * v is as shown in FIG. 5, for example, expressed by the following linear expression. Is done. v * = 0.715 * u * -5.458 While viewing the preview image Pv (FIG. 4), the user can operate the scroll bar B9 to adjust the value of u *. Another one showing the distribution of black skin
By using the following equation, the target of the skin color change can be black.

The value of L *, which is a component indicating the lightness of a color, is calculated by the following equation, for example, where R before the change is Rb. L * = 100 * (Rb / 255) The L * u * v * determined in this way is converted to an RGB value (Ro, Go, B
o).

In step S112, the computer 1 performs a process of changing the determined color change area to the selected color, in this example, changing the skin color. By setting the above conditions, the skin color of the preview image Pv is immediately changed, so that the user can check the preview image Pv to determine whether the change is appropriate (step S113). If the change is appropriate, the flow is terminated by inputting OK information. Since the change is inappropriate, if the user wants to change the condition, the process returns to step S110.

According to the first embodiment described above, the boundary of the skin color can be easily changed by one-dimensional processing, and the mixing ratio of the changed color to the original skin color can be easily changed. it can. Furthermore, since the color to be changed is limited to a single skin color, there is an advantage that image processing is simplified. Therefore, the skin color of the face can be accurately detected irrespective of the scene, and a desired skin color can be selected by a simple operation and converted naturally.

In the first embodiment, the user selects and inputs one-dimensional continuous skin color candidates by using the scroll bar. However, according to another embodiment, For example, "heavy makeup", "tan",
A candidate for a color similar to a skin color such as "baby" can be determined in advance, and the user can select one of them to instantaneously set a skin color similar to them.

Specifically, for example, when "baby" is selected, its representative values u * = 37, v * = 2
0 and RGB in the same manner as in step S111.
Convert to a value. U * = 42, v * for "Wheaty"
= 28, "heavy makeup", u * = 20, v * = 0. In addition, as the user interface, as shown in FIG. 4, a mode in which input is performed by selecting a radio button B10 on the second dialog box D2 is also conceivable.

Next, a second embodiment will be described with reference to the drawings. FIG. 6 is a flowchart illustrating an image processing method for detecting a sky color included in the scenery displayed on the display 1a and converting the color to a desired color. Each step will be described.

First, the computer 1 displays a first dialog box D1 (step S201), and the user selects a part of the sky (step S202). First dialog box D displayed as shown in FIG.
In 1, the position and area of the rectangular area As can be changed using the scroll bars B11 to B13. Therefore, the user surrounds a part of the sky to be changed in the rectangular area As. Note that the rectangular area As can be moved right and left by the scroll bar B11, and the rectangular area As can be moved up and down by the scroll bar B12.
The area of the rectangular area As can be changed by the scroll bar B13. Alternatively, a configuration in which a rectangular area is moved by dragging may be adopted.

Further, the computer 1 displays a second dialog box D2 as shown in FIG.
203) Display the target image as a preview image, and allow the user to input a threshold for detecting an empty area, a mixing ratio with a color before conversion, and a candidate for a color to be converted.

First, the computer 1 detects an empty area by using the scroll bar B14 to select and input a threshold for detecting an empty area (step S204). The method will be described below.

The linear sum X _i , _j of the pixels of the selected rectangular area As is obtained. Here, i and j are vertical and horizontal positions in the image. This linear sum is, for example, X _i , _j = R _i , _j +
G _{i, j} + B _i, are computed by _j.

Average value Ave_sk of X _i , _j in the above area
Find y. Ave_s for the entire target image area
ky is determined, and if the absolute value of this value is smaller than a threshold value for detecting an empty region, the pixel is determined as a candidate for an empty region. If there are empty region candidate pixels near the periphery 8 of the empty region candidate pixel, the same label is determined.

In step S205, the computer 1 displays the detected sky color area. More specifically,
In the sky color region detected as described above, the color of the boundary region pixel with the non-sky color region is set to white (or black), and the preview image P of the second dialog box D2 is set.
v (FIG. 8). The user looks at the boundary area and determines whether the sky color area is appropriately selected (step S206). If it is determined that the selection has not been made appropriately, the process returns to step S204, and the sky color area is detected again.

In step S207, the user uses the scroll bar B15 to select and input a ratio to be mixed with the color before the change. Regarding the detected sky color area, Rb, Gb, and Bb are used as RGB values before conversion, respectively, the sky color of preference selected by the operator is set as Ro, Go, Bo, and the input mixing ratio is Rate, and conversion is performed. RG after
Assuming that the B value is Ra, Ga, Ba, conversion is performed according to the following equation. Ra = Rate * Rb + (1-Rate) * Ro Ga = Rate * Gb + (1-Rate) * Go Ba = Rate * Bb + (1-Rate) * Bo

The value of Rate is a number between 0 and 1, and as this value approaches 1, the amount of change from the original color increases. In general, if the size is too large, the boundary between the converted area and the non-converted area becomes conspicuous and unnatural.

In step S208, the user selects and inputs a color to be converted. The scroll bar B16 is used as a means for selecting and inputting a color to be changed. Here, when the distribution of the blue sky is represented on the u * v * space of CIEL * u * v, it is as shown in FIG. 9 and is represented by, for example, a linear expression as shown in the following expression. v * = − 40−30 ≦ u * ≦ −10 The user operates the scroll bar B16 while watching the preview image Pv (FIG. 8), thereby adjusting the value of u *.

Assuming that the B value before change is Bb, the value of L *, which is a component indicating the color brightness, is calculated by, for example, the following equation. L * = 100 * (Bb / 255) The L * u * v * determined in this way is converted into an RGB value (Ro, Go, B
o).

In step S209, the computer 1 changes the color of the sky. By setting the above conditions, the skin color of the preview image Pv is immediately changed, so that the user can check the preview image Pv to determine whether the change is appropriate (step S21).
0). If the change is appropriate, the flow ends. If the change is inappropriate, and the user wants to change the condition, the process proceeds to step S2.
Return to 07.

Also in the second embodiment described above,
By one-dimensional processing, the boundary of the sky color can be easily changed, and the mixing ratio of the changed color to the original sky color can be easily changed. Further, since the color to be changed is limited to only the sky color, there is an advantage that image processing is simplified. Therefore, the color of the sky can be accurately detected regardless of the scene, and a desired color can be selected by a simple operation and converted naturally.

In the second embodiment, the user selects and inputs one-dimensional continuous sky color candidates using the scroll bar. However, according to another embodiment, For example, in advance, color candidates similar to the sky color such as "sunset" and "cloudy sky" are determined in advance, and the user can select any one of them and instantly set the sky color similar to them. You can also

More specifically, when "sunset" is selected, for example, the representative values u * = 15 and v * = 30 are adopted and converted to RGB values by the same method as in step S207. . In the case of "cloudy sky", u * =-10, v * =
-4 is adopted. In addition, as the user interface, it is conceivable to input using a radio button displayed on the second dialog box D2.

Further, if a program for executing the image processing method according to the present embodiment is incorporated in a digital camera with a liquid crystal screen in advance, the image processing can be performed through the liquid crystal screen.

[0050]

According to the image processing method of the present invention, a step of inputting a parameter for specifying a color change area for changing a color in a display image, and determining the color change area based on the parameter And determining a candidate color to be changed according to the display image,
The method includes a step of selecting a color to be changed from among the determined color candidates and a step of changing the determined color change area to the selected color. Can be selected from among the colors to be changed, thereby facilitating the selection of the color.

[Brief description of the drawings]

FIG. 1 is a flowchart illustrating an image processing method for detecting a skin color of a face displayed on a display and converting the skin color to a desired skin color.

FIG. 2 is a perspective view showing a computer from which a program for executing an image processing method according to the present invention has been read.

FIG. 3 is a flowchart showing a subroutine for detecting an eye E and a mouth M;

FIG. 4 is a diagram showing a second dialog box displayed on a display.

FIG. 5 shows the distribution of skin color of Caucasian and yellow races by CIEL * u *.
6 is a graph showing v in u * v * space.

FIG. 6 is a flowchart illustrating an image processing method for detecting a sky color included in a landscape displayed on a display and converting the color to a desired color.

FIG. 7 is a diagram showing a first dialog box.

FIG. 8 is a diagram showing a second dialog box.

FIG. 9 is a graph showing a blue sky distribution on a u * v * space of CIEL * u * v.

[Description of Signs] 1 Computer 1a Display 1b Mouse B1 to B9, B11 to B16 Scroll bar B10 Radio button D1 First dialog box D2 Second dialog box FD ‥‥ Floppy

Claims (16)

[Claims] 1. A step of inputting a parameter for specifying a color change area for changing a color to a display image; a step of determining a color change area based on the parameter; Determining a color candidate to be changed; selecting a color to be changed from the determined color candidates; and changing the determined color change area to the selected color. An image processing method comprising: 2. The method according to claim 1, further comprising the step of selecting a type of a display image, and the step of determining a candidate for a color to be changed according to the display image, 2. The image processing method according to claim 1, wherein candidates are determined. 3. The image processing method according to claim 1, wherein, in the step of displaying color candidates to be changed, the color candidates to be displayed are determined using only one color information as a variable. . 4. The image processing method according to claim 1, wherein the parameter includes position information in a color change area. 5. The image processing method according to claim 4, wherein the parameter includes threshold information for specifying a color change area. 6. The method according to claim 1, wherein the step of selecting a color from the candidates includes displaying a candidate for a mixture ratio of the color before the change and the selected color in the color change area. Item 6. The image processing method according to any one of Items 1 to 5. 7. Displaying the image on a display means, and, in the step of specifying the region and the step of selecting the color, a parameter for specifying the region and a color to be changed together with the image are displayed on the display device. 7. The image processing method according to claim 1, wherein candidates are displayed. 8. An input device for executing the image processing according to claim 1, wherein a display unit for displaying the image, and the area on the display unit together with the image are displayed on the display unit. Means for displaying information for specifying and color candidates. 9. The method according to claim 1, wherein the step of specifying an area in which the color of the image is to be changed includes the step of further identifying, from the specified area, a human face area whose color is to be changed, based on a human skin color. The image processing method according to claim 1, wherein: 10. The method according to claim 1, wherein the step of specifying an area in which the color of the image is to be changed further includes the step of identifying a human face area from the specified area based on a human face part. Item 7. The image processing method according to any one of Items 1 to 6. 11. The human face part is at least one of a human eyebrow, eyes, and mouth displayed on the display means, and the part is designated via the dialog box. The image processing method according to claim 10, wherein 12. The image processing method according to claim 3, wherein the one color information is information indicating a characteristic of a human skin color. 13. The method according to claim 1, wherein the color to be changed is a sky color recorded as image information.
The image processing method according to 1. 14. The method according to claim 1, wherein the step of identifying an area in which the color of the image is to be changed further includes the step of further identifying an empty area in which the color is to be changed from the identified area based on the sky color. The image processing method according to claim 1, wherein: 15. The image processing method according to claim 3, wherein the one color information is information indicating a feature of a sky color. 16. A computer-readable recording medium on which a program for executing the image processing method according to claim 1 is recorded.