JP6343998B2 - Image processing apparatus, image processing method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus that extracts a subject area from a subject image and combines it with a desired image. More specifically, the subject image is automatically classified into a foreground area, a background area, and an undetermined area. The present invention relates to an image processing device that generates a trimap for image synthesis.

  The ID photo box service includes various image processing services (background synthesis, skin softening correction, etc.) for captured images, and one of them is chroma key processing. The chroma key processing is processing for extracting a subject image region (hereinafter referred to as a foreground region) from a subject image photographed with a single color background and combining it with an arbitrary background image.

  For chroma key processing, a foreground region extraction method based on color information is generally applied. Techniques described in Non-Patent Documents 1 to 3 are known as foreground region extraction methods based on color information.

  In Non-Patent Document 1, for a gray image, a user assigns a marker representing a foreground region and a marker representing another region (background region) in a desired image region in advance, and based on the given marker Thus, a technique for extracting a foreground region by Graph Cuts is described.

  Non-Patent Document 2 describes a technique for extracting a foreground region by applying a graph cut to a color image.

  Non-Patent Document 3 creates in advance a map (hereinafter referred to as a trimap) in which an original image is classified into three areas: a foreground area, a background area, and an indeterminate area that is unknown whether it belongs to a foreground area or a background area. A technique for extracting a foreground area by estimating a mixture ratio of pixels in a foreground image area and pixels in a background area (hereinafter referred to as α value) in an undetermined area is described.

Y. Boykov, M. Jolly, “Interactive graph cuts for optimal boundary and region segmentation of objects in N-Dimages”, ICCV, 2001 C.Rother, V. Kolmogorov, A. Blake, “Grabcut-interactive foreground extractionusing iterated graph cuts”, SIGGRAPH, 2004 C. Rhemann, C. Rother, A. Rav-Acha, T. Sharp, “High Resolution Matting via Interactive Trimap Segmentation”, CVPR, 2008

  However, in the techniques described in Non-Patent Documents 1 to 3, it is necessary to manually input region designations such as the foreground region and the background region, and it is difficult to directly apply to unmanned devices such as ID photo boxes.

  Even if a trimap is automatically created, if the foreground area contains a color similar to the background color, the area is determined to be a background area, and the subject's hair, lace, etc. When the background is mixed with transparent clothes, it may be difficult to distinguish the foreground area from the background area.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide an image processing apparatus that automatically generates a trimap for appropriately executing chroma key processing. .

In order to solve the above-described problem, the first invention is configured such that each pixel constituting an original image obtained by photographing the upper body of a subject includes a background region, a foreground region that is a portion of the subject, and the background region or the foreground region. An image processing apparatus for generating a trimap that is a ternary image classified into an uncertain area that is unknown to which of the above, a silhouette specifying unit that binarizes the original image and specifies the silhouette of the subject; , Specifying pixels outside the silhouette as pixels in the background region, specifying pixels within the silhouette as pixels in the undetermined region, and selecting pixels in the skin color region of the subject among the pixels in the undetermined region Skin area specifying means for updating to a foreground area to generate the trimap, and specifying a breast area by specifying a neck position of the subject, and a pixel corresponding to the breast area is set to the front area Chest area specifying means for updating the area, color histogram determination means for determining the hair length of the subject, generating a color histogram in different ranges, and updating the trimap using the color histogram, An image processing apparatus comprising:
According to the image processing apparatus of the first invention, the chroma key composition process is appropriately executed even when a color similar to the background exists in the foreground area of the original image or an area where the background is transparent exists. The possible tri-map can be automatically generated.

Further, it is preferable that the silhouette specifying means specifies a silhouette from a binarized image generated based on a background color and an edge image generated by performing contour extraction processing on the original image.
Thereby, it is possible to appropriately generate a trimap even for an original image in which the subject's hair is mixed with the background color.

Further, the pixels of the undetermined region specified in the skin area specifying unit based on the background color, updating the trimap using different thresholds in the region of the region and under the above submandibular line of the object It is desirable to further include a threshold value determination means.
As a result, the pixels of the undetermined region can be classified into a region above the submandibular line that is the region of the subject's face and hair and a region below the submandibular line that is the region of the subject's clothes. Therefore, an area that is difficult to distinguish from the background (a state in which background pixels and foreground pixels are mixed on the original image) can be left as undetermined pixels. Therefore, in the synthesis process with the background image after generating the trimap (ternary image), synthesis without any sense of incongruity can be expected. If it is determined as a background pixel or a foreground pixel in a region that is difficult to distinguish from the background, a sudden color change may cause a sense of incongruity in the combined result.

In addition, it is preferable that the chest area specifying unit sets a neck search area, and specifies the neck position by comparing the number of pixels in the foreground area and the number of pixels in the undetermined area in the X-axis direction. .
Thereby, since the position of the neck can be specified accurately, even if the clothes area includes a color similar to the background color (for example, the color of the tie), it can be specified as the foreground area.

Further, the color histogram determination unit specifies the submandibular line of the subject based on the determined hair length, and each pixel value of the original image corresponding to the foreground region of the region below the specified submandibular line. Color histogram generation means for generating a color histogram, and for each pixel value of the original image corresponding to the undetermined region below the submandibular line of the trimap, a color number existing on the color histogram, and A determination unit that determines whether or not to approximate or match; and a first update unit that updates the pixel constituting the undetermined region to the foreground region in the trimap when it approximates or matches. It is desirable.
As a result, among the pixels in the undetermined area in the area below the submandibular line, a pixel having a color that approximates or matches the color of the clothes of the subject person can be set as the foreground area. In addition, it is possible to leave undecided pixels in regions that are difficult to distinguish from the background (regions in which background pixels and foreground pixels that are often found in subjects with long hair are mixed on the original image). Become. Therefore, in the synthesis process with the background image after generating the trimap (ternary image), synthesis without any sense of incongruity can be expected. If it is determined as a background pixel or a foreground pixel in a region that is difficult to distinguish from the background, a sudden color change may cause a sense of incongruity in the combined result.

Further, it is preferable that the color histogram generation means excludes color numbers having a small number of pixels belonging to the color number from the color histogram among color numbers existing on the generated color histogram.
Thereby, an effect of excluding noise existing in the foreground region in the original image can be expected.

In addition, for the trimap updated by the color histogram determination means, a matrix having a predetermined size centered on each pixel constituting the uncertain region is generated, and pixels existing in the generated matrix Is counted by the in-matrix counter means for determining whether the pixel belongs to the foreground area, the background area, or the undetermined area, and counting the number of pixels for each area. Second updating means for updating, in the trimap, a pixel at the center of the matrix to the foreground area when the number of pixels in the foreground area is larger than the counted number of pixels in the uncertain area; It is desirable to provide.
As a result, an undetermined area surrounded by (inside) the foreground area can be set as the foreground area.

Further, when the number of pixels belonging to the background area is counted one or more in the matrix generated by the in-matrix counter means, the second update means sets the center pixel of the matrix as the undetermined area. It is desirable to specify.
As a result, the undetermined area in contact with the background area is an area that is difficult to distinguish from the background (the area where the background pixels and foreground pixels are mixed on the original image). It becomes possible to do. Therefore, the above-described effect can be obtained.

According to a second aspect of the present invention, each pixel constituting an original image obtained by photographing the upper body of a subject is assigned to a background region, a foreground region that is a part of the subject, and whether the background region or the foreground region is unknown. An image processing method for generating a trimap that is a ternary image classified into a fixed region, wherein the original image is binarized to specify a silhouette of the subject, and a pixel outside the silhouette is selected from the silhouette A pixel in the background area, a pixel in the silhouette is specified as a pixel in the undefined area, and a pixel in the skin color area of the subject is updated to the foreground area among the pixels in the undefined area. A skin region specifying step for generating a map, a chest region is specified by specifying the neck position of the subject, and a pixel corresponding to the chest region is updated to the foreground region. A region specifying step; and a color histogram determining step of determining a hair length of the subject, generating a color histogram in different ranges, and updating the trimap using the color histogram. This is an image processing method.
According to the image processing method of the second invention, the chroma key composition process is appropriately executed even when a color similar to the background exists in the foreground area of the original image or an area where the background is transparent exists. The possible tri-map can be automatically generated.

According to a third aspect of the invention, the computer constitutes each pixel constituting the original image obtained by photographing the upper body of the subject, belonging to the background region, the foreground region that is the portion of the subject, the background region, or the foreground region. A program for causing a computer to function as an image processing apparatus that generates a trimap that is a ternary image classified into an unknown uncertain region, and specifying the silhouette of the subject by binarizing the original image Silhouette specifying means, specifying a pixel outside the silhouette as a pixel in the background area, specifying a pixel in the silhouette as a pixel in the undetermined area, and out of the pixels in the undetermined area in the skin color area of the subject Skin region specifying means for generating a trimap by updating pixels to the foreground region; specifying a chest region by specifying a neck position of the subject; Chest area specifying means for updating the pixel corresponding to the breast area to the foreground area, determining the hair length of the subject, generating color histograms in different ranges, and using the color histogram, the trimap Is a program for functioning as color histogram determination means for updating the image.
According to the program of the third invention, the chroma key composition processing can be appropriately executed even when a color similar to the background exists in the foreground area of the original image or an area where the background is transparent exists. Possible trimaps can be automatically generated.

  According to the present invention, it is possible to provide an image processing apparatus or the like that automatically generates a trimap for appropriately executing chroma key processing.

1 is a block diagram showing an example of the hardware configuration of an image processing apparatus according to the present embodiment. 1 is a block diagram illustrating an example of a functional configuration of an image processing apparatus according to the present embodiment. An example of data held by the storage unit of the image processing apparatus Flow chart showing the flow of trimap generation processing Illustration explaining the original image Flow chart showing the flow of silhouette image generation processing The figure which shows the image produced | generated by performing the silhouette image production | generation process to the original image 31 The figure which shows the image produced | generated by performing a silhouette image production | generation process to the original image 36 Flow chart showing the flow of foreground area specifying processing The figure which shows the tri map produced | generated by the foreground area | region specific process A flowchart showing the flow of threshold processing in a color space A diagram for explaining threshold processing in a color space Flow chart showing the flow of the neck area specifying process The figure explaining neck area specific processing Flow chart showing the flow of submandibular line update processing The figure explaining submandibular line update processing Flow chart showing the flow of color histogram determination processing The figure which shows the trimap after color histogram determination processing Flow chart showing flow of counter processing in matrix Diagram explaining counter processing in matrix The figure which shows the tri map after execution of the expansion process Diagram explaining alpha matting process Diagram showing alpha mask

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a hardware configuration diagram of a computer that realizes an image processing apparatus 100 according to an embodiment of the present invention. As shown in FIG. 1, the computer includes, for example, a control unit 21, a storage unit 22, a media input / output unit 23, a communication control unit 24, an input unit 25, a display unit 26, a peripheral device I / F unit 27, etc. 28 is connected and configured.

The control unit 21 is a CPU (Central
A processing unit (ROM), a read only memory (ROM), a random access memory (RAM), and the like.
The CPU calls a program stored in the storage unit 22, ROM, storage medium, etc. to a work memory area on the RAM and executes it, and drives and controls each device connected via the bus 28. The process to be described later is realized. The ROM is a non-volatile memory and permanently holds a computer boot program, a program such as BIOS, data, and the like. The RAM is a volatile memory, and temporarily stores a loaded program, data, and the like, and includes a work area used by the control unit 21 to perform each process.

The storage unit 22 is an HDD (Hard
The application program executed by the control unit 21, data necessary for program execution, an OS (Operating System), and the like are stored. These program codes are read by the control unit 21 as necessary, transferred to the RAM, and read and executed by the CPU.

The media input / output unit 23 is a media input / output device such as a CD drive, a DVD drive, an MO drive, and a floppy (registered trademark) disk drive, and inputs / outputs data such as original images.
The communication control unit 24 includes a communication control device, a communication port, and the like, and is a communication interface that mediates communication between the computer and the network, and controls communication with other devices via the network.

The input unit 25 performs data input and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad. The input data is output to the control unit 21.
The display unit 26 includes, for example, a display device such as a CRT monitor or a liquid crystal panel, and a logic circuit (video adapter or the like) for executing display processing in cooperation with the display device, and is input by the control of the control unit 21. The displayed display information is displayed on the display device.
The input unit 25 and the display unit 26 may be, for example, a display with a touch panel in which those functions are integrated.

The peripheral device I / F unit (interface) 27 is a port for connecting a peripheral device to the computer, and the computer transmits and receives data to and from the peripheral device via the peripheral device I / F unit 27. The peripheral device I / F unit 27 is configured by USB, IEEE 1394, RS-232C, or the like, and usually includes a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless.
The bus 28 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  FIG. 2 is a block diagram illustrating a functional configuration example of the image processing apparatus 100 according to the present embodiment. As shown in FIG. 2, the image processing apparatus 100 includes an original image input unit 11, a subject position input unit 12, a trimap generation unit 13, an alpha mask generation unit 14, a background image synthesis unit 15, an image output unit 16, and the like. Composed.

  The image processing apparatus 100 generates a trimap of the original image input from the original image input unit 11 by the trimap generation unit 13, generates an alpha mask and a foreground image by the alpha mask generation unit 14, and a background image synthesis unit 15. To generate a composite image.

The original image input unit 11 inputs, to the image processing apparatus 100, an original image obtained by photographing a subject with a substantially single color background such as a blue background or a green background.
The original image is obtained by photographing a subject such as a person at a photographing place such as an ID photo box, and may be input from the communication input / output unit 23, the input unit 25, or the communication control unit 24 via the network. It doesn't matter how.

  Based on the input original image, the subject position input unit 12 is input mainly by the input unit 25 with information such as coordinate values of the positions of the top of the subject, the chin, and the center of the left and right sides of the face. .

  The trimap generation unit 13 receives the information input by the subject position input unit 12, the data stored in the storage unit 22 (see FIG. 3), and the like for the original image input by the original image input unit 11. A trimap is generated by classifying each pixel of the original image into one of a background area, a foreground area, and an undetermined area. Details of the trimap generation processing executed by the trimap generation unit 13 will be described later.

The alpha mask generation unit 14 calculates an alpha value of each pixel of the original image based on the original image input by the original image input unit 11 and the trimap acquired from the trimap generation unit 13, and the alpha mask and the foreground Generate an image. The process executed by the alpha mask generation unit 14 is based on the process described in Non-Patent Document 4. Details will be described later.
<Non-Patent Document 4> Eduardo
SLGastal, Manuel M. Oliveira, Shared Sampli-ng for Real-Time Alpha Matting,
Computer Graphics Forum Volume 29 (2010),
November 2
Proceedings of Eurographics 2010, pp.575-584

  The background image synthesizing unit 15 synthesizes the foreground image acquired from the alpha mask generating unit 14 and the background image stored in the storage unit 22 according to the alpha mask generated by the alpha mask generating unit 14, and combines the synthesized image. Is generated. Processing executed by the background image composition unit 15 will be described later.

  The image output unit 16 outputs the composite image synthesized by the image synthesis unit 16 by the display unit 26 or the like.

  The image processing apparatus according to the present invention is realized by at least the functions of the original image input unit 11, the subject position input unit 12, and the trimap generation unit 13 in the functions of the image processing apparatus 100 in the present embodiment. .

  FIG. 3 is a diagram illustrating an example of data stored in the storage unit 22 of the image processing apparatus 100.

  As data held by the storage unit 22, for example, image data 22a of an original image input from the original image input unit 11, a background color 22b of the original image, a Sobel filter 22c used in a trimap generation process described later, a matrix Used by the size 22d and the number of uncertain region expansion processes 22e, the trimap 22f generated by the trimap generator 13, the alpha mask 22g and foreground image 22h generated by the alpha mask generator 14, and the background image synthesizer 15. These are various parameters 22j used in processing executed by the background image 22i, the trimap generation unit 13 and the alpha mask generation unit 14.

  The background color 22b is a representative color of a background such as a blue background used as the background of the original image, and is specified by a combination of hue / saturation / lightness such as an RGB value or an HSV color system.

[Trimap generation processing]
Next, trimap generation processing executed by the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 will be described with reference to FIGS. The trimap generation process is a process for automatically generating a trimap of the original image.

FIG. 4 is a flowchart showing the flow of the trimap generation process.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 receives the input of the original image from the original image input unit 11 (step S101).

FIG. 5 is a diagram illustrating an example of the original images 30 and 31.
As shown in FIG. 5A, the original image 30 is image data of a photograph of the upper body of a subject (person) 30a with a blue background or the like as a background 30b. Here, the entire photo is cartoonized. The representative color (background color 22b) of the background 30b is stored in the storage unit 22.

  Here, it is assumed that a color similar to the background 30b is included in a part of the subject 30a (for example, a portion 30c surrounded by a broken line). In that case, in the conventional method, the region of the trimap is determined as the background region, and the color of a part of the subject of the generated composite image may be lost. By executing the trimap generation process of the present embodiment, it is possible to determine a foreground area and generate a trimap even in a color area similar to the background 30b in the foreground area (subject 30a). Thus, an appropriate composite image can be obtained.

  FIG. 5B shows another example of the original image 31. The original image 31 is image data of a subject (person) 31a obtained by photographing a blue background or the like on a background 31b. Here, the entire photo is cartoonized.

  Here, it is assumed that the background 31b can be seen through the hair of the subject 31a (for example, portions 31c and 31d surrounded by a broken line). In that case, in the conventional method, the region of the trimap is determined to be the foreground region, and the color of the background 31b may remain in a part of the subject 31a of the generated composite image. By executing the tri-map generation process of the present embodiment, a tri-map is generated by appropriately classifying the foreground area, the background area, and the undetermined area even in the area where the background 31b is mixed in the foreground area (subject 31a). This makes it possible to obtain an appropriate composite image.

  Returning to the description of FIG. The control unit 21 (tri-map generating unit 13) of the image processing apparatus 100 receives input of coordinate values of the positions of the subject's top, chin, and right and left centers of the face from the subject position input unit 12 (step). S102).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes a silhouette image generation process (step S103).

  The silhouette image generation process is a binary mask image generated by comparing the color of each pixel of the original images 30 and 31 with the background color 22b, and an edge generated by contour extraction processing of the original images 30 and 31. This is a process of generating a silhouette image by combining the image. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes a foreground region specifying process (step S104).

  The foreground area specifying process specifies an undefined area and background area of the trimap from the silhouette image generated in step S103, and generates a skin mask from the skin color of the subjects 30a and 31a of the original images 30 and 31. In this process, the pixels in the skin mask area in the undetermined area are updated to the foreground area to generate a trimap. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes threshold processing on the color space (step S105).

  In the threshold processing in the color space, the pixels of the original images 30 and 31 corresponding to the undetermined region on the trimap generated in step S104 are input, and the color of each pixel approximates the background color 22b. In this case, the pixel is not updated in the undetermined region, and if the pixel is not approximated with the background color 22b, the pixel is updated to the foreground region. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes a neck region specifying process (step S106).

  The neck area specifying process specifies the position of the subject's neck on the trimap updated in step S105, and updates all the pixels in the breast area set with the specified position as the upper end to the foreground area. It is processing to do. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes a submandibular line update process (step S107).

  The submandibular line update process is a process of updating the submandibular line of the subject set in step S105 according to the length of the subject's hair. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 performs color histogram determination processing (step S108).

  The color histogram determination process generates a color histogram based on the pixel value of each pixel in the foreground area in the area below the submandibular line of the trimap updated in step S106, and the undefined area in the area In this process, it is determined whether or not the pixel value of each pixel is on the color histogram, and if it exists, the pixel is updated to the foreground region. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes the in-matrix counter process (step S109).

  In the matrix counter process, a matrix having a matrix size 22d (see FIG. 3) centering on the pixel is arranged for each pixel in the undefined region in the trimap updated in step S108. If the ratio of the number of pixels in the foreground area is larger than the ratio of the number of pixels in the undefined area, the pixel is updated to the foreground area. Details will be described later.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 generates a trimap by executing dilation processing on the pixels in the undetermined area in the trimap generated in step S109 (step S109). S110), the generated trimap is stored in the storage unit 22, and the process is terminated.

  The expansion process is a process in which the pixels in the background area or the foreground area around the pixels in the undetermined area are set as the undetermined areas. For example, pixels surrounded by an arbitrary constant with each pixel in the undetermined area as the center. In this process, pixels in a region (for example, 3 × 3 pixels, 5 × 5 pixels) are set as undefined regions.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 repeats the expansion process a predetermined number of times in accordance with the expansion process count 22e of the undetermined area stored in the storage unit 22. In addition, the calculation time increases as the value of the number of expansions increases, and a highly accurate result can be obtained.

  As described above, the image processing apparatus 100 receives the input of the original images 30 and 31 and executes a silhouette image generation process to generate a silhouette image. Then, a foreground area specifying process is executed to generate a trimap that classifies the original image into a foreground area, a background area, and an undetermined area. Subsequently, the image processing apparatus 100 performs threshold processing on the color space, neck area specifying processing, submandibular line update processing, color histogram determination processing, and intra-matrix counter processing on the generated trimap. Then, the pixels belonging to the undetermined area in the trimap are updated to the foreground area, and the expansion process is performed to generate the trimap.

  As a result, even if there is a color similar to the background in the foreground area of the original image or there is an area where the background is transparent, a trimap that can execute the chroma key composition processing automatically is automatically generated. Can be generated.

[Silhouette image generation processing]
Next, details of the silhouette image generation process performed by the image processing apparatus 100 in step S103 of FIG. 4 will be described with reference to FIGS. Here, an example in which the silhouette image generation process is executed to generate the silhouette image 35 shown in FIG. 7C from the original image 31 shown in FIG. 5B will be described.

FIG. 6 is a flowchart showing a flow of silhouette image generation processing.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the pixel value (color) of the pixel of the original image 31 (step S201).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not the input pixel color approximates the background color 22b (step S202). The color approximation is determined by, for example, a distance in the color space. The allowable amount of the color range regarded as the background area is set in advance.

  In the case of approximation (YES in step S202), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 specifies the pixel value of the input pixel as 0 (black) (step S203). If not approximated (NO in step S202), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 proceeds to step S204.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not all the pixels of the original image 31 have been input (step S204). If it has not been input (NO in step S204), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 returns to step S201.

  If it has been input (YES in step S204), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 specifies a pixel value in step S203 among the pixels of the input original image 31. A binary mask image 33 is generated with the pixel value of the pixel that has not been set as 255 (white) (step S205).

  FIG. 7A shows a binary mask image 33 generated from the original image 31. Pixels determined to be close to the background color 22b in step S203 are represented by black, and pixels that are not determined to be approximate are represented by white.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 applies the Sobel filter 22c to the original image 31 to generate the edge image 34 (step S206).

  Specifically, the original image 31 is converted into a gray scale, and the gray scaled image is processed using the Sobel filter 22c to generate an image. The pixel value of the generated image is binarized according to the density, and an edge image 34 in which each pixel has 255 (white) or 0 (black) is generated. Here, the threshold used for binarization is preferably close to 0 (black) in order to detect as many edges as possible. The method for generating the edge image 34 of the original image 31 is not particularly limited as long as it is a method capable of detecting the edge of the original image 31.

  FIG. 7B shows an edge image 34 generated from the original image 31. The detected contour is shown in white.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 generates a silhouette image 35 obtained by combining the binary mask image 33 and the edge image 34 (step S207), and ends the process. The silhouette image 35 means that the pixel value of a pixel having a pixel value of 255 (white) in the binary mask image 33 or the edge image 34 is 255 (white), and the edge of the edge image 34 and the binary mask image 33 This is an image generated by setting the pixel value of the pixel in the region surrounded by the outline to 255 (white) and the pixel value of the pixel in the other region to 0 (black).

FIG. 7C shows a silhouette image 35 generated from the original image 31.
Here, the hair protrudes from the subject 31a in the portion 31c (see FIG. 5) surrounded by the broken line of the original image 31, but in the binary mask image 33, the hair is cut off in the corresponding portion 33a and is used as the region of the subject 31a. It has not been extracted. However, since the hair can be detected as an edge in the corresponding portion 34a of the edge image 34, the region of the subject 31a including the hair can be extracted from the corresponding portion 35a of the silhouette image 35.

FIG. 8 shows an example in which the silhouette image generation process is applied to another original image 36.
In the original image 36 shown in FIG. 8A, the subject 36a has a strap of a color approximating the background color hung around the neck (portion 36b surrounded by a broken line). In such a case, as shown in FIG. 8B, a binary mask image 37 in which the pixel value of the pixel in the strap portion is specified as 0 (black) is generated (portion 37b surrounded by a broken line). . However, by combining the edge image (not shown) generated from the original image 36 and the binary mask image 37, the strap portion is also extracted as the region of the subject 36a as shown in FIG. 8C. A silhouette image 38 can be generated (portion 38b surrounded by a broken line).

  As described above, the image processing apparatus 100 executes the silhouette image generation process, generates the binary mask image 33 and the edge image 34 of the original image 31, and combines them to generate the silhouette image 35. As a result, it is possible to reliably identify the portion that is the background region as the background region and sufficiently extract the region that may be included in the subject. By using the edge image 34 to generate the silhouette image 35, even if a thin part such as hair is included in the subject area or a part close to the background color is mixed, the part and the peripheral part are the background. Without being designated as a region, it can be extracted as a subject region (foreground region + undetermined region).

[Foreground area identification processing]
Next, details of the foreground area specifying process executed by the image processing apparatus 100 in step S104 of FIG. 4 will be described with reference to FIGS. Here, an example in which the foreground area specifying process is executed to generate the trimap 39 of FIG. 10 from the original image 30 and the silhouette image (not shown) of the original image 30 shown in FIG. .

FIG. 9 is a flowchart showing the flow of the foreground area specifying process.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 identifies the silhouette inner region of the silhouette image generated in step S103 as an undetermined region (gray), and determines the silhouette outer region of the silhouette image. A background area (black) is specified (step S301).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 identifies the representative color of the skin of the subject 30a of the original image 30 (step S302). The image processing apparatus 100 estimates the vicinity of the nose region based on the eye position of the subject 30a, and sets the region as the representative color of the skin (the average value of the region may be the representative color of the skin).

The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 measures the Mahalanobis distance between the representative color of the skin and the pixel value of each pixel in the undetermined region of the original image 30, and the distance is within a predetermined value. Are extracted as a skin mask which is the skin color portion of the subject 30a (step S303). Details of the process of extracting the skin mask are described in Patent Document 1.
<Patent Document 1> Japanese Patent No. 5029545

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 identifies the skin region of the skin mask in the unconfirmed region as the foreground region (white) (step S304).

  Based on the information specified in step S301 and step S304, the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether each pixel of the original image 30 is a background region, an undefined region, or a foreground region. The trimap 39 classified into the crab is generated, and the process ends.

  FIG. 10 shows a trimap 39 of the original image 30 generated by the foreground area specifying process. The foreground area of the trimap 39 is shown in white, the unconfirmed area in gray, and the background area in black.

  As described above, the image processing apparatus 100 executes the foreground region specifying process, specifies the background region and the uncertain region by the silhouette image of the original image 30, and specifies the foreground region by the skin mask of the original image 30. Then, a trimap 39 is generated. Thereby, it is possible to generate the trimap 39 having the skin portion of the face of the subject 30a as the foreground region.

[Threshold processing in color space]
Next, details of threshold processing in the color space executed by the image processing apparatus 100 in step S105 of FIG. 4 will be described with reference to FIGS. Here, threshold processing in the color space is executed, and the original image 30 shown in FIG. 12A (same as the original image 30 shown in FIG. 5A) and the trie generated in step S104. An example of generating the trimap 40 shown in FIG. 12B from the map 39 will be described.

FIG. 11 is a flowchart showing the flow of threshold processing in the color space .
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 sets the pixel value (color) of each pixel of the original image 30 corresponding to the undetermined region of the trimap 39 generated in step S104 to the LUV color. The space is converted (step S401). Conversion to the LUV color space facilitates the setting of parameters used for the discrimination processing executed in step S404 and step S409, but the processing after step S402 may be executed using another color space. .

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 sets the submandibular line 30g of the original image 30 (step S402).

  A setting example of the submandibular line 30g of the original image 30 will be described with reference to FIG. The original image 30 shows the parietal position 30e and the jaw position 30f that received the input in step S102. The distance from the top position 30e to the chin position 30f is set as the face width 30h, and the submandibular line 30g is set at a position lower than the chin position 30f by a distance corresponding to 8% of the face width 30h (chin width 31i). The above method is an example of setting, and for example, a method of setting the submandibular line 30g at a position lower than the jaw position 30f by a predetermined number of pixels may be used.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the pixel value (color) of the pixel of the original image 30 corresponding to the undetermined region in the region above the submandibular line 30g (step S403). .

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not the input pixel color approximates the background color 22b (step S404). The color approximation is determined by, for example, a distance in the color space. The allowable amount of the color range that is regarded as the foreground area or the undetermined area is set in advance.

  In the case of approximation (YES in step S404), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 does not update the input pixel (step S405). If not approximated (NO in step S404), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 updates the input pixel to the foreground region (white) (step S406).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not all the pixels of the original image 30 corresponding to the undetermined region in the region above the submandibular line 30g have been input (step). S407). If it has not been input (NO in step S407), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 returns to step S403.

  When the input has been completed (YES in step S407), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 reads the original image 30 corresponding to the undetermined region in the region below the submandibular line 30g. The pixel value (color) of the pixel is input (step S408).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not the input pixel color approximates the background color 22b (step S409). The color approximation is determined by, for example, a distance in the color space. The allowable amount of the color range that is regarded as the foreground area or the undetermined area is set in advance.

  It is preferable that the set value of the color threshold and the allowable range used in the determination process in step S409 is different from the set value used in the determination process in step S404. The region above the submandibular line 30g includes many faces and hair portions of the subject 30a, and the region below the submandibular line 30g includes many clothing portions of the subject 30a. By using different setting values for each region, it is possible to reliably leave pixels that may belong to the undetermined region as the undetermined region.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 executes the same processes as in steps S405 to S407, generates the trimap 40 (steps S410 to S412), and ends the process.

  FIG. 12B shows a trimap 40 of the original image 30 generated by threshold processing on the color space.

  As described above, the image processing apparatus 100 performs the threshold processing on the color space, divides the original image 30 into the upper region and the lower region of the submandibular line, and sets the pixels in the undetermined region for each divided region. The trie map 40 is generated by updating to the foreground area based on the closeness to the background color 22b.

[Neck area identification process]
Next, details of the neck area specifying process executed by the image processing apparatus 100 in step S106 of FIG. 4 will be described with reference to FIGS. Here, an example will be described in which the neck region specifying process is executed to generate the trimap 49 shown in FIG. 14C from the trimap 40 generated in step S105.

FIG. 13 is a flowchart showing the flow of the neck area specifying process.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the trimap 40 generated in step S105 (step S501), and sets the neck search area 42 (step S502).

A setting example of the neck search area 42 of the trimap 40 will be described based on FIG. FIG. 14A shows the center line 41c and jaw position 30f of the subject 30a that has received the input in step S102, and the submandibular line 30g set in step S402 of the threshold processing (FIG. 11) on the color space. Is a trimap 40 showing Search end lines 41a and 41b are lines obtained by expanding one-eighth the width of the original image 30 from the center line 41c to the left and right. Here, a region surrounded by the jaw position 30f, the submandibular line 30g, and the search end lines 41a and 41b is set as a neck search region 42.
The above method is an example of setting, and for example, a method of setting the search end lines 41a and 41b at positions expanded from the center line 41c to the left and right by a predetermined number of pixels may be used.

  FIG. 14B shows an enlarged view of the neck search area 42. The intersection of the center line 41 c and the submandibular line 30 g is set as a search start point 43. Hereinafter, description will be made on an XY coordinate space where the search start point 43 is the origin (0, 0), the right direction of the image is the X-axis direction, and the upward direction of the image is the Y-axis direction. Let the intersection of the center line 41c and the jaw position 30f be the search end point 44, and the coordinate value be (0, k). The search end point 44 is a point moved from the search start point 43 by k pixels in the Y-axis direction.

The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 sets i = 0 (step S503).
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 changes from the foreground pixel to the uncertain pixel in the X-axis positive (negative) direction from the search start point 43 in the first row in the neck search region. The first position is set (stored) as an initial value of the neck position candidate (step S504).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the search start point (0, i) (step S505). FIG. 14B shows an example of the search start point 43a.

  Subsequently, the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 scans from the search start point 43a (0, i) to the search end point (m, i) in the X-axis positive (negative) direction, and foreground area. The number of pixels (hereinafter referred to as foreground pixels) and the pixels in the undetermined region (hereinafter referred to as undetermined pixels) are counted for each row (step S506). FIG. 14B shows an example of the search end point 44a.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines the number of foreground pixels and the number of unconfirmed pixels existing at the search start point 43a (0, i) to the search end point 44a (m, i). Is identified (step S507).

  The control unit 21 (the trimap generation unit 13) of the image processing apparatus 100 compares the number of identified foreground pixels with a value obtained by tripling the number of undetermined pixels (step S508).

When the number of foreground pixels is three times or less than the number of undetermined pixels (NO in step S508), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 proceeds to step S510.
When the number of foreground pixels is larger than three times the number of undetermined pixels (YES in step S508), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 coordinates the pixel scanned in step S506. The value (first coordinate value changed from the foreground pixel to the undetermined pixel) is stored in the RAM or the like of the control unit 21 as a neck position candidate (step S509).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 moves the search start point 43a by one pixel in the Y-axis direction (in the direction of the white arrow shown in FIG. 14B) (step S510). ) The newly set Y coordinate value (i) of the search start point 43a is compared with the Y coordinate value (k) of the search end point 44 (step S511).

If i <k (YES in step S511), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 returns to step S505.
When i ≧ k (NO in step S511), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 selects the foreground pixel closest to the search end line 41b among the stored neck position candidates. Is specified as the neck position 45a.
(Step S512).

The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 similarly specifies the neck position 45b in the X-axis negative direction.
FIG. 14B shows an example of neck positions 45a and 45b.

The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 sets the chest region 48 with the neck positions 45a and 45b as the upper base (step S513).
A setting example of the chest region 48 in the trimap 49 (trimap 40) will be described with reference to FIG. A region surrounded by a line segment 47a, 47b having a length corresponding to 80% of the face width 30h centered on the center line 41c at the bottom of the image and a line segment connecting the neck positions 45a, 45b as the upper base. The chest area 48 is set. The above method is an example of setting. Since the neck positions 45a and 45b are specified in the left-right direction, the neck positions 45a and 45b (X coordinate value and Y coordinate value) are different on the left and right.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 updates the chest area 48 in the trimap 40 to the foreground area, generates the trimap 49 (step S514), and ends the process.

  As described above, the image processing apparatus 100 executes the neck region specifying process and accurately specifies the position of the neck of the subject 30a, so that it can be applied to the lower region (clothing region) of the subject 30a as much as possible. A wide range can be used as the foreground region. As a result, even if the clothes area includes a color similar to the background color (for example, the color of the tie), the area can be specified as the foreground area.

[Submandibular line update process]
Next, details of the submandibular line update process performed by the image processing apparatus 100 in step S107 of FIG. 4 will be described with reference to FIGS. 15 and 16. Here, an example in which the submandibular line update process is executed on the trimap 49 generated in step S106 will be described.

FIG. 15 is a flowchart showing the flow of the submandibular line update process.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the trimap 49 generated in step S106 (step S601), and sets the hair determination region 50 (step S602).

A setting example of the hair determination area 50 of the trimap 49 will be described with reference to FIG. FIG. 16A shows the position 30e and chin position 30f of the subject 30a that has received the input in step S102, and the submandibular line 30g set in step S402 of the threshold processing (FIG. 11) on the color space. Is a tri-map 49 showing A position where the distance corresponding to 60% of the face width 30h is lowered from the top position 30e is defined as a determination end line 48, and a region sandwiched between the submandibular line 30g and the determination end line 48 is defined as a hair determination region 50.
The above method is an example of setting. For example, a method in which a position that is a predetermined number of pixels lower than the top position 30e is used as the determination end line 48 may be used.

  As shown in FIG. 16A, the right end of the submandibular line 30g is set as the determination start point 51. Hereinafter, description will be made on an XY coordinate space where the determination start point 51 is the origin (0, 0), the left direction of the image is the X-axis direction, and the upward direction of the image is the Y-axis direction. The right end of the determination end line 48 is a determination end point 52, and the coordinate value is (0, k). The determination end point 52 is a point moved from the determination start point 51 by k pixels in the Y-axis direction.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 first sets i = 0 (step S603) and inputs the search start point (0, i) (step S604). The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 searches for undefined pixels from the right end to the left end of the image in the left direction from the search start point (step S605), and the number of detected undefined pixels. Are compared with a preset number of pixels (step S606).

When the detected number of unconfirmed pixels is smaller (NO in step S606), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 ends the process.
If the number of detected uncertain pixels is larger (YES in step S606), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 sets the search start point by one pixel in the Y-axis direction. It is moved (in the direction of the white arrow shown in FIG. 16A) (step S607), the newly set Y coordinate value (i) of the search start point and the Y coordinate value (k) of the determination end point 52 Are compared (step S608).

If i <k (YES in step S608), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 returns to step S604.
When i ≧ k (NO in step S608), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 newly sets a position where the submandibular line 30g is lowered by several pixels (for example, 50 pixels). The line is updated to the submandibular line (step S609), and the process ends.

  FIG. 16B shows the trimap 60 after the neck region specifying process (step S106) of the original image 31. FIG. The subject 31a of the original image 31 is long hair, and hair is included in the entire hair determination region 64 surrounded by the submandibular line 62 and the determination end line 61. In such a case, the submandibular line is updated in step S609 (updated submandibular line 63). On the other hand, in the trimap 49 shown in FIG. 16A, the subject has short hair, and in such a case, the submandibular line 30g is not updated.

  As described above, the image processing apparatus 100 executes the submandibular line update process, and determines that the subject is long hair when the undetermined area in the hair determination area is always large, and in the case of long hair. The submandibular line is lowered by several pixels, and the color histogram determination process described below is executed. When the subject is long hair, there may be many uncertain areas where the background and hair are mixed around the chin and shoulder. It can be prevented in advance that the color histogram determination process is performed on such an area and the area is updated to the foreground area.

  In the present embodiment, the example in which the position where the distance corresponding to 60% of the face width 30h is lowered from the top position 30e is used as the determination end line 48 has been described, but this may be used as the determination start line. In that case, the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 moves the undetermined pixels while moving in the negative Y-axis direction (in the reverse direction of the white arrow in FIG. 16A) from the determination start line. Explore.

[Color histogram judgment processing]
Next, the color histogram determination process executed by the image processing apparatus 100 in step S108 of FIG. 4 will be described with reference to FIGS. Here, the color histogram determination process is executed, and from the trimap 60 (FIG. 16B) generated in step S106, the trimap 70 shown in FIG. An example of generating the trimap 71 shown in (b) will be described.

FIG. 17 is a flowchart showing the flow of the color histogram determination process.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the trimap 60 generated in step S106 (step S701), and the foreground with respect to the region below the position of the submandibular line 63. Pixels belonging to the region are extracted (step S702).

The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the pixel value of the pixel of the original image 31 corresponding to the pixel extracted in step S702, and generates a color histogram (step S703).
Here, the pixel value is a value representing the color of each pixel of the original image, and is represented by a combination of hue / saturation / lightness such as RGB value or HSV color system. The horizontal axis of the color histogram is a color number indicating a color level. For example, the luminance level is represented by 0 to 255, and the vertical axis indicates the number of pixels (frequency) existing in the level.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 removes a color number with less frequency from the color histogram generated in step S703 (step S704). A frequency threshold for determining whether or not to remove is set in advance.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 includes pixels of the original image 31 corresponding to undefined pixels in the region below the position of the submandibular line 63 in the input trimap 60. A value is input (step S705).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not the pixel value of the pixel input in step S705 matches the color number existing on the color histogram generated in step S704. Determination is made (step S706).

  If they match (YES in step S706), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 updates the input pixel to the foreground area in the trimap 60 (step S707). If they do not match (NO in step S706), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 proceeds to step S708.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not all the undetermined pixels in the region below the submandibular line 63 have been input in the trimap 60 (step S708). ). If it has not been input (NO in step S708), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 returns to step S705. If the input has been completed (YES in step S708), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 ends the process.

  In step S706 of the color histogram determination process described above, it is determined whether or not the pixel value of the input pixel matches the color number existing on the generated color histogram, but it is similar to the color number (color number It is also possible to determine whether or not

FIG. 18 shows the trimaps 70 and 71 after the color histogram determination process is executed.
Compared with the tri-map 60 before the color histogram determination process is performed, the tri-map 60 has many gray portions indicating uncertain areas in the clothing portion of the subject 31a, whereas the tri-maps 70 and 71 have the clothing portion of the subject 31a. There are many white portions indicating the foreground area. As a result, it can be seen that an undetermined pixel similar to the color of the pixel updated to the foreground area by the neck area specifying process in the subject 31a area is updated to the foreground area by executing the color histogram determination process.

  The image shown in FIG. 18A is a trimap 70 generated by executing the color histogram determination process on the trimap 60 generated in step S106 without executing the submandibular line update process. The image shown in (b) is a trimap 71 generated by executing color histogram determination processing on the trimap 60 after execution of the submandibular line update processing.

  In the trimap 70, since the color histogram determination process is executed for the area below the submandibular line 62 before update, the area where the background is mixed in the hair (area 71a surrounded by a broken line) is also updated to the foreground area. Will be. On the other hand, in the trimap 71, the submandibular line 63 is below the pre-updated submandibular line 62, so the color histogram determination process is executed for the region where the background is mixed in the hair (region 71b surrounded by the broken line). Instead, it can be left as an undetermined area.

  As described above, the image processing apparatus 100 executes the color histogram determination process, and based on the color of the pixel in the foreground area in the area below the submandibular line, the undefined pixel in the area below the submandibular line. Are updated to pixels in the foreground area. As a result, it is possible to leave an area in which the pixels in the background area and the pixels in the foreground area, which are often seen in subjects with long hair, are left as undetermined areas, and an appropriate alpha mask can be generated.

[In-matrix counter processing]
Next, the in-matrix counter process executed by the image processing apparatus 100 in step S109 of FIG. 4 will be described with reference to FIGS.

FIG. 19 is a flowchart showing the flow of the in-matrix counter process.
The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the trimap generated in step S108 (step S801), and inputs an undetermined pixel (step S802).

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 stores the matrix size (pixel size) 22d stored in the storage unit 22 centered on the pixel input in step S802 on the input trimap. Is generated (step S803).

  FIG. 20A shows an example of a 3 × 3 pixel size matrix centered on the target pixel 80. “F” in the pixel indicates a pixel in the foreground area, and “U” in the pixel indicates a pixel in the undetermined area. The target pixel 80 is a pixel input in step S802.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not there is a background region pixel in the matrix generated in step S803 (step S804). If there is a pixel in the background area (YES in step S804), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 proceeds to step S808.

  If there is no pixel in the background area (NO in step S804), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 performs the pixel in the undefined area (U) in the matrix generated in step S803. The number (number of pixels) and the number of pixels (number of pixels) in the foreground area (F) are counted (step S805).

  In the example shown in FIG. 20A, the number of pixels in the undefined area (U) in the matrix is 3 pixels, and the number of pixels in the foreground area (F) is 6 pixels.

The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 compares the number of pixels in the unconfirmed region (U) with the number of pixels in the foreground region (F) (step S806), and the unconfirmed region (U). If the number of pixels is greater or equal (NO in step S806), the process proceeds to step S808.
When the number of pixels in the foreground area (F) is larger (YES in step S806), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 inputs the pixel (target pixel 80) in step S802. ) Is updated to the foreground area (F) (step S807).

  In the example shown in FIG. 20A, since the foreground area (F) is larger, the pixel of interest 80 is updated from the undetermined area (U) to the foreground area (F) in FIG. 20B. The matrix of is shown.

  The control unit 21 (trimap generation unit 13) of the image processing apparatus 100 determines whether or not all the pixels belonging to the undetermined region of the trimap have been input (step S808). If it has not been input (NO in step S808), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 returns to step S802. If the input has been completed (YES in step S808), the control unit 21 (trimap generation unit 13) of the image processing apparatus 100 ends the process.

  FIG. 21 is a diagram showing a trimap 85 of the original image 30 obtained by performing the expansion process (step S110 in FIG. 4) on the trimap after execution of the in-matrix counter process. The trimap 85 is used by the alpha mask generator 14.

[Alpha matting process]
Next, an alpha matting process (see Non-Patent Document 4) executed by the alpha mask generation unit 14 of the image processing apparatus 100 will be described with reference to FIG.

  The alpha matting process is a process for generating a foreground image of the original image 30 and an alpha mask of the original image 30 from the original image 30 and the trimap 85 of the original image 30.

  The alpha mask generation unit 14 of the image processing apparatus 100 divides the original image 30 into a foreground area, a background area, and an undetermined area based on the trimap 85. The color of each pixel in the undetermined area in the original image 30 is modeled by a color obtained by mixing the color of the pixel in the foreground area and the color of the pixel in the background area specified by the alpha matting process. Of the mixture ratios in the pixels in the undefined area, the color ratio of the pixels in the foreground area is represented by an α value (0 ≦ α ≦ 1). Hereinafter, the foreground area is denoted by Tf, the pixels belonging to the foreground area are denoted by F, the background area is denoted by Tb, the pixels belonging to the background area are denoted by B, and the undetermined area is denoted by Tu.

  The alpha mask generation unit 14 of the image processing apparatus 100 samples, for each pixel in the undetermined area, the foreground area pixel in the vicinity of the pixel and the background area pixel in the vicinity of the pixel. By substituting, the α value that is the closest to the pixel value of the pixel in the original image is adopted from the sampled combinations of (F, B).

  The alpha mask generation unit 14 of the image processing apparatus 100 generates a foreground image and an alpha mask. The foreground image is an image having the color components of the pixels in the foreground area and the color components of the pixels in the foreground area selected for each pixel in the undefined area. The alpha mask labels “1” for the pixels corresponding to the pixels belonging to the foreground area, labels “0” for the pixels corresponding to the pixels belonging to the background area, and belongs to the indeterminate area. It is an image labeled with an α value (0 ≦ α ≦ 1) calculated for a pixel corresponding to the pixel.

  First, for the pixel pεTu, the background region pixel and the foreground region pixel in the vicinity of the pixel p are sampled. Starting from the pixel p, F corresponding to an arbitrary number of samplings (hereinafter also referred to as kg) and B corresponding to kg are searched. For the pixel pεTu, the optimum combination of F and B for the pixel p is selected from the sampled combinations of F and B.

Arrows 90a, 90b, 90c, and 90d starting from the pixel pεTu shown in FIG. 22 are paths for searching for foreground and background area samples for the pixel p. The searched foreground region pixels are represented by a black four-drawn shape, and the background region pixels are represented by a black circle.
Among the pixels in the foreground area and the background area existing on the arrows 90a, 90b, 90c, and 90d, one pixel in the foreground area that is the shortest distance from the pixel p and the background area that is the shortest distance from the pixel p One pixel is sampled.

  As shown in FIG. 22, samples of the foreground area and the background area are searched for each angle corresponding to 2π / kg starting from the pixel p. The larger the value of kg, the greater the number of samplings. Therefore, although the accuracy of the combination (F, B) of the foreground region pixel and the background region finally selected for the pixel p is increased, the calculation time is increased.

  Similarly, arrows 91a, 91b, 91c, and 91d starting from the pixel qεTu shown in FIG. 22 are paths for searching for foreground and background samples for the pixel q. The searched foreground region pixels are represented by white four-drawing shapes, and the background region pixels are represented by white circles.

  Subsequently, the alpha mask generation unit 14 of the image processing apparatus 100 mixes various combinations of the sampled pixels in the foreground area and the pixels in the background area at various mixing ratios. Then, the most suitable foreground region pixel (color component) and background region pixel (color component) combination and mixing ratio for selecting the color of the pixel p are selected.

  The alpha mask generation unit 14 of the image processing apparatus 100 sets the α value of each pixel in the background region to 0, sets the α value of each pixel in the foreground region to 1, and selects the α value of each pixel in the undefined region. Then, an alpha mask that is an image having an α value as a pixel value is generated.

  FIG. 23 shows an alpha mask 95 of the original image 30 generated by the alpha matting process. The alpha mask 95 indicates pixels with α = 0 as black, pixels with α = 1 as white, and pixels with 0 <α <1 as intermediate colors.

  The alpha mask generation unit 14 of the image processing apparatus 100 generates a foreground image that is an image having the color components of the pixels in the foreground area and the color components of the pixels in the foreground area selected for each pixel in the undetermined area.

[Background composition processing]
The background image synthesis unit 15 of the image processing apparatus 100 cuts out the foreground image area from the foreground image generated by the alpha matting process and the alpha mask 95, and synthesizes the desired background image 22i. The background image composition unit 15 of the image processing apparatus 100 calculates P = α · F + (1−α) · B for each pixel (each color component) to determine a pixel value. P is the pixel value of the synthesized image, α is the α value of the alpha mask image, F is the pixel value of the foreground image, and B is the pixel value of the desired background image 22i. The background image composition unit 15 outputs the generated composite image to the image output unit 16.

  As described above, the image processing apparatus 100 can accurately detect a region even when a color similar to the background exists in the subject region of the original image or a region where the background is mixed with the hair of the subject. It is possible to automatically generate a specified trimap, thereby generating an appropriate composite image.

  Since it is not necessary to generate a trimap manually, for example, an unattended device such as an ID photo box automatically synthesizes a chroma key between the original image of the user and the desired background image of the user. It becomes possible to provide.

  The preferred embodiments of the image processing apparatus 100 and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. Understood

DESCRIPTION OF SYMBOLS 100 ......... Image processing apparatus 11 ......... Original image input part 12 ......... Subject position input part 13 ......... Tri-map generation part 14 ......... Alpha mask generation part 15 ......... Background image composition part 16 ... ...... Image output unit 21 ......... Control unit 22 ......... Storage unit 23 ......... Media input / output unit 24 ......... Communication control unit 25 ......... Input unit 26 ......... Display unit 27 ......... Peripheral equipment I / F part 30, 31, 36 ......... Original image 33, 37 ......... Binary mask image 34 ......... Edge image 35, 38 ......... Silhouette image

Claims (10)

Each pixel constituting the original image obtained by photographing the upper body of the subject is classified into a background region, a foreground region that is a part of the subject, and an uncertain region that is unknown whether it belongs to the background region or the foreground region. An image processing apparatus that generates a trimap that is a ternary image,
Silhouette specifying means for binarizing the original image and specifying the silhouette of the subject;
A pixel outside the silhouette is specified as a pixel in the background area, a pixel in the silhouette is specified as a pixel in the undetermined area, and a pixel in the skin color area of the subject is selected from the pixels in the undetermined area. Skin region specifying means for updating the region to generate the trimap,
A chest region specifying means for specifying a chest region by specifying a neck position of the subject, and updating a pixel corresponding to the chest region to the foreground region;
Color histogram determination means for determining hair length of the subject and generating color histograms in different ranges, and updating the trimap using the color histogram;
An image processing apparatus comprising: The silhouette specifying means is:
The image processing apparatus according to claim 1, wherein a silhouette is specified from a binarized image generated based on a background color and an edge image generated by performing contour extraction processing on the original image. The pixels of the undetermined region specified in the skin area specifying unit based on the background color, the threshold updating the trimap using different thresholds in the region of the region and under the above submandibular line of the object The image processing apparatus according to claim 1, further comprising a determination unit. The chest region specifying means includes
4. A neck search area is set, and the neck position is identified by comparing the number of pixels in the foreground area and the number of pixels in the undetermined area in the X-axis direction. An image processing apparatus according to any one of the above. The color histogram determination means includes
Color histogram generation that identifies the submandibular line of the subject according to the determined hair length and generates a color histogram of each pixel value of the original image corresponding to the foreground region below the identified submandibular line Means,
Determination means for determining whether or not each pixel value of the original image corresponding to the undetermined area below the submandibular line of the trimap approximates or matches a color number existing on the color histogram. When,
4. The image processing apparatus according to claim 3 , further comprising: a first update unit configured to update the pixel constituting the undetermined region to the foreground region in the trimap when the two approximate or coincide with each other. . The color histogram generation means includes
The image processing apparatus according to claim 5, wherein color numbers having a small number of pixels belonging to the color number among color numbers existing on the generated color histogram are excluded from the color histogram. For the trimap updated by the color histogram determination means, a matrix having a predetermined size centered on each pixel constituting the uncertain region is generated, and pixels existing in the generated matrix are In-matrix counter means for determining whether it belongs to the foreground area, the background area, or the undefined area, and counting the number of pixels for each area;
When the number of pixels in the foreground area counted by the in-matrix counter means is larger than the counted number of pixels in the undetermined area, the center pixel of the matrix is updated to the foreground area in the trimap. The image processing apparatus according to claim 1, further comprising: a second update unit. The second updating means includes
In the resulting said matrix of said matrix in counter means, according to claim 7 when pixels belonging to the background area is counted 1 above, characterized in that the central pixel of the matrix is specified as the undetermined region An image processing apparatus according to 1. Each pixel constituting the original image obtained by photographing the upper body of the subject is classified into a background region, a foreground region that is a part of the subject, and an uncertain region that is unknown whether it belongs to the background region or the foreground region. An image processing method for generating a trimap which is a ternary image,
A silhouette specifying step of binarizing the original image to specify the silhouette of the subject;
A pixel outside the silhouette is specified as a pixel in the background area, a pixel in the silhouette is specified as a pixel in the undetermined area, and a pixel in the skin color area of the subject is selected from the pixels in the undetermined area. A skin region specifying step of updating to a region and generating the trimap;
A chest region specifying step of specifying a chest region by specifying a neck position of the subject, and updating a pixel corresponding to the chest region to the foreground region;
A color histogram determination step of determining the hair length of the subject to generate color histograms in different ranges, and updating the trimap using the color histogram;
An image processing method comprising: Computer, each pixel constituting the original image obtained by photographing the upper body of the subject, a background region, a foreground region that is a part of the subject, an uncertain region that is unknown whether it belongs to the background region or the foreground region A program for functioning as an image processing device for generating a trimap which is a ternary image classified into
Silhouette specifying means for binarizing the original image and specifying the silhouette of the subject;
A pixel outside the silhouette is specified as a pixel in the background area, a pixel in the silhouette is specified as a pixel in the undetermined area, and a pixel in the skin color area of the subject is selected from the pixels in the undetermined area. Skin region specifying means for updating the region to generate the trimap,
A chest region specifying means for specifying a chest region by specifying a neck position of the subject and updating a pixel corresponding to the chest region to the foreground region;
Color histogram determination means for determining the length of the hair of the subject and generating color histograms in different ranges, and updating the trimap using the color histogram;
Program to function as.

Images