JP5732799B2 - Image processing apparatus, imaging apparatus, and image processing program - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  In the white balance adjustment disclosed in Patent Document 1, a main subject region is extracted from a captured image, and the frequency of colorimetric values extracted from the main subject region is set to be higher than the frequency of colorimetric values extracted from other regions. By making a high estimate (multiplying by a large weighting factor), the accuracy of white balance adjustment for the main subject is improved.

  In the white balance adjustment disclosed in Patent Document 2, the white balance adjustment amount for the main subject region is set separately from the white balance adjustment amount for the other regions, thereby improving the accuracy of the white balance adjustment for the main subject. .

Japanese Patent Laid-Open No. 2006-2111416 JP 2008-219414 A

  However, in any case, when it is difficult to estimate the illumination color of the main subject from the color of the main subject area (for example, when the original color of the main subject accidentally resembles the color of some light source) There is a risk that the white balance adjustment of the main subject area may fail (cause color failure).

  It is an object of the present invention to provide an image processing apparatus, an imaging apparatus, and an image processing program that can suppress a white balance adjustment color failure for a specific area in a captured image with high probability.

The image processing apparatus according to the present invention includes an input unit that inputs a color photographed image, and an evaluation value that indicates a high possibility that the color of the area indicates the light source color from each of the plurality of areas on the image. Extraction means for extracting, estimation means for estimating the illumination direction of a subject reflected in a specific area that is a part of the plurality of areas, and distribution of the plurality of extracted evaluation values on the image, A weighting unit for weighting with a weight distribution according to the illumination direction, and a white balance adjustment amount to be applied to the specific region by regarding the color of the extraction source region of the evaluation value showing a high value after weighting as the illumination color of the subject comprising a calculation unit calculating for a said object is reflected in a particular region according to whether the person, the selection criteria of the areas to estimate the illumination direction you wherein different.

  The imaging apparatus of the present invention includes an imaging unit that acquires a color captured image, and an image processing apparatus of the present invention that calculates a white balance adjustment amount to be applied to a specific area of the image acquired by the imaging unit.

The image processing program of the present invention includes an input procedure for inputting a color photographed image, and an evaluation value indicating a high possibility that the color of the area indicates the light source color from each of the plurality of areas on the image. An extraction procedure for extracting a plurality of evaluation values, an estimation procedure for estimating an illumination direction of a subject in a specific area that is a part of the plurality of areas, and a distribution of the extracted evaluation values on the image. A weighting procedure for weighting with a weight distribution according to the illumination direction, and a white balance adjustment amount to be applied to the specific area by regarding the color of the extraction source area of the evaluation value showing a high value after weighting as the illumination color of the subject The computer is caused to execute a calculation procedure for calculating and a procedure for changing a selection criterion of the region for estimating the illumination direction according to whether or not the subject in the specific region is a person .

  According to the present invention, an image processing apparatus, an imaging apparatus, and an image processing program that can suppress a white balance adjustment color failure for a specific region in a color image with high probability are realized.

The block diagram of an electronic camera. The operation | movement flowchart of CPU21 in imaging | photography mode. 6 is an operation flowchart of white balance adjustment processing by the white balance adjustment circuit 15; Detailed image example (for oblique shooting). The schematic diagram of the area | region which divided | segmented the detailed image. The figure explaining a light source likelihood degree map. The figure explaining main subject field (here person field o). The figure explaining a reference field (here face field of). The figure explaining the reference area (here face area of ') after binarization. The figure explaining division | segmentation curve Cs. The figure explaining the brightness gravity center gh and the dark gravity center gs. The figure explaining a light-dark vector. 6A and 6B illustrate gradation conversion characteristics. 10A and 10B are diagrams illustrating an effect of gradation conversion characteristics. The figure explaining a weight map. The figure explaining step S16-6 to S16-9 (in the case of oblique photographing). The figure explaining step S16-12 to S16-9 (in the case of front light photography). The figure explaining step S16-15 to S16-9 (in the case of backlight photography).

  Hereinafter, an electronic camera will be described as an embodiment of the present invention.

  FIG. 1 is a configuration diagram of an electronic camera. As shown in FIG. 1, the electronic camera 1 includes an image sensor 11, a signal processing circuit 12, a buffer memory 13, an image processing circuit 14, a white balance adjustment circuit 15, a display control circuit 16, an internal monitor 17, a compression / expansion circuit 18, A card interface 19, a CPU 21, an imaging circuit 22, an input device 23, an external connection terminal 24, a strobe device 30 and the like are provided. Among these, the buffer memory 13, the image processing circuit 14, the white balance adjustment circuit 15, the display control circuit 16, the compression / expansion circuit 18, the card interface 19, and the CPU 21 are connected to a common bus.

  The image sensor 11 is a color image sensor (for example, a single plate type color image sensor) that captures an object scene image formed by the photographing lens 2.

  The signal processing circuit 12 sequentially performs analog signal processing and A / D conversion processing on the analog image signal acquired by the imaging device 11 through imaging, and converts the analog image signal into a digital image signal.

  The buffer memory 13 sequentially accumulates digital image signals output from the signal processing circuit 12. When this accumulation is performed over the charge reading period for one frame of the image sensor 11, digital image data for one frame is accumulated in the buffer memory 13. Hereinafter, digital image data for one frame is simply referred to as “image”, and in particular, an image stored in the buffer memory 13 immediately after imaging (an image before being subjected to image processing by the image processing circuit 14) is referred to as a “RAW image”. ".

  The white balance adjustment circuit 15 performs white balance adjustment processing on the RAW image stored in the buffer memory 13. Details of the white balance adjustment processing will be described later.

  The image processing circuit 14 performs predetermined image processing (color interpolation processing, gradation conversion processing, edge enhancement processing, contrast enhancement processing, etc.) including debayer processing (color interpolation processing) on the image after white balance adjustment. . Note that the image processing circuit 14 can also perform display thinning processing (size reduction processing) on the image on the buffer memory 13 as necessary.

  The display control circuit 16 has a display memory, and displays an image on the internal monitor 17 by sending the image written in the display memory to the internal monitor 17. Note that during the period in which an external monitor is connected to the external connection terminal 24, the display destination of the image by the display control circuit 16 is not the internal monitor 17 but the external monitor. However, for simplicity, it is assumed that an external monitor is not connected to the external connection terminal 24.

  The internal monitor 17 is a monitor that is provided on the back surface of the electronic camera 1 and used when a user confirms an image acquired by the electronic camera 1. This monitor is composed of, for example, an LCD panel.

  The external connection terminal 24 is a terminal for connecting the display control circuit 16 to an external monitor. The external connection terminal 24 is configured by, for example, a USB port.

  The compression / decompression circuit 18 performs data compression processing of a predetermined method on the image after the predetermined image processing described above. Further, the compression / decompression circuit 18 can also perform the same type of data expansion processing on the data-compressed image. Hereinafter, it is assumed that a well-known JPEG method is applied as the compression / decompression method.

  The card interface 19 writes a data-compressed image (JPEG file) or a non-data-compressed RAW image (RAW image file) to the card memory 20 installed in the card slot of the electronic camera 1, or the card memory 20 Or a JPEG file or a RAW image file written in the. The card memory 20 is a portable storage medium.

  The imaging circuit 22 adjusts the focal length (zoom position) of the photographic lens 2, the focusing distance of the photographic lens 2, the aperture value of the photographic lens 2, and the like by giving a drive signal to the photographic lens 2. Further, the imaging circuit 22 controls the drive timing of both by providing a drive signal to the image sensor 11 and the signal processing circuit 12. The imaging circuit 22 controls the charge accumulation time per frame of the imaging device 11. Note that the shutter speed of the electronic camera 1 can be set by a combination of the charge accumulation time of the image sensor 11 and the opening period of a mechanical shutter (not shown), but here it is set only by the charge accumulation time of the image sensor 11 for simplicity. Assume that

  The CPU 21 controls each of the above units in accordance with a user instruction input via the input device 23 and firmware written in advance in the internal memory (ROM) of the CPU 21. The firmware of the CPU 21 is appropriately updated via a communication circuit (not shown) (known network terminal or the like) mounted on the electronic camera 1.

  The input device 23 is an operation member such as a release button provided on the top of the electronic camera 1 or a multi-selector provided on the back of the electronic camera 1. For example, the user can switch the mode of the electronic camera 1 between the shooting mode and the playback mode via the input device 23. The user can also set the mode of the electronic camera 1 to the flash emission prohibition mode via the input device 23.

  The strobe device 30 is a device that emits flash light toward the object field of the photographing lens 2. The flash emission timing of the strobe device 30 is controlled by the imaging circuit 22 together with the driving timing of the imaging element 11 and the like. Further, the necessity of flash emission by the flash device 30 is determined by the CPU 21 for each photographing.

  FIG. 2 is an operation flowchart of the CPU 21 in the photographing mode. Hereafter, each step of FIG. 2 is demonstrated in order. Here, for the sake of simplicity of explanation, it is assumed that the RAW recording function (function of recording a RAW image file together with a JPEG file) of the electronic camera 1 is turned off.

  Step S10: The CPU 21 starts to continuously drive the image pickup device 11 in the draft mode via the image pickup circuit 22. As a result, the image sensor 11 starts acquiring a through image. The CPU 21 extracts the AE evaluation value from the through image output from the signal processing circuit 12 at this time, and the shutter speed of the electronic camera 1 and the photographing lens 22 through the imaging circuit 22 so that the AE evaluation value becomes appropriate. The combination with the aperture value is controlled (AE control). In parallel with this, the CPU 21 extracts the AF evaluation value from the through image output from the signal processing circuit 12, and sets the in-focus distance of the photographic lens 22 via the imaging circuit 22 so that the AF evaluation value is appropriate. Control (AF control). Note that when a zoom adjustment instruction is input from the input device 23 during the through image acquisition period, the CPU 21 adjusts the zoom position of the photographing lens 2 in accordance with the instruction.

  Step S11: The CPU 21 determines whether or not the release button has been half-pressed. If the release button has been half-pressed, the process proceeds to step S12.

  Step S12: The CPU 21 fixes the current aperture value and shutter speed (AE lock), fixes the current in-focus distance (AF lock), and also fixes the current zoom position (zoom lock). However, if the electronic camera 1 is not set to the flash prohibition mode, the CPU 21 detects whether or not there is a possibility of underexposure at this time, and if there is such a possibility, the strobe device 30 is detected. Are prepared for light emission, and the aperture value and shutter speed are adjusted to values suitable for flash photography. Thereafter, the CPU 21 reads data such as the aperture value, shutter speed, focusing distance, zoom position, presence / absence of flash light emission preparation (flash light emission presence / absence) set in the electronic camera 1 as shooting condition data.

  Step S13: The CPU 21 determines whether or not the release button has been fully pressed. If the release button has not been fully pressed, the process proceeds to step S14. If the release button has been fully pressed, the process proceeds to step S15.

  Step S14: The CPU 21 determines whether or not the half-press of the release button has been released. If not released, the process proceeds to step S13. If released, the AE lock, AF lock, zoom lock, flash emission is performed. After canceling each of the preparations, the process proceeds to step S11.

  Step S15: The CPU 21 drives the image pickup device 11 for one frame in the frame mode via the image pickup circuit 22. If it is determined in the previous step that flash emission is necessary, the CPU 21 drives the strobe device 30 via the imaging circuit 22 and performs flash emission along with the driving of the image sensor 11. As a result, a detailed image for storage is acquired. This detailed image is stored in the buffer memory 13 via the signal processing circuit 12.

  Step S16: The CPU 21 gives the photographing condition data (at least data indicating the presence or absence of flash emission) read in Step S12 to the white balance adjustment circuit 15 and performs a white balance adjustment process on the detailed image. When the CPU 21 gives an instruction to 15 and then receives a completion notification from the white balance adjustment circuit 15, the CPU 21 proceeds to step S17.

  Step S17: The CPU 21 performs predetermined image processing (color interpolation processing, gradation conversion processing, edge enhancement processing, contrast enhancement processing, etc.) by the image processing circuit 14 on the detailed image after white balance adjustment.

  Step S18: The CPU 21 performs thinning processing for display (size reduction processing by the image processing circuit 14) on the detailed image after the image processing, and then writes it in the display memory of the display control circuit 16, and then on the internal monitor 17. Is displayed for a predetermined period. As a result, the user can confirm the detailed image acquired by photographing.

  Further, the CPU 21 in this step creates a JPEG file by performing data compression processing by the compression / expansion circuit 18 on the detailed image after white balance adjustment (detailed image before thinning processing), and the shooting conditions read in step S12 After the data is attached to the JPEG file, the data is written to the card memory 20 via the card interface 19, and the flow is terminated. As a result, the detailed image is saved.

  FIG. 3 is an operation flowchart of white balance adjustment processing (step S16 in FIG. 2) by the white balance adjustment circuit 15. Hereafter, each step of FIG. 3 is demonstrated in order.

  Step S16-1: The white balance adjustment circuit 15 reads a detailed image (a detailed image before image processing) on the buffer memory 13.

  Step S16-2: The white balance adjustment circuit 15 detects a plurality of object areas existing in the detailed image by performing pattern recognition processing on the read detailed image. Then, the white balance adjustment circuit 15 divides the entire detailed image into a plurality of regions based on the contours of the object regions. For example, in the detailed image, as shown in FIG. 4, indoor lighting fixtures (shades, lamps, arms), persons (hair, face, body), windows with mountains, sun and sky, walls, floors In this step, the detailed image is divided into regions a, b, c, d, e, f, g, h, i, j, k, and o as shown in FIG.

  Step S16-3: The white balance adjustment circuit 15 is a kind of color evaluation value from each of the areas a, b, c, d, e, f, g, h, i, j, k, o on the detailed image. A certain “lightness likelihood” is extracted.

  Here, the lightness likelihood of a certain area is an evaluation value indicating the high possibility that the color of the area indicates a certain light source color. For example, when the luminance distribution and chromaticity distribution in a certain area are similar to those of a fluorescent lamp, the degree of light source in that area is a high value. In addition, when the luminance distribution and chromaticity distribution of another certain area are similar to those of a light bulb, the degree of lightness in that area is a high value. In addition, when only the chromaticity of another certain area is close to the chromaticity of the light bulb, the degree of light source in that area becomes a medium value. In addition, when neither the luminance distribution nor the chromaticity distribution in another certain area is similar to that of the light source, the degree of lightness in that area is a low value.

  Therefore, the degree of light source likelihood of each of the areas a, b, c, d, e, f, g, h, i, j, k, and o in FIG. 4 is as shown in FIG. 6, for example. In the example shown in FIG. 6, the lamp area f has a light source likelihood of +4, the shade area g has a light source likelihood of +3, and the solar area a has a light source likelihood of +5, and the other areas c, d, e , H, i, j, k, o have a light source likelihood of +1. Hereinafter, the distribution of the degree of light source likelihood on the detailed image as shown in FIG. 6 is referred to as a “light source degree of degree map”.

  The calculation of the light source likelihood may be performed based on the object recognition result for each object area, that is, whether each object area is a light source object (for example, illumination, sun, fire, etc.). Alternatively, it may be performed based on a power distribution of an input signal to a sub image sensor (photometric sensor) provided separately from the main image sensor 11. Alternatively, it may be performed based on a combination of the object recognition result and the power distribution.

  Subsequently, the white balance adjustment circuit 15 compares the regions a, b, c, d, e, f, g, h, i, j, k, o on the detailed image with each other according to the pattern and area of the region, Any one of these areas a, b, c, d, e, f, g, h, i, j, k, o is selected as the main subject area. For example, the white balance adjustment circuit 15 excludes the areas a, b, c, d, e, f, g, h, i, j, k, and o whose area is less than the threshold, and the remaining plural The region having the highest contrast among the regions is defined as the main subject region. The selection of the main subject area may be performed based on the above-described AF control information (such as the position of the AF area from which the AF evaluation value is extracted).

  Further, the white balance adjustment circuit 15 divides the selected main subject area into a plurality of small areas having different chromaticities, excludes those areas whose area is less than the threshold value, and most of the remaining areas. A small area with high contrast is set as a reference area.

  However, when the selected main subject area is the person area o, the white balance adjustment circuit 15 sets the skin color area (that is, the face area of) of the person area o as the reference area regardless of the area and contrast. (FIG. 7). Incidentally, the symbol oh in FIG. 7 is a hair region, and the symbol ob is a body region. Hereinafter, it is assumed that the person area o is selected as the main subject area and the face area of is set as the reference area.

  Step S16-4: The white balance adjustment circuit 15 extracts the light / dark difference and the light / dark vector from the reference area (here, the face area of) on the detailed image by the following procedures (a) to (f).

  (A) The reference area (here, the face area of) (FIG. 8) on the detailed image is binarized with a predetermined luminance threshold. Each pixel in the binarized reference area (here, the face area of ') is either a black pixel or a white pixel (FIG. 9).

  (B) A separation curve Cs of a predetermined order that spatially separates the black pixel group and the white pixel group on the binarized reference area (here, the face area of ') is calculated. The separation curve Cs is, for example, a curve that maximizes the total of the Euclidean distance from the black pixel to the separation curve Cs and the Euclidean distance from the white pixel to the separation curve Cs. For the calculation of the separation curve Cs, for example, a method similar to the calculation of the separation hyperplane in the support vector machine can be applied. In order to reduce the amount of calculation, it is assumed here that the order of the separation curve Cs is set to about 2, for example. In this case, there is a possibility that a white pixel (enclave) exists also on the black pixel group side of the separation curve Cs, and a possibility that a black pixel (enclave) exists also on the white pixel group side of the separation curve Cs. There is.

  (C) The reference area before binarization (here, the face area of) is divided by the separation curve Cs (FIG. 10). As a result, the reference area (here, the face area of) is divided into a bright part AH containing many bright pixels and a dark part AS containing many dark pixels. In the following calculation, each pixel corresponding to the above-described enclave in the bright part AH and the dark part AS is treated as an invalid pixel (a pixel not used in the calculation).

  (D) The difference (IH−IS) between the average brightness IH of the bright part AH and the average brightness IS of the dark part AS is calculated as the brightness difference of the reference area (here, the face area of).

  (E) The brightness centroid gh in the bright part AH and the dark centroid gs in the dark part AS are calculated (FIG. 11). The brightness centroid is a weighted average coordinate to which a higher weight is assigned to a pixel with higher luminance, and the dark centroid is a weighted average coordinate to be assigned a higher weight to a pixel with lower luminance. is there.

  (F) A unit vector in the direction from the brightness centroid gh to the dark centroid gs is calculated as the brightness vector of the reference area (here, the face area of) (FIG. 12).

  Step S16-5: The white balance adjustment circuit 15 determines whether or not the brightness difference of the reference area (here, the face area of) exceeds a predetermined threshold value. If it is not oblique shooting, the process proceeds to step S16-6. If not, the process proceeds to step S16-11 because it is assumed that the detailed image has been captured in backlight or backlight.

  Therefore, for example, as shown in FIG. 4, when the main subject (here, a person) is illuminated mainly from an oblique direction with a lamp, a shadow is generated only in a part of the reference area (here, the face area of). The difference becomes large, and it is considered that the detailed image was captured obliquely.

  On the other hand, for example, as shown in FIG. 17A (details will be described later), when the main subject (here, a person) is mainly illuminated from the front with flash light, a shadow is generated in the reference area (here, the face area of). Therefore, the difference in brightness is reduced, and it is considered that the detailed image has been captured in the backlight or backlight mode.

  Further, for example, as shown in FIG. 18A (details will be described later), when the main subject (here, a person) is mainly illuminated from behind with natural light, almost the entire reference area (here, the face area of). Since the shadow becomes a shadow, the difference in brightness becomes small, and it is considered that the detailed image was captured in the backlight or backlight mode.

  Step S16-6: The white balance adjustment circuit 15 determines the illumination direction of the object (here, a person) existing in the main subject area (here, the person area o) and the direction of the brightness vector of the reference area (here, the face area of). Is considered the same.

  Step S16-7: The white balance adjustment circuit 15 outputs information necessary for adjusting the brightness of the main subject area (here, the person area o) to the CPU 21. Receiving it, the CPU 21 creates tone conversion parameters (tone conversion region and tone conversion characteristics) based on the information and sets them in the tone conversion processing unit of the image processing circuit 14. This gradation conversion characteristic is, for example, a conversion characteristic as shown in FIG. 13, and the main subject is maintained while maintaining the luminance of the pixels belonging to the bright luminance range in the main subject region (here, the human region o). The characteristic is that the luminance of the pixels belonging to the dark luminance range in the region (here, the human region o) is raised to the bright luminance range.

  Therefore, when the detailed image is captured by shading, the shadowed portion of the main subject region (here, the human region o) is shown in FIG. 14 by the gradation conversion processing (step S17) executed later. The brightness is corrected as shown in. Here, the brightness adjustment target is the entire main subject region (here, the human region o), but may be limited to only the reference region (here, the facial region of).

  Step S16-8: The white balance adjusting circuit 15 creates a weight map (FIG. 15) for oblique photographing by the following procedures (g) to (h). Here, the weight map (FIG. 15) is a map for weighting the light source likelihood map (FIG. 6).

  (G) The entire map is divided into a plurality of fan-shaped areas by a plurality of dividing lines extending radially from the center point Po of the main subject area (here, the person area o) (see FIG. 15). However, the main subject area (here, the person area o) is not divided by those dividing lines, and remains as one area. Further, the arrangement relationship of the dividing lines is symmetric with respect to the light / dark vector Bo toward the center point Po, and all the dividing lines are arranged so as not to overlap the light / dark vector Bo.

  (H) A weight value is assigned to each of the plurality of fan-shaped areas and the main subject area (here, the person area o) on the map. At this time, a relatively large weight value (+0.3 to +1) is assigned to the plurality of fan-like regions located on the upstream side in the illumination direction (upper left half in the figure) when viewed from the center point Po, and from the center point Po. A relatively small weight value (0) is assigned to a plurality of fan-shaped regions located on the downstream side in the illumination direction as viewed (lower right half in the figure). In particular, the maximum weight value (+1) is assigned to a specific fan-shaped area (upper-left fan-shaped area) located immediately upstream in the illumination direction when viewed from the center point Po, and the specific fan-shaped area (upper left) A larger weight value is assigned to a fan-shaped area located near the fan-shaped area (a fan-shaped area closer to the circumferential direction). Further, the maximum weight value (+1) is assigned to the main subject area (here, the person area o) on the map.

  Here, although a common weight value (0) is assigned to the five fan-shaped regions located on the downstream side in the illumination direction (lower right half in the figure) when viewed from the center point Po, individual weight values are assigned. It may be assigned. In that case, a larger weight value is assigned to a fan-shaped area located near a specific fan-shaped area (upper-left fan-shaped area).

  Step S16-9: The white balance adjustment circuit 15 weights the light source likelihood map (FIG. 6) with the weight map (FIG. 15), and obtains a weighted light source likelihood map. At this time, the weight value multiplied by the light source likelihood degree of each area of the light source likelihood degree map is, for example, the weight value assigned to the area (fan-shaped area or main subject area) to which the center coordinates of the area belong on the weight map. Is set the same as

  FIGS. 16A, 16B, 16C, and 16D visualize the function of the weight map used in this step. FIG. 16A is a detailed image (in the case of oblique shooting), FIG. 16B is a light source likelihood degree map before weighting, FIG. 16C is a weight map, and FIG. It is a lightness-likeness degree map after weighting.

  In other words, the weight map (FIG. 16C) used in this step (when the detailed image is captured obliquely) has the illumination direction as viewed from the main subject area (here, the person area o). Suppressing the likelihood of a light source in an area (for example, the sun area a) located on the downstream side (lower right half) and located on the upstream side (upper left half) in the illumination direction when viewed from the main subject area (here, the person area o) There is a function of relatively improving the light source likelihood of the area to be performed (for example, the lamp area g and the shade area g).

  Step S16-10: The white balance adjustment circuit 15 finds one or a plurality of areas having the highest light source likelihood from the weighted light source likelihood degree map, and illuminates the main subject (here, a person) in the area. It is determined that the light source exists (hereinafter, this region is referred to as a “light source region”). Subsequently, the white balance adjustment circuit 15 sets a white balance gain (or a white balance gain for achromatic color) for making the average chromaticity of the light source region close to an achromatic color, and a main subject region (here, a human region o). Is calculated as an optimal white balance gain.

  In the examples shown in FIGS. 16A, 16B, 16C, and 16D (oblique shooting), the light source likelihood after weighting has the highest value in the lamp region f. A white balance gain for bringing the average chromaticity closer to an achromatic color (or a white balance gain for achromatic color) is calculated as an optimum white balance gain for the main subject area (here, the person area o).

  However, when the maximum value of the light source likelihood after weighting is less than a threshold value (for example, less than +2), the white balance adjustment circuit 15 further selects a specific color from one or more regions to which the highest value belongs. A white balance gain (or a white balance gain) for extracting a partial area where an object (for example, an achromatic color object or a human face) is present and making the average chromaticity of the partial area close to the original color of the object The white balance gain for obtaining the original color is calculated as the optimum white balance gain for the main subject area (here, the person area o).

  The white balance control circuit 15 multiplies each pixel of the detailed image by the calculated white balance gain to obtain a detailed image after white balance adjustment, gives a completion notification to the CPU 21, and ends the flow of FIG. (Note that the image processing is performed on the detailed image after the end of the flow of FIG. 3).

  Step S16-11: The white balance adjustment circuit 15 determines whether or not the detailed image has been shot with flash photography based on the shooting condition data given from the CPU 21. The process proceeds to step S16-12 on the assumption that the photographing is a continuous light photographing, and if it is not the flash photographing, the photographing of the detailed image is regarded as the backlight photographing, and the process proceeds to step S16-14.

  Therefore, for example, as shown in FIG. 17A, when the main subject (here, a person) is mainly illuminated from the front with flash light, it is considered that the detailed image has been captured in the front light. For example, FIG. As shown in A), when the main subject (here, a person) is illuminated mainly by natural light from behind, it is considered that the shooting of the detailed image is the backlight shooting.

  Step S16-12: The white balance adjustment circuit 15 relates the illumination direction of the object (here, a person) existing in the main subject area (here, the person area o) to the brightness vector of the reference area (here, the face area of). It is regarded as the forward direction (the direction from the front to the back).

  Step S16-13: The white balance adjustment circuit 15 creates a weight map for follow light photography by the following procedures (g) to (h), and proceeds to step S16-9.

  (G) The entire map is divided into a main subject area (here, a person area o) and a background area other than that.

  (H) A relatively high weight value (+1) is assigned to the main subject area (here, the person area o) on the map, and a relatively low weight value (0) is assigned to the background area on the map.

  Note that FIGS. 17A, 17B, 17C, and 17D show the function of the weight map created in this step. FIG. 17A is a detailed image (in the case of direct light imaging), FIG. 17B is a light source likelihood degree map before weighting, FIG. 17C is a weight map, and FIG. Is a weighted lightness likelihood degree map.

  That is, in the weight map created in this step (FIG. 17C), the degree of light source in the background area is suppressed, and the degree of light source in the main subject area (here, the person area o) is relatively improved. There is a good work.

  Step S16-14: The white balance adjustment circuit 15 determines whether or not the pattern of the main subject area (here, the person area o) or the reference area (here, the face area of) of the detailed image has a backlight photographing feature. If so, the process proceeds to step S16-15, and if not, the detailed image is captured in the following light. It moves to step S16-12 mentioned above.

  Step S16-15: The white balance adjustment circuit 15 relates the illumination direction of the object (here, a person) existing in the main subject area (here, the person area o) to the brightness vector of the reference area (here, the face area of). It is regarded as the backlight direction (the direction from the heel to the front).

  Step S16-16: The white balance adjustment circuit 15 outputs information necessary for adjusting the brightness of the main subject area (here, the person area o) to the CPU 21. Receiving it, the CPU 21 creates tone conversion parameters (tone conversion region and tone conversion characteristics) based on the information and sets them in the tone conversion processing unit of the image processing circuit 14. This gradation conversion characteristic is, for example, the conversion characteristic shown in FIG. 13, and maintains the luminance of the pixels belonging to the bright luminance range in the main subject region (here, the human region o) while maintaining the main subject region ( Here, the characteristic is such that the luminance of the pixels belonging to the dark luminance range in the human region o) is raised to the bright luminance range.

  Therefore, when the detailed image is captured by backlighting, the shadowed portion (backlighting) of the main subject region (here, the person region o) is obtained by a gradation conversion process (step S17) executed later. In the case of, most of the person) is corrected brightly. Here, the brightness adjustment target is the entire main subject region (here, the human region o), but may be limited to only the reference region (here, the facial region of).

  Step S16-17: The white balance adjustment circuit 15 creates a weight map for backlight shooting according to the following procedures (g) to (h), and proceeds to step S16-9.

  (G) The entire map is divided into a main subject area (here, a person area o) and other background areas.

  (H) A relatively low weight value (0) is assigned to the main subject area (here, the person area o) on the map, and a relatively high weight value (+1) is assigned to the background area on the map.

    18A, 18B, 18C, and 18D show the function of the weight map created in this step. 18A is a detailed image (in the case of backlight photographing), FIG. 18B is a light source likelihood degree map before weighting, FIG. 18C is a weight map, and FIG. 18D is FIG. It is a lightness-likeness degree map after weighting.

  That is, the weight map (FIG. 18C) created in this step suppresses the light source likelihood of the main subject region (here, the human region o) and relatively improves the light source likelihood of the background region. (Therefore, description of step S16-17.)

  As described above, the white balance adjustment circuit 15 of the present embodiment uses the weight map (FIGS. 16C, 17C, and 18C) reflecting the illumination direction of the main subject (here, a person) to generate a light source. After the weighting degree maps (FIG. 16B, FIG. 17B, and FIG. 18B) are weighted, the white balance gain is calculated.

  Each of these weight maps (FIG. 16C, FIG. 17C, and FIG. 18C) is an area located upstream of the illumination direction as viewed from the main subject area (here, the person area o). There is a function of relatively improving the light source likelihood and relatively suppressing the light source likelihood of the region located on the downstream side.

  Therefore, the white balance adjustment circuit 15 according to the present embodiment, regardless of the direction in which the light source that illuminates the main subject (here, a person) is seen from the main subject area (here, the person area o), It is possible to reliably calculate the optimum white balance gain for the main subject area (here, the person area o).

  In addition, the white balance adjustment circuit 15 of the present embodiment considers whether the detailed image shooting is oblique shooting, backlight shooting, or backlight shooting when estimating the illumination direction of the main subject (here, a person). In addition, when the detailed image is captured obliquely, the brightness vector of the reference area (here, the face area of) in the main subject area (here, the person area o) is considered. Therefore, the white balance adjustment circuit 15 of this embodiment can cope with various shooting scenes with different illumination directions.

  In the white balance adjustment circuit 15 of the present embodiment, the application destination of the white balance gain calculated in step S16-10 is the entire area of the detailed image, but it may be only the main subject area (here, the person area o). However, in that case, it is necessary to separately calculate the white balance gain applied to the background region. In this case, the white balance gain applied to the background area may be calculated in the same manner as the white balance gain applied to the main subject area (here, the person area o). That is, the above-described white balance gain calculation procedure can be applied to regions other than the main subject region.

  Further, in FIG. 15 (weight map) described above, the arrangement pitch of the dividing lines in the circumferential direction is set to 30 °, but the arrangement pitch of the dividing lines in the circumferential direction may be set to a value other than 30 °. For example, in order to simplify the calculation, it may be set roughly as 90 °, 180 °, or the like.

  Further, in FIG. 15 described above (weighted map for oblique shooting), the arrangement pitch of the dividing lines in the circumferential direction is made uniform, but it may be uneven. For example, the arrangement pitch on the upstream side in the illumination direction (upper left half in the figure) may be set finely, and the arrangement pitch on the downstream side in the illumination direction (lower right half in the figure) may be set coarsely (refer to the example in FIG. 15). The arrangement pitch is uniform, but since the common weight value is assigned to the plurality of downstream fan-shaped regions, the same effect as when the arrangement pitch on the downstream side is made coarse is obtained.

  Further, in FIG. 15 described above (weight map for oblique photographing), the weight value is assigned to each region, so the distribution of the weight value is stepped, but the weight value is set for each position (for example, for each circumferential position). By assigning, the distribution of weight values may be smoothed. However, it is preferable to assign a weight value to each region because the calculation required for weighting can be simplified.

  In addition, the white balance adjustment circuit 15 in step S16-2 divides the detailed image into a plurality of regions having different sizes and shapes, but the detailed image is divided into a plurality of regions (for example, a plurality of rectangles having the same size and shape). It may be divided into (regions).

  Further, the white balance adjustment circuit 15 of the present embodiment uses the detailed image itself as the image used for calculating the white balance gain. However, in order to reduce the amount of calculation required for the calculation, the detailed image is used instead of the detailed image. It is good also as a size reduction version. In addition, when the electronic camera 1 is provided with a sub image sensor (photometric sensor) that monitors the chromaticity distribution (and luminance distribution) of the object field separately from the main image sensor 11, the white balance gain The image used for the calculation may be an image acquired by the sub image sensor.

  In the electronic camera 1 according to the present embodiment, the white balance adjustment is performed at the time of photographing. However, the same white balance adjustment may be performed at the time of reproducing the RAW image (as one type of RAW image development processing). In this case, the determination in step S16-11 may be performed based on the shooting condition data added to the RAW image file.

  In addition, the present embodiment is an embodiment of an electronic camera, but the operation of the electronic camera related to white balance adjustment is an image processing apparatus provided separately from the electronic camera, such as an electronic photo frame, a printer, It may be executed by a computer or the like. When the general-purpose computer executes the operation, the computer program for the operation is installed in the computer via a communication network or a storage medium.

DESCRIPTION OF SYMBOLS 11 ... Image sensor, 12 ... Signal processing circuit, 13 ... Buffer memory, 14 ... Image processing circuit, 15 ... White balance adjustment circuit, 16 ... Display control circuit, 17 ... Internal monitor, 18 ... Compression / decompression circuit, 19 ... Card Interface, 21 ... CPU, 22 ... Imaging circuit, 23 ... Input device, 24 ... External connection terminal, 30 ... Strobe device

Claims (7)

An input means for inputting a color photographed image;
Extraction means for extracting an evaluation value indicating the high possibility that the color of the region indicates the light source color from each of the plurality of regions on the image;
A guessing means for guessing the illumination direction of a subject in a specific area that is a part of the plurality of areas;
Weighting means for weighting the distribution of the plurality of extracted evaluation values on the image with a weight distribution according to the illumination direction;
A calculating means for calculating a white balance adjustment amount to be applied to the specific area by regarding the color of the extraction source area of the evaluation value showing a high value after weighting as the illumination color of the subject;
Equipped with a,
2. An image processing apparatus according to claim 1, wherein a selection criterion for the area for estimating the illumination direction differs depending on whether or not the subject in the specific area is a person . The image processing apparatus according to claim 1.
The inference means is
An image processing apparatus characterized in that a bright and dark direction from a bright part to a dark part in the specific region is regarded as the illumination direction. The image processing apparatus according to claim 2,
The inference means is
Regardless of the illumination direction, when the brightness difference between the bright part and the dark part in the specific area is less than a threshold value, the forward light direction from the front to the back of the image is regarded as the illumination direction. Processing equipment. The image processing apparatus according to claim 3.
The inference means is
Even if the brightness difference between the bright part and the dark part in the specific area is less than the threshold value, if the image is captured by backlighting, the backlight direction from the back to the front of the image is referred to as the illumination direction. An image processing apparatus characterized by being regarded. In the image processing device according to any one of claims 1 to 4,
The weighting means is
An image processing apparatus, wherein a weight on the upstream side in the illumination direction is set higher than a weight on the downstream side in the illumination direction. Imaging means for acquiring a color photographed image;
The image processing apparatus according to any one of claims 1 to 5, wherein a white balance adjustment amount to be applied to a specific area of an image acquired by the imaging unit is calculated;
An imaging apparatus comprising: Input procedure to input color shot image,
From each of the plurality of regions on the image, an extraction procedure for extracting an evaluation value indicating a high possibility that the color of the region indicates a light source color;
A guessing procedure for guessing the illumination direction of a subject in a specific area that is a part of the plurality of areas;
A weighting procedure for weighting a distribution on the image of the plurality of extracted evaluation values with a weight distribution according to the illumination direction;
A calculation procedure for calculating a white balance adjustment amount to be applied to the specific region by regarding the color of the extraction source region of the evaluation value indicating a high value after weighting as the illumination color of the subject;
A procedure for changing a selection criterion of an area for estimating the illumination direction according to whether or not a subject in the specific area is a person,
An image processing program for causing a computer to execute.

Images