JP5163735B2 - Image processing method, image processing program, image processing apparatus, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing method, an image processing program, an image processing apparatus, and an imaging apparatus that extract flare information from a captured image.

  When photographing with a photographing optical system having a diffractive optical element (DOE), if an extremely bright object (bright spot) exists within the field angle, the bright spot on the photographed image Flares with color blur (color flare) occur around the image. This color flare is significantly unnatural compared to the colorless flare.

  Incidentally, in Patent Document 1, image formation characteristic data of a photographing optical system is stored in advance, a flare component is estimated based on the data and the position of a bright spot on the photographed image, and an electronic device that removes it from the photographed image A camera is disclosed.

Japanese Patent No. 4250513

  However, since a normal captured image such as a JPEG image is often subjected to development processing, there is a possibility that flare component estimation may fail even if the imaging characteristic data is known.

  Therefore, the present invention provides an image processing method, an image processing program, an image processing apparatus, and an imaging apparatus capable of accurately extracting the influence of the color flare from the color flare occurrence area in the photographed image.

  According to one aspect of the image processing method of the present invention, an input procedure for inputting a color processing target image acquired by a photographing optical system and whether or not a color flare generation area on the processing target image is subjected to color correction. And a restoration procedure for restoring the color distribution of the generation area to a state before color correction when the generation area has been subjected to color correction, and the color flare in the generation area. And a flare estimation procedure for estimating the applied influence based on the color distribution of the generated area after restoration.

  According to one aspect of the image processing program of the present invention, an input procedure for inputting a color processing target image acquired by a photographing optical system, and whether or not color correction is performed on a color flare generation area on the processing target image. And a restoration procedure for restoring the color distribution of the generation area to a state before color correction when the generation area has been subjected to color correction, and the color flare in the generation area. And causing the computer to execute a flare estimation procedure for estimating the applied influence based on the color distribution of the generated area after restoration.

  According to an aspect of the image processing apparatus of the present invention, color correction is performed on an input unit that inputs a color processing target image acquired by a photographing optical system and a color flare generation area on the processing target image. Determination means for determining whether or not color generation has been performed on the generation area, restoration means for restoring the color distribution of the generation area to a state before color correction, and the color flare in the generation area. Flare estimation means for estimating the applied effect based on the color distribution of the generated area after restoration.

  One aspect of the imaging apparatus of the present invention includes an imaging element that captures a subject image formed by a photographing optical system, and one aspect of the image processing apparatus of the present invention.

  According to the present invention, an image processing method, an image processing program, an image processing apparatus, and an imaging apparatus that can accurately extract the influence of a color flare from a color flare occurrence region in a captured image are realized.

It is a block diagram which shows the structural example of the image processing apparatus of 1st Embodiment. 4 is an operation flowchart of flare color component removal processing by a CPU. 4 is a flowchart of color correction component extraction processing by a color correction component extraction processing unit 22; 4 is a flowchart of flare extraction processing by a flare color component extraction processing unit 21. It is an example of an input image. It is an example of the detected high-intensity area | region. It is an example of the high-intensity area | region (saturation area | region) resulting from a flare. It is an example of a labeled saturation region. It is a figure explaining the relationship between saturation area | region _Si , flare area | region _Ai, and process area | region _Ai '. It is a figure which compares the pixel profile of a certain process area A _i ′ before saturation enhancement with the pixel profile of the same process area A _i ′ after saturation enhancement. It is a schematic diagram explaining the preparation procedure of a reference | standard image. It is a schematic diagram explaining the extraction procedure of a color correction component. It is a figure explaining a binary mask image. It is a figure explaining a gray scale mask image. It is a figure which shows the relationship between R component (with a flare color component) and G component of process area _| region _Ai '. It is a schematic diagram explaining the extraction procedure of a flare color component. It is a figure which shows the relationship between R component (there is no flare color component) and G component of process area _| region _Ai '. It is a schematic diagram explaining the modification of the extraction procedure of a color correction component. It is a block diagram which shows the structural example of the electronic camera of 2nd Embodiment.

[First Embodiment]
Hereinafter, a first embodiment of the present invention will be described.

  FIG. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to the present embodiment. The image processing apparatus according to this embodiment includes a computer 11 in which an image processing program is installed.

  A computer 11 illustrated in FIG. 1 includes a data reading unit 12, a storage device 13, a CPU 14, a memory 15, an input / output I / F 16, and a bus 17. The data reading unit 12, the storage device 13, the CPU 14, the memory 15, and the input / output I / F 16 are connected to each other via a bus 17. Further, an input device 18 (keyboard, pointing device, etc.) and a monitor 19 are connected to the computer 11 via an input / output I / F 16. The input / output I / F 16 receives various inputs from the input device 18 and outputs display data to the monitor 19.

  The data reading unit 12 is used when reading image data to be subjected to image processing from the outside. For example, the data reading unit 12 conforms to a reading device (such as an optical disk drive, a magnetic disk drive, or a magneto-optical disk drive) that acquires image data from a storage medium inserted into the data reading unit 12 or a known communication standard. It is composed of communication devices (USB interface, LAN module, wireless LAN module, etc.) that communicate with external devices.

  The storage device 13 is configured by a storage medium such as a hard disk or a nonvolatile semiconductor memory. The storage device 13 stores an image processing program. The storage device 13 can also store image data read from the data reading unit 12.

  The CPU 14 is a processor that executes an image processing program stored in the storage device 13 and comprehensively controls each unit of the computer 11. Here, the image processing program of the present embodiment has a function of flare color component removal processing, and when executing the flare color component removal processing, the CPU 14 functions as a control unit that controls each unit. It also functions as a flare color component extraction processing unit 21 and a color correction component extraction processing unit 22 (the operations of the flare color component extraction processing unit 21 and the color correction component extraction processing unit 22 will be described later).

  The memory 15 temporarily stores various calculation results in the image processing program. The memory 15 is composed of, for example, a volatile SDRAM.

  FIG. 2 is an operation flowchart of flare color component removal processing by the CPU 14. Hereinafter, each step will be described in order.

  Step S11: The CPU 14 reads image data of a color image designated as a target of flare color component removal processing by the user onto the memory 15 (hereinafter referred to as “input image”). This input image is an image written in the storage medium of the data reading unit 12, for example.

  Here, the input image is an image in which color flare has occurred. The image in which the color flare is generated is an image taken by the photographing optical system including the DOE, for example, as shown in FIG. 5 (however, since FIG. 5 is a monochrome image, the color flare is not colored. ). The color flare usually has a zonal pattern or a pattern close to it, but the pattern does not always form a complete zonal pattern, and the shooting conditions (imaging characteristics of the photographic optical system used to shoot the image, It depends on the wavelength spectrum of the bright spot that existed in the angle of view. The color flare distribution (flare color component) also depends on the shooting conditions.

  However, since the flare color component removal processing of this embodiment can be executed even when the color flare pattern and the distribution with color are unknown, the input image may not be accompanied by the shooting condition data.

  Step S12: The CPU 14 performs size reduction processing on the input image read into the memory 15. Hereinafter, the input image after size reduction is referred to as a “reduced image”. The reason why the size reduction process is performed on the input image is to reduce the calculation load of subsequent steps S13 to S19 and S23.

  Step S13: The CPU 14 compares a brightness of each pixel of the reduced image with a predetermined value (here, a saturation level), thereby reducing a bright area (high brightness area) where the luminance exceeds the predetermined value. (See FIG. 6). As shown in FIG. 6, the detected high luminance area may include not only a high luminance area caused by flare but also a high luminance area unrelated to flare.

  In this step, in order to reduce the calculation load, instead of using the luminance value (R, G, B weighted sum) of each pixel as an index of the luminance of each pixel, the sum of each color component (R , G, B) may be used. Alternatively, a color component value (G intensity) representing each pixel may be used as an index of luminance of each pixel. Further, such a method of reducing the calculation load can be employed in other steps for handling the luminance of the pixel.

  Step S14: The CPU 14 determines whether or not each of the high luminance areas detected in step S13 is caused by flare, and excludes those other than those caused by flare (see FIG. 7).

  For example, the determination as to whether or not each high-luminance region is caused by flare is performed as follows. That is, the CPU 14 compares the high-luminance area with a circular pattern prepared in advance, and if the correlation between the two is greater than a certain level, the CPU 14 regards the high-luminance area as being caused by flare, and the correlation is less than a certain level. If it is, the high brightness area is not considered to be caused by flare. As a circular pattern to be compared, an average pattern may be used as a high luminance area caused by flare.

  Alternatively, the determination as to whether or not each high brightness area is caused by flare may be performed as follows. That is, the CPU 14 sets a circular area having a predetermined radius from the center of the high luminance area, and when there is a high luminance area in the entire circular area, the high luminance area is caused by the flare. If at least a part of the circular area is out of the high luminance area, the high luminance area is not attributed to flare. Note that the radius of the circular region is set to be approximately the same as the radius of the smallest high-luminance region caused by flare.

  Therefore, the high luminance area that is not excluded in this step corresponds to a saturated area in the center of the flare. Hereinafter, the high luminance area that is not excluded in this step is simply referred to as “saturated area”.

  Step S15: The CPU 14 labels the one or more saturated regions detected in step S14 and assigns serial numbers (flare numbers i = 1, 2,...) (See FIG. 8). As a result, the total number of flares occurring in the input image (that is, the final value of the flare number i) is determined.

  Step S16: The CPU 14 sets the flare number i to an initial value (1).

Step S17: As shown in FIG. 9, when focusing on the saturation region S _i corresponding to the flare number i on the reduced image, the CPU 14 calculates a coordinate g _i corresponding to the luminance center of gravity of the saturation region S _i and coordinates g _i the average distance from to the contour of the saturation region S _i is calculated as the radius r _i of the saturation region S _i. Further, CPU 14 is centered on the coordinates g _i on the reduced image, and radius of the circular area is (α × r _i), set to the flare region A _i corresponding to the flare number i (where, alpha> 1 ).

Here if, when the flare is generated in the flare region A _i was color flare believed flare color component is superimposed on the area indicated by the hatched portion in FIG.

Furthermore, the CPU 14 sets the inclusion area that includes the entire flare area A _i on the reduced image and has a certain margin γ as the processing area A _i ′ corresponding to the flare number i (where γ> 0).

Here, _'although the size of the set for each flare number i, the processing area A _i' processing area A _i if the can include a flared region A _i, all of the flare size of the processing area A _{i '} It may be common among the numbers i. Further, the shape of the processing area A _i ′ may be circular or rectangular. Further, the center of the processing area A _i ′ may be deviated from the center of the flare area A _i . Further, here, the luminance center of the saturation region S _i and a coordinate g _i, in order to reduce the computational load, the center of the saturation region S _i may coordinate g _i.

Step S18: When paying attention to the flare area A _i corresponding to the flare number i, the CPU 14 determines whether or not the flare generated in the flare area A _i is a color flare. , the processing area a _{i 'proceeds} to step S19 in order to execute a color correction component extraction process and flare color extracting process related, if not the color flare, the processing area a _i' color correction component extraction processing related and flare color component In order to omit the extraction process, the process proceeds to step S26.

Here, the determination as to whether or not the flare occurring in the flare area A _i is a color flare is performed as follows, for example. That is, the CPU 14 calculates the average value of the flare area A _i for each color component (for each RGB), and if the variation between the color components of the average value exceeds the predetermined range, the flare of the flare area A _i is calculated. Is considered to be a color flare, and if not exceeded, the flare is not a color flare (a colorless flare).

Step S19: The CPU 14 instructs the color correction component extraction processing unit 22 to perform color correction component extraction processing on the processing area A _i ′ corresponding to the flare number i. The color correction component extraction processing unit 22 executes color correction component extraction processing, and performs color correction (saturation emphasis processing, color balance adjustment, color conversion processing, etc.) previously applied to the processing area A _i ′. Estimate the correction component. The color correction component estimated in this step is composed of one or two color correction components. Details of this step will be described later.

Step S20: The CPU 14 subtracts the color correction component (one color or two color correction components) estimated in step S19 from the processing area A _i ′. As a result, the color of the processing area A _i ′ is restored to the state before the color correction.

  Step S21: The CPU 14 performs size enlargement processing on the color correction component (one color or two color correction components) estimated in step S19. When the size of the input image is reduced to 1 / M times in the size reduction process in step S12 described above, the size of the color correction component (one color correction component or two color correction components) is increased M times in the size enlargement process in this step. To. Therefore, in this step, a color correction component having a size corresponding to the input image (one color or two color correction components) is acquired.

Step S22: The CPU 14 subtracts the color correction component (one color correction component or two color correction components) after the size enlargement process from the area corresponding to the processing area A _i ′ on the input image. However, in the subtraction, the CPU 14 obtains a coordinate g _i ′ on the input image corresponding to the coordinate g _i based on the coordinate g _i on the reduced image and the magnification M described above, and performs color correction on the coordinate g _i ′. The centers of the components (one color correction component or two color correction components) are matched. As a result, the color of the area corresponding to the processing area A _i ′ on the input image is restored to the state before color correction.

Step S23: The CPU 14 instructs the flare color component extraction processing unit 21 to perform the flare color component extraction processing on the processing area A _i ′ (restored processing area A _i ′) corresponding to the flare number i. The flare color component extraction processing unit 21 executes the flare color component extraction process and estimates the flare color component generated in the processing area A _i ′. The flare color component estimated in this step is composed of two flare color components. Details of this step will be described later.

  Step S24: The CPU 14 performs size enlargement processing on the flare color components (two flare color components) estimated in step S23. When the size of the input image is 1 / M times in the size reduction process in step S12 described above, the size of the flare color component (flare color components for two colors) is increased M times in the size enlargement process in this step. . Therefore, in this step, flare color components of a size corresponding to the input image (two flare color components) are acquired.

Step S25: The CPU 14 subtracts the flare color components after the size enlargement process (two flare color components) from the area corresponding to the processing area A _i ′ on the input image (the input image after restoration). However, in the subtraction, the CPU 14 obtains the coordinate g _i ′ on the input image corresponding to the coordinate g _i based on the coordinate g _i on the reduced image and the magnification M described above, and the flare color with respect to the coordinate g _i ′. The centers of the components (two flare color components) are made to coincide. As a result, the flare color component of the color flare generated in the area corresponding to the processing area A _i ′ on the input image is removed. That is, the color flare generated in the area is a colorless flare.

  Step S26: The CPU 14 determines whether or not the flare number i has reached the final value. If the flare number i has not been reached, the process proceeds to step S27, and if it has reached, the process proceeds to step S28.

  Step S27: The CPU 14 increments the flare number i and then proceeds to step S17. Therefore, the color correction components and flare color components of all flare numbers are sequentially removed from the input image.

  Step S28: The CPU 14 displays the input image from which the color correction component and the flare color component are removed (hereinafter referred to as “final image”) on the monitor 19. Thereafter, when a save instruction is input from the user, the final image is saved in a save destination designated by the user (for example, a storage medium inserted in the data reading unit 12), and the flow ends.

  FIG. 3 is a flowchart of color correction component extraction processing by the color correction component extraction processing unit 22. Hereinafter, each step will be described in order.

Step S191: When the color correction component extraction processing unit 22 refers to each color component (R component, G component, B component) of the flare area A _i corresponding to the flare number i, the color correction pixel is included in the flare area A _i. If there is at least one color correction pixel, the process proceeds to step S192 to execute the color correction component extraction process. If there is no color correction pixel, color is detected. The flow of the correction component extraction process is terminated.

Here, the color correction pixel is a pixel having an unnatural color as a pixel of the flare area A _i before color correction. Hereinafter, the case where the color correction is saturation enhancement will be described.

10A shows a pixel profile (pixel profile on the center line) of a certain processing area A _i ′ before saturation enhancement, and FIG. 10B shows the same processing area A _i ′ after saturation enhancement. This is a pixel profile. The saturation enhancement applied to the processing area A _i ′ is saturation enhancement for enhancing the G component and the B component relative to the R component.

As shown in FIG. 10A, in the processing area A _i ′ before saturation enhancement, there are only pixels in which all color components are equal to or higher than the noise level T1, whereas FIG. As described above, pixels in which the R component is less than the noise level T1 are generated in the processing region A _i ′ after saturation enhancement. In addition, in at least some of the pixels, at least one of the G component and the B component reaches the saturation level T2. Thus, a pixel in which at least one color component has reached the saturation level T2 and the other at least one color component is less than the noise level T1 is a color correction pixel.

Incidentally, as shown in FIG. 10B, the color correction pixel is generated in the area inside the flare area A _i and outside the saturation area S _i , and this area is superimposed with the flare color component. Corresponds to the area. This is because the G component and B component of many pixels in this region have reached the saturation level T2 at the time before saturation enhancement. This is because it was necessary to push the value of the R component below the noise level T1 in order to emphasize it. Such a color correction pixel is generated because color flare is generated in the processing area A _i ′.

However, even if the flare color component extraction process described later is performed on the processing area A _i ′ that includes the color correction pixel, the flare color component may not be correctly estimated. Must be extracted and removed in advance.

Step S192: The color correction component extraction processing unit 22 detects all color correction pixels from the flare area A _i, and calculates the number of pixels having a noise level less than T1 from the flare area A _{i for} each color component (for each RGB). Then, one or two color components having a certain number or more (here, zero or more) are found, and the one or two color components are determined as correction target color components. In the example shown in FIG. 10, only the R component is determined as the correction target color component. In the following, it is assumed that the correction target color component is only the R component.

Step S193: The color correction component extraction processing unit 22 pays attention to the R component of the processing area A _i ′ as shown on the left side of FIG. By smoothing, a reference image I _R ″ as shown in FIG. 11B is created. This reference image I _R ″ is used as a reference for calculating a color correction component. Here, the creation of the reference image I _R ″ can be performed by the procedure shown in FIG. 11 (A) → FIG. 11 (B).

That is, the color correction component extraction unit 22, first, the R component I _R shown on the left side in FIG. 11 (A), by spatially smoothed by smoothing filter such as a Gaussian filter, FIG. 11 (B R component I _R ′ as shown in FIG. According to this smoothing, the pixel value of the region having a low value (dotted line frame portion) compared to the surrounding area is higher than that before the smoothing, and the pixel value of the region having a high value compared to the surrounding region is smoothed. Lower than before. Subsequently, the color correction component extraction processing unit 22 compares the two R components I _R and I _R ′ before and after smoothing for each pixel, and collects the pixels with the higher value to create one composite image. Then, the synthesized image is set as a reference image I _R ″ (FIG. 11B). This reference image I _R ″ is an area (dotted line frame portion) whose value is lower than that of the periphery in the original R component I _R. ) Only.

Step S194: As shown in FIG. 12A, the color correction component extraction processing unit 22 subtracts the reference image I _R ″ from the original R component I _R to obtain a color correction component as shown in FIG. I _D is calculated, however, a high frequency error remains in the color correction component I _D calculated here.

Step S195: The color correction component extraction processing unit 22 spatially smoothes the color correction component _ID with a smoothing filter such as a Gaussian filter, thereby obtaining a color correction component I _C as shown in FIG. calculate. The color correction component I _C is the color correction component for R component.

Step S196: The color correction component extraction processing unit 22 determines whether or not the estimation of the color correction component I _C has been completed for all the correction target color components. If not, the process returns to Step S193. When the remaining correction target color components are estimated and completed, the flow in FIG. 3 is terminated (step S196).

  FIG. 4 is a flowchart of flare extraction processing by the flare color component extraction processing unit 21. Hereinafter, each step will be described in order.

Step S231: The flare color component extraction processing unit 21 calculates an average value of the restored flare area A _i corresponding to the flare number i for each color component (for each RGB), and one color whose average value is the smallest The component is defined as a reference color component. Hereinafter, it is assumed that the G component is set as the reference color component. In this case, the flare color component is estimated for each color component other than the G component, that is, each of the R component and the B component.

Step S232: The flare color component extraction processing unit 21 creates a mask image corresponding to the flare number i. As shown in FIG. 13, the mask image is a mask image having the same shape and size as the processing area A _i ′, and a binary mask image in which an opening is provided in the area corresponding to the flare area A _i . Further, the flare color component extraction processing unit 21 creates a gray scale mask image as shown in FIG. 14 by spatially smoothing the mask image with a smoothing filter such as a Gaussian filter. The gray scale mask image corresponds to a binary mask image in which the contrast of the boundary between the opening and the non-opening is intentionally reduced. The average value per unit area of the gray scale mask image decreases from the center to the periphery of the processing area A _i ′.

Note that the size of the smoothing filter (filter diameter) used in this step is desirably set larger as the flare area A _i is larger.

Step S233: The flare color component extraction processing unit 21 sets one color component other than the reference color component (here, the G component) as the target color component. Hereinafter, it is assumed that the R component is set as the target color component. FIG. 15 is a diagram (pixel profile) showing the relationship between the R component of a certain processing area A _i ′ and the G component of the same processing area A _i ′. As shown in FIG. 15, in the R component, a flare color component is superimposed on a region inside the flare region A _i and outside the saturation region S _i . That is, the R component in that region is significantly higher than the G component in the same region.

Flare color component extraction section 21 of this step, by dividing each pixel in the G component of the _'R component of the processing area A _i' processing area A _i, correlated color distribution of the R component relative to the G component Create an image _IGR . A schematic diagram of the correlated color distribution image _IGR is shown on the left side of FIG.

Here, many pixels in the central region a of the correlated color distribution image _IGR have pixel values close to 1, and many pixels in the ring-shaped region b located around the central region a are more than 1. And many pixels in the outermost peripheral region c located outside the ring-shaped region b have lower pixel values than the ring-shaped region b. Among these, the central region a corresponds to the saturation region S _i shown in FIG. 15, and the ring-shaped region b is a region inside the flare region A _i and outside the saturation region S _i shown in FIG. That is, it corresponds to a region where flare color components are superimposed.

Step S234: The flare color component extraction processing unit 21 smoothes only the regions (the central region a and the most peripheral region c) whose values are lower than those in the periphery in the correlated color distribution image _IGR . The correlated color distribution image _IGR "as shown in (B) is acquired. The acquisition of the correlated color distribution image _IGR " can be performed by the procedure shown in FIG. 16 (A) → FIG. 16 (B). .

That is, the flare color component extraction processing unit 21 first spatially smoothes the entire correlated color distribution image _IGR shown on the left side of FIG. 16A with a smoothing filter such as a Gaussian filter. A correlated color distribution image I _GR ′ as shown on the right side of FIG. According to this smoothing, the value of the region having a low value compared to the surroundings is higher than that before the smoothing, and the value of the region having a high value compared to the surroundings is lower than that before the smoothing. Subsequently, the flare color component extraction processing unit 21 compares the two correlated color distribution images I _GR and I _GR ′ before and after smoothing for each pixel, collects the pixels having the higher value, and forms one composite image. And the synthesized image is set as a correlated color distribution image I _GR ″ (FIG. 16B). This correlated color distribution image I _GR ″ is a superposition of flare color components in the original correlated color distribution image I _GR . This corresponds to a smoothed area (dotted line frame portion) that has not been performed.

Step S235: flare color component extracting unit 21 _"by performing contraction processing by the minimum value filter or the like to the correlated color distribution image I _GR" correlated color distribution image I _GR high values as compared to peripheral in shrink the area of the region, to obtain a correlated color distribution image I _H as shown in FIG. 16 (C). Furthermore, flare color component extracting unit 21, by spatially smoothing the correlated color distribution image I _H smoothing filter such as a Gaussian filter, correlated color as shown in FIG. 16 (D) distribution image I _H 'Get. This correlated color distribution image I _H ′ corresponds to an image obtained by interpolating the area where the flare color component is superimposed in the original correlated color distribution image I _GR with the surrounding area (dotted line frame portion).

Step S236: The flare color component extraction processing unit 21 multiplies the correlation color distribution image I _H ′ for each pixel by the reference color component (G component in this case) of the processing area A _i ′ to thereby calculate the flare color component. The removed R component I _Rnew is calculated. As a result, the flare color component related to the R component is removed from the processing area A _i ′. FIG. 17 is a pixel profile of the processing area A _i ′ after removal.

As apparent from a comparison of FIG. 17 with FIG. 15, after removal, the R intensity of the region where the flare color component is superimposed (the region inside the flare region A _i and outside the saturation region S _i ) is the same. It can be seen that it is close to the G intensity of the region. After the removal, the R / G intensity of the area where the flare color component is superimposed is determined by the R / G intensity of the area outside the flare area A _i and the R / G intensity inside the saturation area S _i . You can see that it is interpolated. Further, after the removal, it can be seen that the R intensity distribution shape of the region where the flare color component is superimposed is a shape corresponding to the G intensity distribution shape of the same region.

Note that the flare color component extraction processing unit 21 in this step calculates the flare color component I _FR related to the R component by subtracting the R component I _Rnew from which the flare color component has been removed from the original R component ( The R component I _Rnew from which the flare color component is removed is converted into the flare color component I _FR .) This is because in this embodiment, it is necessary to perform some processing (mask processing, size enlargement processing, etc.) on the flare color component I _FR before calculating the final image.

Step S237: The flare color component extraction processing unit 21 applies the grayscale mask created in Step S232 to the flare color component I _FR acquired in Step S236, thereby blurring the peripheral portion of the flare color component I _FR . If blurring in this way the peripheral portion, _'upon removal of the flare color component I _FR from R components (subtraction) processing area A _i' processing area A _i an unnatural gradation level difference is generated in the It is because it can prevent. Further, as described above, the size of the smoothing filter used at the time of creating the grayscale mask is set to a size according to the size of the flare area A _i , so that the peripheral portion of the flare color component I _FR after applying the mask Will be more natural. Thereby, the estimation of the flare color component I _FR regarding the R component is completed.

  Step S238: The flare color component extraction processing unit 21 determines whether or not the estimation for all the color components other than the reference color component (here, the R component and the B component) has been completed. The process proceeds to step S233 to estimate a certain color component, and if completed, the flow ends. Although only the estimation regarding the R component has been described here, the estimation regarding the B component is the same as the estimation regarding the R component, and thus the description thereof is omitted (step S238).

Above, in this embodiment, the estimation of the flare color component superposed on the flare region A _i, is performed based on the color distribution of the flare area A _i, prior to the estimation, the color correction is subjected to the flare region A _i If the color correction has been performed, the color distribution of the flare area A _i is restored to the state before the color correction, so that the possibility of failure in estimating the flare color component can be suppressed.

  Incidentally, if the color distribution is not restored before the flare color component is estimated, the flare color component is erroneously estimated, and as a result, an unnatural black ring noise is generated on the final image. .

Further, in the restoration of the color distribution of the present embodiment, a processing area A _i ′ that includes the flare area A _i is set, and the correction target color component of the processing area A _i ′ is smoothed in an area that has a lower value than the surrounding area. Since the color correction component related to the correction target color component is estimated based on the difference between the correction target color components before and after smoothing (see FIGS. 11 and 12), accurate restoration is possible.

In the flare color component estimation of the present embodiment, a processing area A _i ′ that includes the flare area A _i is set, and a reference color component that is a reference among the color components of the flare area A _i is specified. a reference color component in the processing area a _{i 'and} the correlation between the target color component, based on the reference color component of the flare area a _i, estimating the flare color component contained in the flare area a _i. That is, in this embodiment, instead of extracting the flare component itself, a flare color component (= colored flare distribution) is extracted. If this flare color component is used, the color flare can be a colorless flare, so that unnaturalness peculiar to the color flare can be surely eliminated.

Further, in the estimation of the flare color component of the present embodiment, the correlation color distribution of the processing area A _i ′ is calculated, the area of the correlation color distribution having a lower value than the surroundings is smoothed, and the correlation after smoothing is calculated. shrink the area of the high value region compared to the peripheral of the color distribution, the correlated color distribution after shrinking by multiplying the reference color component of the processing area a _{i ',} non-flared color processing area a _i The component is calculated (see FIGS. 16 and 17). Therefore, in this embodiment, the non-flare color component can be estimated almost accurately.

Further, in the present embodiment, _'by subtracting from, the processing area A _i' non flare color component processing area A _i is calculated flare color component contained in the flare area A _i for the flare color component After applying a mask having openings of the same shape, the flare color component is subtracted from the processing area A _i ′. In this embodiment, a gray scale mask (FIG. 14) in which the value of the contour of the opening is smoothed is used as the mask. Therefore, in this embodiment, it is possible to prevent an unnatural gradation step from occurring in the final image.

  Further, in the estimation of the color correction component and the flare color component of the present embodiment, the size reduction version of the input image is used instead of using the input image, so that the calculation load can be suppressed.

[Modification of First Embodiment]
Note that the color correction component extraction processing unit 22 according to the first embodiment creates the reference image I _R ″ in steps S193 to S195, but the creation of the reference image I _R ″ may be omitted. In that case, the color correction component extraction processing unit 22 may execute the following steps S193 ′ to S195 ′ instead of steps S193 to 195. Here again, it is assumed that the color component to be corrected is an R component.

Step S193 ′: The color correction component extraction processing unit 22 refers to the R component inside the flare area A _i and outside the saturation area S _i , calculates the maximum value of the R component, and sets the value as the reference value T. Set. This reference value T is used as a reference for calculating the color correction component.

Step S194 ': color correction component extraction unit 22, as shown in FIG. 18 (A), the processing area _{A i'} by subtracting the reference value T from the R component _{I R} of, as shown in FIG. 18 (B) An offset R component I _R ′ is calculated. Then, the color correction component extraction processing unit 22 replaces the value of the pixel having a positive value in the offset R component I _R ′ with zero, thereby changing the color correction component _ID as shown in FIG. get. However, a high frequency error remains in the color correction component _ID calculated here.

Step S195 ′: The color correction component extraction processing unit 22 spatially smoothes the color correction component _ID with a smoothing filter such as a Gaussian filter, so that the color correction component I _C as shown in FIG. Is calculated. The color correction component _{I C} is the color correction component for R-component (or, the step S195 ').

Further, the color correction component extraction processing unit 22 of the first embodiment automatically determines whether or not the flare area A _i is color-corrected based on the flare area A _i , but the input image is a RAW image ( In the case of an image before development processing), there is no possibility that color correction has been performed, so it may be determined that color correction has not been performed immediately without being based on the flare area A _i .

Even if the input image is not a RAW image, if a parameter indicating the content of the color correction is added to the input image, the color correction component extraction processing unit 22 does not use the flare area A _i , The color correction component may be calculated based on the parameter.

In addition, the flare color component extraction processing unit 21 according to the first embodiment performs the estimation of the flare color component for each of the two color components other than the reference color component, but the estimation for one of the two color components. May be omitted. Assuming that the G component is the reference color component, for example, when the difference between the average value of the G component and the average value of the B component in the flare area A _i is within a predetermined range, that is, the G component and the B component. If there is no difference from the component, estimation regarding the B component may be omitted.

  In addition, although the flare color component extraction processing unit 21 of the first embodiment has automatically set the reference color component, the user may specify it.

  In the flare color component removal process described above, a reduced size version (reduced image) of the input image is used instead of using the input image as it is in order to reduce the calculation load in steps S13 to S19 and S23. When it is desired to prioritize accuracy improvement over load reduction, the input image may be used as it is. In this case, the processes in steps S21, S22, and S24 are omitted.

  In the flare color component removal process described above, the CPU 14 automatically detects the flare occurrence position in the input image.

Further, in the above-described flare color component removal treatment is a flare region A _i CPU 14 detects automatically, it may be specified by the user.

In the flare color component removal process described above, the processing area A _i ′ is automatically set by the CPU 14, but may be set by the user.

In the flare color component removal process described above, the CPU 14 automatically determines whether or not the flare occurring in the flare area A _i is a color flare, but it may be determined by the user.

Further, the flare color component extraction processing unit 21 of the present embodiment determines the flare color component of the flare area A _i , the correlation between the reference color component and the target color component in the processing area A _{i, and} the reference of the flare area A _i . Although the estimation is based on the color components, the estimation may be performed by another method based on the color distribution of the flare area A _i .

[Second Embodiment]
Hereinafter, a second embodiment of the present embodiment will be described.

  FIG. 19 is a block diagram illustrating a schematic configuration of the electronic camera. As shown in FIG. 19, the electronic camera 111 includes an imaging optical system 112, a lens driving unit 113, a diaphragm 114, a diaphragm driving unit 115, a color imaging element 116, an AFE 117, an image processing engine 118, and a first. It has a memory 119, a second memory 120, a media I / F 121, a communication I / F 122, a monitor 123, and a release button 124. Among these, a lens driving unit 113, an aperture driving unit 115, and an AFE 117 are provided. The first memory 119, the second memory 120, the media I / F 121, the communication I / F 122, the monitor 123, and the release button 124 are each connected to the image processing engine 118. The photographing optical system 112 is a high-functional photographing optical system including a DOE lens, for example.

  The first memory 119 is composed of a volatile storage medium (SDRAM or the like), and temporarily stores a captured image in a pre-process or post-process of image processing by the image processing engine 118. On the other hand, the second memory 120 is configured by a nonvolatile storage medium such as a flash memory, and stores a program executed by the image processing engine 118 for a long period of time.

  The first memory 119 stores an image processing program to be executed by the image processing engine 118, and this image processing program has the same flare color component removal processing function as that of the first embodiment. Yes. When executing the flare color component removal processing, the image processing engine 118 functions as the flare color component extraction processing unit 125 and the color correction component extraction processing unit 126. The flare color component extraction processing unit 125 and the color correction component extraction processing unit 126 operate in the same manner as the flare color component extraction processing unit 21 of the first embodiment.

  Therefore, the image processing engine 118 of the present embodiment is the same as that of the first embodiment for images acquired by the electronic camera 111 by shooting or images read via the media I / F 121 (or communication I / F 122). A similar flare color component removal process can be performed.

  Note that the flare color component removal processing of the present embodiment does not require shooting condition data, and therefore the portion indicated by the dotted line in FIG. 19 may be replaceable with the electronic camera 111. Needless to say, the flare color component removal processing of the present embodiment does not require imaging condition data, and therefore the image processing engine 118 does not need to read the imaging characteristic data of the imaging optical system 112.

  Further, the flare color component removal processing of the present embodiment can be modified similarly to that of the first embodiment.

  DESCRIPTION OF SYMBOLS 11 ... Computer, 12 ... Data reading part, 13 ... Memory | storage device, 14 ... CPU, 15 ... Memory, 16 ... Input-output I / F, 17 ... Bus

Claims (11)

An input procedure for inputting a color processing target image acquired by the photographing optical system;
A determination procedure for determining whether or not color correction has been performed on a color flare generation region on the processing target image;
When color correction has been performed on the generation area, a restoration procedure for restoring the color distribution of the generation area to a state before color correction;
A flare estimation procedure for estimating the influence of the color flare on the occurrence area based on the color distribution of the occurrence area after restoration;
An image processing method comprising: The image processing method according to claim 1,
The image processing method, wherein the color correction is saturation enhancement. The image processing method according to claim 2,
In the determination procedure,
When at least one color correction pixel having a first color component lower than the noise level and a second color component equal to or higher than the saturation level is present in the generation area, the generation area is subjected to color correction. An image processing method characterized in that it is considered possible. The image processing method according to claim 3.
The restoration procedure is:
An area setting procedure for setting a processing area including the generation area on the processing target image;
A smoothing procedure for smoothing a region having a low value compared to the surroundings in the target color component of the processing region;
A correction component calculation procedure for calculating a color correction component for the target color component based on a difference between the target color components before and after smoothing. In the image processing method according to any one of claims 1 to 4,
The flare estimation procedure includes:
An area setting procedure for setting a processing area including the generation area on the processing target image;
A reference color setting procedure for setting a reference color component to be a reference among the color components of the generation region;
A correlation calculation procedure for calculating correlation information indicating a correlation between a reference color component and other color components in the processing region;
A flare color component estimation procedure for estimating a flare color component which is a colored distribution of the color flare based on the correlation information of the processing region and a reference color component of the generation region;
An image processing method comprising: In the image processing method according to any one of claims 1 to 5,
In the restoration procedure and the flare estimation procedure,
An image processing method comprising using a reduced size version of the processing target image instead of using the processing target image. An input procedure for inputting a color processing target image acquired by the photographing optical system;
A determination procedure for determining whether or not color correction has been performed on a color flare generation region on the processing target image;
When color correction has been performed on the generation area, a restoration procedure for restoring the color distribution of the generation area to a state before color correction;
A flare estimation procedure for estimating the influence of the color flare on the occurrence area based on the color distribution of the occurrence area after restoration;
An image processing program for causing a computer to execute. Input means for inputting a color processing target image acquired by the photographing optical system;
A discriminating means for discriminating whether or not color correction is applied to a color flare generation region on the processing target image;
When color correction has been performed on the generation area, restoration means for restoring the color distribution of the generation area to a state before color correction;
Flare estimation means for estimating the influence of the color flare on the occurrence area based on the color distribution of the occurrence area after restoration;
An image processing apparatus comprising: An image sensor that captures a subject image formed by the imaging optical system;
An image processing apparatus according to claim 8,
An imaging apparatus comprising: The imaging device according to claim 9,
An imaging apparatus, further comprising a photographic lens including a diffractive optical element. The imaging device according to claim 9,
An imaging device, characterized in that a photographic lens including a diffractive optical element can be attached.