JP2013012089A - Image processing device, camera and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing device capable of detecting a foreign-object shadow satisfactorily.SOLUTION: An image processing device comprises a correction part for correcting a detected foreign-object image 100. The correction part comprises: a frame forming part for surrounding the foreign-object shadow 100 with a frame 120; a pixel selection part for selecting a correction target pixel 100A within the foreign-object shadow, selecting a first pixel 111 and a second pixel 112 that are respectively located at an intersection with the frame on a line x passing through the pixels, and selecting a third pixel 113 and a fourth pixel 114 that are respectively located at an intersection with the frame on a line y; and a compensating luminance calculation part for obtaining first compensating luminance Y_x at a position of the correction target pixel through linear compensation using luminance of the first and second pixels, obtaining second compensating luminance Y_y through the linear compensation using the luminance of the third and fourth pixels, and, with respect to the first and second compensating luminance, obtaining average compensating luminance Y_ave of the first and second compensating luminance through weighted average that decreases weight as the correction target pixel gets closer to either line of the frame.

Description

  The present invention relates to an image processing device, a camera, and an image processing program.

  In an imaging device such as a digital camera, if a foreign object adheres to an optical member such as an optical filter that is on the imaging optical path and is disposed in the vicinity of the image sensor, a foreign object shadow caused by the adhered foreign material appears in the captured image, and that portion is an image. It becomes a defect. In particular, in an interchangeable lens type imaging apparatus, foreign matter such as dust or dust is likely to enter the interior during lens replacement, and image defects are likely to occur. In order to solve such a problem, a reference image as foreign object shadow information attached to the optical member is photographed, and information such as the position of the foreign object shadow is obtained from the reference image, and the foreign object shadow reflected in the normally captured image is captured. Image processing apparatuses that detect and correct image defects are known. There has also been proposed a method of detecting a foreign object shadow from one image and correcting the image defect portion (see Patent Document 1).

JP 2004-220553 A

However, in the detection from the light transmittance of the shadow portion, it is not possible to determine whether or not it is a foreign object shadow if it is erroneously detected. Further, since a large amount of pixel information is required to correct the foreign object shadow, the calculation takes time. Furthermore, correction marks may remain depending on the image of the correction portion.
An object of the present invention is to provide an image processing apparatus, a camera, and an image processing program that can detect a foreign object shadow satisfactorily.

The present invention solves the above problems by the following means. In addition, in order to make an understanding easy, although the code | symbol corresponding to embodiment of this invention is attached | subjected and demonstrated, it is not limited to this.
The invention according to claim 1 corrects the foreign object shadow (100) detected by the foreign object shadow (100) in the processing target image and the foreign object shadow (100) detected by the foreign object shadow detector (60). A correction unit (70), and the correction unit (70) includes a first straight line (101) and a second straight line (102) parallel to each other around the foreign object shadow (100), and a first straight line (101). And a frame forming part (71) surrounded by a frame (120) formed by a third straight line (103) and a fourth straight line (104) perpendicular to the second straight line (102) and parallel to each other, and the foreign object shadow ( 100) is selected as a correction target pixel (100A) and passes through the correction target pixel (100A) and is parallel to the third straight line (103) and the fourth straight line (104) (x ) And intersect with the first straight line (101) A first pixel (111) at a position where the second pixel (112) intersects the second straight line (102), and passes through the correction target pixel (100A) and the first straight line. A third pixel (113) on a sixth line (y) parallel to (101) and the second line (102) and intersecting the third line (103); and a fourth line ( 104) and a fourth pixel (114) at a position intersecting with the pixel selection unit (72) for selecting, and linear interpolation using the luminance of the first pixel (111) and the second pixel (112). The first complementary luminance (Y_x) at the position of the correction target pixel (100A) is obtained, and the correction target pixel ((X_x)) is obtained by linear interpolation using the luminance of the third pixel (113) and the fourth pixel (114). 100A) at the second complementary luminance ( _y), and for the first complementary luminance (Y_x), when the correction target pixel (100A) is in the center between the third straight line (103) and the fourth straight line (104), the weight is Strongly, the weight decreases as one of the straight lines is approached, and the correction target pixel (100A) is between the first straight line (101) and the second straight line (102) for the second complementary luminance (Y_y). The weight of the first complementary luminance (Y_x) and the second complementary luminance (Y_y) is the average complementary luminance (Y_ave) by a weighted average that has a strong weight when it is in the center and decreases as one of the straight lines is approached. And a correction processing for correcting the luminance of the correction target pixel (100A) using the calculated average complementary luminance (Y_ave). And it is performed for all the pixels of the foreign substance shadow (100), an image processing apparatus according to claim (50).
The invention according to claim 2 is the image processing apparatus (50) according to claim 1, wherein the average complementary luminance (Y_ave) calculation unit (73) calculates the weighted average as a correction target pixel (100A). The first distance (x) from the first pixel (111) and the second pixel (112) to the closer one is a ratio to the distance between the first pixel (111) and the second pixel (112). The second distance (y) to the closer of the third pixel (113) and the fourth pixel (114) is expressed as a ratio to the distance between the third pixel (113) and the fourth pixel (114). When shown, the average complementary luminance (Y_ave) is obtained by multiplying the first complementary luminance (Y_x) by the second distance (y) and the second complementary luminance (Y_y) (Y_y) by the first distance (x). An image processing apparatus (50) characterized in that it is obtained by multiplying.
The invention described in claim 3 is the image processing apparatus (50) according to claim 1 or 2, wherein
The complementary luminance calculation unit (73) uses the first pixel (111), the second pixel (112), and the second pixel (112), which are used when obtaining the first complementary luminance (Y_x) and the second complementary luminance (Y_y). An image characterized in that, as the luminance of the third pixel (113) and the fourth pixel (114), an average luminance of a plurality of pixels including both adjacent pixels on the frame (120) where each pixel is located is used. A processing device (50).
The invention described in claim 4 is the image processing device (50) according to claim 3, wherein the correction unit (70) averages the luminance of a plurality of pixels surrounding the correction target pixel (100A). An image processing apparatus (50) characterized by calculating a ratio between the secondary luminance obtained in this way and the average complementary luminance (Y_ave) and correcting the luminance of the correction target pixel (100A) according to the ratio. ).
A fifth aspect of the present invention is the image processing apparatus (50) according to any one of the first to fourth aspects, wherein the processing target image passes through red, green and blue color filters (13a). In the case of an image captured by an imaging device (16) and nonlinearly corrected, the image processing apparatus (50) is characterized in that the correction processing is performed for each of red, green, and blue images.
The invention described in claim 6 is a camera (1) including the image processing device (50) according to any one of claims 1 to 5.
The invention according to claim 7 is an image processing program for causing a computer to execute the function of the image processing apparatus (50) according to any one of claims 1 to 6.

  According to the present invention, it is possible to provide an image processing apparatus, a camera, and an image processing program that can detect a foreign object shadow satisfactorily.

It is a figure which shows the structure of a lens interchangeable camera. 1 is a block diagram showing a personal computer (PC) and peripheral devices constituting a camera and an image processing apparatus. It is a figure explaining an image sensor. It is a functional block diagram of an image processing device that performs foreign object shadow detection / correction processing. It is a functional block diagram of a foreign object shadow detection unit in the image processing apparatus. It is a conceptual diagram showing the hexagonal pyramid model of HSV conversion process. It is a functional block diagram of a correction | amendment part. It is a figure explaining the correction method of a foreign object shadow. 6 is a flowchart of foreign object shadow detection / correction processing in the image processing apparatus. It is a flowchart of the correction process in an image processing apparatus.

Hereinafter, a first embodiment of the present invention will be described with reference to the drawings and the like.
FIG. 1 is a diagram illustrating a configuration of a lens-interchangeable camera 1. FIG. 2 is a block diagram showing a personal computer (PC) 30 and peripheral devices constituting the camera 1 and the image processing apparatus.
The camera 1 is an electronic still camera that outputs image data obtained by converting a subject image into an electrical signal.

  The camera 1 includes a camera body 2 and an imaging lens 3 that can be attached to and detached from the camera body 2. The imaging lens 3 includes an imaging optical system 3L that includes a plurality of optical lens groups (shown as one lens in FIG. 1).

The camera 1 includes an imaging optical system 3L (imaging lens 3) and a camera body 2.
The camera body 2 includes a shutter 11, an optical filter 12, an image sensor 13, an image processing unit 14, an operation unit 15, a display unit 16, a memory card interface unit 17, an external interface unit 18, a power source 19, a control unit 20, and the like. ing.

The shutter 11 adjusts the exposure time by shielding and passing photographing light from the imaging optical system 3L toward the image sensor 13.
The optical filter 12 is configured by an optical low-pass filter or the like that prevents generation of false colors (color moire) or the like during imaging. Reference numeral 4 denotes a foreign substance such as dust or dirt attached to the surface of the optical filter 12. If the foreign matter 4 is attached to the optical filter 12 that is on the optical path of the photographing light and transmits the photographing light, pixels that are affected by the foreign matter 4 in the image data acquired by the image sensor 13. May be included.

The imaging element 13 is a photoelectric conversion element such as a CCD or a CMOS, and has a rectangular imaging area composed of a plurality of pixels. FIG. 3 is an explanatory diagram of the image sensor 13. On the light receiving surface of the image sensor 13, a red (R) filter 13R, a green (G) filter 13G, and a blue (B) filter 13B are provided corresponding to each pixel. The provided color filter 13a is arranged based on a so-called Bayer array.
The image sensor 13 converts the image formed on the image plane into an electrical signal and outputs an image signal (imaging signal) corresponding to the subject image. The image signal output from the image sensor 13 is A / D converted and input to the image processing unit 14 after predetermined analog signal processing.

The image processing unit 14 performs image processing such as complementation, gradation conversion, and edge enhancement on the image data input from the image sensor 13. The complement processing includes gamma processing that complements an unprocessed image output from the image sensor 13 with an image close to an impression seen by human eyes.
In the camera 1 of the present embodiment, the image data supplied to the image processing unit 14 in the image sensor 13 is shown in the RGB color system. Each pixel constituting the image data has color information of any one of RGB color components.
In addition, although one photoelectric conversion element which comprises the image pick-up element 13 is called a pixel, 1 unit of the image data corresponding to this pixel is also called a pixel. An image is also a concept composed of a plurality of pixels. It is assumed that the image data for which image processing has been completed has undergone interpolation processing, and color information of all RGB color components is present in each pixel.

The operation unit 15 includes a release button for inputting a signal for determining a shooting timing by the photographer, a selection button for mode switching, and the like. And the operation part 15 inputs those operation information into a control part.
The display unit 16 is configured by an LCD or the like, and displays various setting menus in the camera 1, reproduced images based on images captured by the image sensor 13 and image data stored in a memory card.

The memory card interface unit 17 interfaces with a memory card (card-like removable memory) 40.
The external interface unit 18 interfaces with an external device such as the PC 30 via a predetermined cable or wireless transmission path.
The power source 19 supplies power to each functional unit of the camera 1.

  The control unit 20 performs overall control of each functional unit in the camera 1. That is, when a shooting instruction signal is input from the release switch in the operation unit 15, the control unit 20 controls the driving of the shutter 11 to control the shooting operation such as the exposure operation. Further, the control unit 20 transfers image data acquired by the image sensor 13 and subjected to image processing by the image processing unit 14 to a memory card 40 or the like that is detachably attached to the memory card interface unit 17. Remember and save. Further, the control unit 20 calls the image data stored in the memory card 40 or the like and causes the display unit 16 to display the image data.

The camera 1 configured as described above is controlled by the control unit 20 at the time of shooting and operates as follows.
When the release button of the operation unit 15 is pressed by the operator, the shutter 11 is opened for a predetermined time. The imaging device 13 generates an image signal corresponding to the optical image formed in the imaging region by the imaging optical system 3L. The image signal generated by the image sensor 13 is input as image data to the image processing unit 14, and image processing such as gamma processing is performed by the image processing unit 14. Then, the captured image is displayed on the display unit 16 based on the image data processed by the image processing unit 14, and the image data is subjected to compression processing by JPEG or the like as necessary to the memory card interface unit 17. Store / store in the installed memory card 40. In addition, image data is sent to a PC 30 (described later) connected via the external interface unit 18.

  The image data stored / saved in the memory card 40 or transmitted to the PC 30 via the external interface unit 18 is recorded in, for example, an Exif file. In the Exif file, additional information such as aperture value, camera name, camera manufacturer name, shutter speed, exposure conditions such as exposure correction, and shooting time are recorded.

  Although not shown, the personal computer (PC) 30 has an interface with a CPU, memory, hard disk, memory card interface unit that interfaces with the memory card 40, and an external device such as the camera 1 via a cable or a wireless transmission path. An external interface unit is provided. The PC 30 is connected to a monitor 31, a printer 32, and the like, and an application program related to foreign object shadow detection / correction processing recorded in the CD-ROM 33 is installed in advance.

Next, with reference to FIG. 4 to FIG. 6, a process (foreign object shadow detection / correction process) for detecting and correcting a foreign object shadow in the captured target image data will be described.
FIG. 4 is a functional block diagram of the image processing apparatus 50 that performs foreign object shadow detection / correction processing. FIG. 5 is a functional block diagram of the foreign object shadow detection unit 60 in the image processing apparatus 50. FIG. 6 is a conceptual diagram showing a hexagonal pyramid model of HSV conversion processing.

  The foreign object shadow detection / correction processing is performed by the PC 30 on the image data after completion of the image processing provided through the memory card 30 or transmitted from the external interface unit 18 of the camera 1. As described above, an application program for foreign object shadow detection / correction processing is installed in the PC 30, and the PC 30 functions as an image processing apparatus that performs foreign object shadow detection / correction processing by executing this program.

  As shown in FIG. 4, the image processing apparatus 50 that functions by the PC 30 and the program includes an input unit 52 that receives target image data, a foreign object shadow detection unit 60 that detects a foreign object shadow region, and a foreign object shadow detection unit 60. And a correction unit 70 for correcting the foreign object shadow based on the foreign object shadow information detected by the output unit 53, and an output unit 53 to which the corrected image data is output.

1. Foreign Object Shadow Detection Unit As shown in FIG. 5 which is a functional block diagram, the foreign object shadow detection unit 60 includes a connected region determination unit 61, a target region extraction unit 62, a luminance determination unit 63, a hue determination unit 64, and a saturation. A degree determination unit 65, a face determination unit 66, and a drop ratio determination unit 67.

  The foreign object shadow is generated, for example, by photographing a foreign object shadow caused by the foreign object 4 (see FIG. 1) attached to the front surface of the optical filter 12. Hereinafter, a foreign object shadow area generated by such a foreign object shadow (see FIG. 1) will be described as an example.

1-1 Connected Area Determining Unit The connected area determining unit 61 is a part that extracts a region that may have a foreign object shadow from the brightness (for example, luminance) information of each pixel in the target image. The foreign object shadow is detected by the connected region determination unit 61 by first performing inverse gamma correction on the target image data, HSV conversion is performed on the obtained image data of each pixel by RGB, and a hue plane and a saturation plane are obtained. The luminance plane is obtained, and the luminance gradient is further obtained.

(Reverse gamma correction)
The connected region determination unit 61 first performs inverse gamma correction to return the RGB information of the image used for luminance information calculation from a non-linear state to a linear state. In the case of a RAW image, linear RGB information can be obtained by using an image immediately after passing through the color filter 13a (FIG. 3) having a deBayer arrangement before various correction processes are applied. In the case of a JPEG image, since gamma correction has already been performed, reverse gamma correction is performed as follows in order to obtain linear RGB information.
R ′ = (R) ^γ
G ′ = (G) ^γ
B ′ = (B) ^γ
For example, inverse gamma correction is performed with a value of γ = 2.2. You may convert into linear with reference to a gamma table.

(HSV conversion)
The connected region determination unit 61 performs an HSV conversion process for each pixel constituting the inverse gamma corrected image.
FIG. 6 is a conceptual diagram of a hexagonal pyramid model for HSV conversion. The coordinate axis of the brightness value V is represented by the center axis of the hexagonal pyramid, the coordinate axis of the saturation S is represented by an axis orthogonal to the center axis of the hexagonal pyramid, and the hue H is represented by a rotation angle that rotates the central axis of the hexagonal pyramid. The
The elements of each pixel of the lightness value V (Value), the hue value H (Hue), and the saturation value S (Saturation) are expressed by the following mathematical formulas (1) to (10) from the RGB elements of each pixel in the RGB image data. Used to convert.

Vmax = max {R, G, B} (1)
Vmin = min {R, G, B} (2)
As
V = Vmax (3)
S = (Vmax−Vmin) / Vmax (4)
H = (π / 3) (b−g) (when R = Vmax) (5)
H = (π / 3) (2 + r−b) (when G = Vmax) (6)
H = (π / 3) (4 + gr) (when B = Vmax) (7)

However,
r = (Vmax−R) / (Vmax−Vmin) (8)
g = (Vmax−G) / (Vmax−Vmin) (9)
b = (Vmax−B) / (Vmax−Vmin) (10)
When H <0 as a result of conversion by the equations (5) to (7), 2π is added to H. When Vmax = 0, S = 0 and H = undefined.

で求めてから画素の勾配（ｇｒａｄ）を以下の式（１３）で計算する。

(Luminance information acquisition)
Luminance information (Y) is calculated from linear RGB values subjected to inverse gamma correction, and the following formula is used to obtain luminance information close to human appearance.

Y = (0.299 * R ′ + 0.589 * G ′ + 0.114 * B ′) / 255 (11)

The luminance plane Y thus obtained is subjected to vertical and horizontal differential filters, and the respective differences are expressed by the following equation (12).

Then, the pixel gradient (grad) is calculated by the following equation (13).

  In the luminance gradient plane, a high pixel value is shown at the peripheral part of the foreign object shadow, and a low pixel value is shown in a uniform plane such as the sky. That is, a high pixel value is shown in the edge portion, and thereby the edge portion of the foreign object shadow can be detected.

(Concatenated pixel area extraction)
The connected area determination unit 61 sets a predetermined threshold value and converts the luminance gradient plane into a binary image. That is, a pixel having a luminance gradient value higher than the threshold is replaced with “1”, and a pixel having a luminance gradient value lower than the threshold is replaced with “0”. In the present embodiment, the threshold value is 0.3.
Then, the connected region determination unit 61 extracts a region (connected pixel region) in which “1” pixels are connected from the binary image obtained in this way. The connected pixel region is, for example, a pixel region that is continuous (near 4) on at least one of four sides of a rectangular pixel. This connected pixel area is an area where there is a possibility of a foreign object shadow due to the foreign object shadow.

1-2 Target Area Extraction Unit The target area extraction unit 62 selects a region that is likely to be a foreign object shadow from the connected pixel regions extracted by the connected region determination unit 61 based on the size (large or small) of the foreign object shadow. Extract.
Here, the target area extraction unit 62 uses the size of a foreign object shadow that causes a foreign object shadow as a reference when extracting an area that may be a foreign object shadow part from the connected pixel area.

First, a method serving as a reference for determining the size of a foreign object shadow will be described.
Here, with reference to a threshold value defined by the number of pixels so that a foreign object shadow projected by a foreign object having a minimum size of 100 μm × 100 μm to a maximum of 400 μm × 400 μm, which is the size of a general foreign material, is used as a reference, A region whose size is included in this threshold is determined as a foreign object shadow that can be a correction target, and is determined as a region having a possibility of a foreign object shadow part.

1-3 Luminance Determination Unit The luminance determination unit 63 uses the luminance value Y obtained by the HSV conversion described in the previous connected region determination unit 61 and the luminance plane corresponding to the image plane, and the inner and outer luminances in the connected pixel region. Compare Here, the foreign object shadow has a lower internal luminance than the peripheral luminance, and the luminance ratio is much smaller than one. For this reason, a predetermined threshold value smaller than 1 is set, and only when the luminance ratio of the connected pixel region is smaller than this threshold value, it is determined that the region has a possibility of a foreign object shadow as a foreign object shadow that can be corrected. ,select.

1-4 Hue Determination Unit The hue determination unit 64 obtains the hue value H of each pixel by the HSV conversion described in the previous connected region determination unit 61 and generates a hue plane corresponding to the image plane. The hue determination unit 64 uses the hue value H and the hue plane to compare the inner and outer hues in the connected pixel region. Here, although the foreign object shadow has a great influence on the luminance, it hardly affects the hue, and the hue ratio inside and outside the connected pixel area is almost 1. For this reason, a predetermined range close to 1 is set, and only when the hue ratio is within this range, it is determined and selected as a region that may have a foreign object shadow as a foreign object shadow to be corrected. .

1-5 Saturation Determination Unit The saturation determination unit 65 generates the saturation plane corresponding to the image plane by obtaining the saturation value S of each pixel by the HSV conversion described in the previous connected region determination unit 61. The saturation determination unit 65 uses the saturation value S and the saturation plane to compare the internal and external saturations in the connected pixel region.

The reason for performing determination based on such saturation ratio is as follows.
In the foreign matter portion, the brightness inside the foreign matter is lower than the brightness around the foreign matter, and the brightness ratio is much smaller than 1. Since the foreign object shadow hardly affects the hue information, the hue ratio is almost 1 inside and outside the foreign object. However, there are subjects whose luminance ratio is smaller than 1 and hue ratio is almost close to 1, depending on the image, and false detection occurs only with these two conditions.

  Therefore, the saturation ratio inside and outside the foreign object is incorporated in the determination. In general, the foreign matter is a shadow and appears as a black point image in the image. Therefore, the saturation value of the foreign matter portion is lower than that of the peripheral portion, and the saturation ratio inside and outside the foreign matter is smaller than 1. On the other hand, there is a portion where the saturation ratio in the subject portion that is not a foreign object is larger than 1. By discriminating using this feature, it is possible to discriminate whether the extracted area is a foreign object or a subject, and it is possible to eliminate erroneous detection and increase the foreign object detection accuracy.

In the saturation calculation in this saturation determination, the saturation expression (4) in the HSV conversion expression described above in the present embodiment takes a ratio with Vmax, and therefore a dark portion in the image. In some cases, the saturation does not become low, that is, the foreign matter portion may have a value greater than 1 when the saturation ratio with the surroundings is calculated. For example, there is a dark pixel of R = 12, G = 7, and B = 3, and when this saturation is obtained by the above equation (4),

(12-3) /12=0.75

Thus, the maximum saturation value of 1 is 0.75, which is a considerably high value.

  If a saturation ratio of 1 or less is used as a foreign matter determination condition, there arises a problem that dark foreign matter is not detected. If the saturation ratio allows more than 1, false detection will occur. In order to solve this problem, in this embodiment, the saturation determination unit obtains the saturation S as the corrected saturation S instead of the above equation (4) according to the following equation (14).

上述した、Ｒ＝１２、Ｇ＝７、Ｂ＝３という暗いピクセルについて、この修正彩度Ｓで計算してみると、最大値１の彩度に対して０．０３となり、暗い部分で彩度が低い値で算出される。
この修正彩度Sを用いることで、画像全体の暗い部分の彩度を人間の感覚にそった値で算出でき、異物検出パラメータの彩度比パラメータも範囲を限定することで誤検出を排除することができる。

When the above-described dark pixels of R = 12, G = 7, and B = 3 are calculated with this corrected saturation S, the saturation value of 0.01 is obtained with respect to the saturation of the maximum value 1, and saturation is obtained in the dark portion. Is calculated with a low value.
By using this corrected saturation S, the saturation of the dark part of the entire image can be calculated with values that match human senses, and the false detection is eliminated by limiting the range of the saturation ratio parameter of the foreign object detection parameter. be able to.

1-6 Face determination unit Even with the countermeasures described so far, erroneous detection may occur, and it is particularly easy to detect erroneously with a mole or the like on a human face. Misdetection on the human face has a greater effect on the appearance of the image than other misdetections, and if the misdetection part is corrected, the impression may change greatly. Therefore, in the present embodiment, the face determination unit 66 is provided, and it is determined whether or not there is a portion that may have a foreign object shadow on the human face.

  The face determination unit 66 detects a face region in an image that is a target for detecting a foreign object shadow by a known method. Then, it is determined whether or not the portion determined as a foreign object shadow in the connected region determination unit 61 to the saturation determination unit 65 up to the face determination unit 66 is located in the face region.

1-7 Drop Ratio Determination Unit Next, if there are portions determined to be foreign object shadows in the region determined to be a face by the face determination unit 66, the connected region determination unit 61 performs reverse gamma correction on these portions. The linear RGB information obtained in this way is examined.

  If the portion determined to be a foreign object shadow is a foreign object shadow, the linear RGB information is affected at an equal ratio. For example, in the case of a foreign object shadow, a gain drop of about 20% occurs with respect to RGB around the foreign object. However, in the area of a mole that is not a foreign object shadow, for example, the drop in the peripheral portion is not uniform for each of RGB. From this feature, it is possible to distinguish whether a foreign object shadow candidate in the face part is actually a foreign object shadow or not.

2. Correction Unit FIG. 7 is a functional block diagram of the correction unit 70. FIG. 8 is a diagram illustrating the foreign object shadow 100 in the processing target image, and is a diagram illustrating a correction method.
The correction unit 70 corrects the shadow determined as the foreign object shadow by the foreign object shadow detection unit 60 described above. As illustrated in FIG. 7, the correction unit 70 includes a frame formation unit 71, a pixel selection unit 72, and a complementary luminance calculation unit 73.

2-1. Frame Forming Unit The frame forming unit 71 includes a first straight line 101 and a second line parallel to each other around a portion (foreign object shadow 100) determined as a foreign object shadow by the foreign object shadow detection unit 60, as shown in FIG. It is surrounded by a frame 120 formed by a straight line 102 and a third straight line 103 and a fourth straight line 104 which are orthogonal to the first straight line 101 and the second straight line 102 and are parallel to each other.

2-2 Pixel Selection Unit The pixel selection unit 72 selects the correction target pixel 100 </ b> A located inside the foreign object shadow 100.
The pixel selection unit 72 includes a first pixel 111 that is parallel to the third straight line 103 and the fourth straight line 104 and intersects the first straight line 101 on the straight line x passing through the correction target pixel 100A. The second pixel 112 at a position intersecting the straight line 102 is selected.

  The pixel selection unit 72 further includes a third pixel 113 that is parallel to the first straight line 101 and the second straight line 102 and that intersects the third straight line 103 on the straight line y passing through the correction target pixel 100A, The fourth pixel 114 at a position intersecting with the straight line 104 is selected.

2-3 Complementary Luminance Calculation Unit The complementary luminance calculation unit 73 sets the luminance of the first pixel 111 as A and the distance from the correction target pixel 100A as x1. The brightness of the second pixel 112 is B, and the distance from the correction target pixel 100A is x2. For the third pixel 113, the luminance is C and the distance from the correction target pixel 100A is y1. For the fourth pixel 114, the brightness is D and the distance to the correction target pixel 100A is y2.

  Then, the luminances A, B, C, and D of the first pixel 111, the second pixel 112, the third pixel 113, and the fourth pixel 114 are calculated as average values of three pixels including two adjacent pixels. The luminance values averaged by the three pixels are defined as a first average luminance A ′, a second average luminance B ′, a third average luminance C ′, and a fourth average luminance D ′, respectively.

The complementary luminance calculation unit 73 obtains the first complementary luminance Y_x at the position of the correction target pixel 100A in the x direction and the third average by linear interpolation using the first average luminance A ′ and the second average luminance B ′. The second complementary luminance Y_y at the position of the correction target pixel 100A in the y direction is obtained by the following equation (15) by linear interpolation using the luminance C ′ and the fourth average luminance D ′.

  Here, for example, when the first pixel 111, the second pixel 112, the third pixel 113, and the fourth pixel 114 are specifically bright or dark due to noise or the like, the following description will be given based on that pixel. When the complementary luminance to be obtained is obtained, the luminance after complementation in the correction target pixel 100A exhibits abnormal luminance with respect to all other pixels along the line where the luminance value of the pixel is used for calculation. However, according to the present embodiment, as described above, A, B, C, and D are calculated as average values of three pixels including two adjacent pixels. Since the luminance values averaged by these three pixels are set as the first average luminance A ′, the second average luminance B ′, the third average luminance C ′, and the fourth average luminance D ′, respectively. On the other hand, the phenomenon that shows abnormal luminance is alleviated.

そして第一補完輝度と第二補完輝度とを平均した平均補完輝度Ｙ＿ａｖｅを、以下の式（１８）により求める。 Next, in this way, the final interpolation value of the correction target pixel 100A is obtained using Y_x and Y_y obtained from the linear interpolation in the x direction and the y direction.
At this time, the blend ratio of Y_x and Y_y (the ratio of weighting when taking an average) is changed depending on the position of the correction target pixel 100A. In addition, about the following E and F, you may obtain | require by the ratio or the number of pixels.

Then, an average complementary luminance Y_ave obtained by averaging the first complementary luminance and the second complementary luminance is obtained by the following equation (18).

  Next, the correction unit 70 calculates a ratio between the average complementary luminance Y_ave of Expression (18) and the original luminance ori_Y of the correction target pixel 100A, and performs the process on the entire foreign object shadow 100, thereby obtaining the gain map Y_gmap [i, j] can be created.

この平均値である二次的オリジナル輝度でゲインマップを式（２０）により作成し、オリジナル画像の輝度を補正することで、元から存在する画像のノイズ感の再現が可能となる。 However, if the correction is made as it is, the image of the correction portion will feel flat and the naturalness of the original image will disappear. Therefore, the value of the original luminance ori_Y of the correction target pixel 100A for calculating the gain is replaced with the secondary original luminance that is an average value of the correction target pixel 100A and the surrounding eight pixels by using the following equation (19). .

By creating a gain map with the secondary original luminance, which is the average value, using Equation (20) and correcting the luminance of the original image, it is possible to reproduce the noise feeling of the original image.

In the case of a RAW image, the above-described gain map is calculated for each original RGB surface that is luminance, and in the case of a JPEG image, which is not linear.
For the JPEG image, the foreign object shadow 100 is corrected using the obtained gain map of each surface of RGB. If the RGB components of the original image are ori_r, ori_g, ori_b, and the gain maps of the RGB surfaces are final_rgmap, final_ggmap, and final_bgmap, the corrected RGB is calculated as in the following equation (21).

The final_R, final_G, and final_B obtained in this way are obtained by correcting the foreign object shadow 100, and these are output as a single JPEG image. For a RAW image, each of the RGB planes can be corrected using the gain map of luminance obtained above. Assuming that each of the linear RGB components is ori_r, ori_g, ori_b, and the luminance gain map is final_ygmap, the corrected RGB is calculated by the following equation (22).

  Next, the flow of foreign object shadow detection / correction processing in the image processing apparatus 50 will be described with reference to the flowcharts shown in FIGS. 9 and 10 and the following description, the step is also abbreviated as “S”.

  The image processing apparatus 50 reads correction target image data input via the memory card 40 or the external interface unit 18 of the camera 1 (S01).

The RGB data of the read target image data is subjected to inverse gamma correction (S02).
Then, the luminance Y is obtained from the RGB subjected to the inverse gamma correction by using the equation (11), and noise processing is performed by applying an averaging filter to the luminance plane Y (S03).

Thereafter, the gradient (Ygrad) of Y is calculated (S04).
A region where the luminance gradient Ygrad is greater than or equal to the threshold is extracted (S05).

Subsequently, a connected pixel region is extracted by the connected region determination unit 61 (S06).
Further, the size of the extracted connected pixel area is determined (S07).

Next, the luminance determination unit 63 determines whether or not the region is a region that may be a foreign object shadow portion based on the internal / external luminance ratio in the connected pixel region (S08).
Further, the hue determination unit 64 determines whether or not the region is a possible foreign object shadow portion based on the internal / external hue ratio in the connected pixel region (S09), and then performs saturation determination (S10). ).

The face determination unit 66 determines whether or not an area with a possibility of foreign object shadow is in the face area (S11).
If the position is detected as a face, the linear RGB information is examined. Here, when the drop of linear RGB is equal, it is determined that there is a possibility of foreign object shadow (S12).

  Then, the foreign object information is stored (S13), and the foreign object shadow 100 is corrected by the correcting unit 70 for the area (foreign object shadow 100) that is determined to be a possible foreign object shadow part in all the above steps (S14).

FIG. 10 is a flowchart showing the correction of the foreign object shadow 100 by the correction unit 70 in S14.
First, the frame forming unit 71 of the correction unit 70 forms the frame 120 around the foreign object shadow 100 (S21).
Next, the pixel selection unit 72 selects the correction target pixel 100A located inside the foreign object shadow 100 (S22).
Then, the correction target pixel 100A inside the foreign object shadow 100, the first pixel 111 at a position intersecting the first straight line 101 on the straight line x, the second pixel 112 at a position intersecting the second straight line 102, on the straight line y The third pixel 113 at the position intersecting with the third straight line 103 and the fourth pixel 114 at the position intersecting with the fourth straight line 104 are selected (S23).

The luminances A, B, C, and D of the first pixel 111, the second pixel 112, the third pixel 113, and the fourth pixel 114 are averaged by three pixels including two adjacent pixels, respectively, and the first average luminance A ′ The second average brightness B ′, the third average brightness C ′, and the fourth average brightness D ′ are obtained (S24).
Using the linear interpolation using the first average luminance A ′, the second average luminance B ′, the third average luminance C ′, and the fourth average luminance D, the first complementary luminance Y_x and the second complementary luminance Y_y are expressed by Expression (15). (S25).

  Using Y_x and Y_y obtained from linear interpolation in the x and y directions, the blend ratio of Y_x and Y_y is changed according to the position of the correction target pixel 100A, and the final interpolation value of the correction target pixel 100A is obtained (S26). ).

  The value of the original luminance ori_Y of the correction target pixel 100A is replaced with the secondary original luminance that is the average value of the correction target pixel 100A and the surrounding eight pixels using Equation (19) (S27).

In the case of a RAW image, the above-described gain map is calculated for each original RGB surface that is luminance in the case of a RAW image and non-linear in the case of a JPEG image (S28).
For the JPEG image, the foreign object shadow 100 is corrected using the obtained gain map of each surface of RGB (S29).

As described above, this embodiment has the following effects.
(1) For example, when calculating the brightness map of a large number of surrounding pixels when obtaining a gain map of a foreign object shadow, the calculation is complicated, a calculation load is applied, and the calculation time becomes long. Furthermore, since a plurality of pixels around the foreign object shadow are referred to, there is a problem that a correct gain map cannot be created if there is a subject near the foreign object shadow.
However, according to the present embodiment, the amount of change in the luminance gain of one pixel of the foreign object shadow is calculated based on the luminance of the four pixels outside the foreign object shadow. Therefore, calculation is easy, and calculation load can be reduced and calculation time can be shortened.

(2) In this case, if a pixel outside the foreign object shadow is very bright or dark compared to other pixels due to noise or the like, the luminance value of the pixel is affected by the influence of the pixel. There is a possibility that the whole line that uses for calculation of complementation becomes brighter. If this is corrected as it is, a line-shaped correction mark is generated particularly at the boundary of the portion used for the outer frame.
However, according to the present embodiment, the influence of what is used for calculation of the edge direction is reduced as going to the edge, and therefore no line is formed at the edge.
Furthermore, since the three frame calculations are averaged, it becomes smoother.

(3) Furthermore, even if the first pixel 111, the second pixel 112, the third pixel 113, and the fourth pixel 114 are not specifically near the edge but only one pixel is specifically bright or dark due to noise or the like, When the complementary luminance is obtained based on the pixel, the luminance after complementation in the correction target pixel 100A does not become compatible with all other pixels along the line used for calculating the luminance value of the pixel. However, according to the present embodiment, since the average of three adjacent pixels is taken, even when a pixel is specifically bright or dark due to noise or the like, the line used for the calculation in the other pixels The phenomenon that shows an abnormal luminance with respect to the portion is alleviated.

(4) In the present embodiment, in the case of a JPEG image or the like, a gain map is created and corrected on each original RGB surface before nonlinear correction, so that a foreign object shadow can be corrected without correction traces even in a nonlinear RGB image. .

(Deformation)
The present invention is not limited to the above-described embodiment, and various modifications and changes as described below are possible, and these are also within the scope of the present invention.

(1) In the above embodiment, the complementary luminance calculation unit 73 sets the distance between the first pixel 111 and the correction target pixel 100A to x1, and the second pixel 112 sets the distance from the correction target pixel 100A to x2. For the third pixel 113, the distance from the correction target pixel 100A is y1, and for the fourth pixel 114, the luminance is D and the distance from the correction target pixel 100A is y2.
However, the distance between the correction target pixel 100A, the first pixel 111, and the second pixel 112 is x, and the distance between the correction target pixel 100A, the first pixel 111, and the second pixel 112, Can be expressed as 1-x (when the correction target pixel 100 </ b> A is intermediate between the first pixel 111 and the second pixel 112, either may be used).
In this case, the equations (16), (17), (18)
E = x, F = y (16 ')
G = x / x + y, H = y / x + y (17 ')
Y_ave = Y_x * y / (x + y) + Y_y * x / (x + y) (18 ')
Can be expressed as

(2) In the above embodiment, an example in which image data captured by the camera 1 which is an electronic still camera is processed has been described. However, the present invention is not necessarily limited to this content. The present invention can also be applied to image data taken by a video camera that handles moving images. It can also be applied to image data taken with a camera-equipped mobile phone. Furthermore, it can be applied to a copying machine, a scanner, and the like. That is, the present invention can be applied to any image data captured using an image sensor.

(3) In the above embodiment, the example in which the image data captured by the camera 1 which is an electronic still camera is processed by the PC (personal computer) 31 to remove the influence of the foreign matter has been described. However, the present invention is not limited to this. Absent. The electronic camera 1 may be provided with such a program. Further, such a program may be provided in a printer, a projection device, or the like. That is, the present invention can be applied to any apparatus that handles image data.

(4) The program executed on the PC 31 is not limited to a recording medium such as a CD-ROM, but can be provided through a data signal such as the Internet.
In addition, although embodiment and a deformation | transformation form can also be used in combination as appropriate, detailed description is abbreviate | omitted. Further, the present invention is not limited to the embodiment described above.

  1: Camera, 12: Optical filter, 13: Image sensor, 13a: Color filter, 17: Interface unit for memory card, 18: External interface unit, 30: Personal computer (PC), 33: CD-ROM, 40: Memory Card: 50: image processing device, 60: foreign object shadow detection unit, 61: connected region determination unit, 62: target region extraction unit, 63: luminance determination unit, 64: hue determination unit, 65: saturation determination unit, 66: Face determination unit, 67: drop ratio determination unit, 70: correction unit, 71: frame formation unit, 72: pixel selection unit, 73: complementary luminance calculation unit, 100: foreign object shadow, 100A: correction target pixel

Claims (7)

A foreign object shadow detection unit for detecting a foreign object shadow in the processing target image;
A correction unit for correcting the foreign object shadow detected by the foreign object shadow detection unit,
The correction unit is
A frame forming portion surrounding the foreign object shadow with a frame formed by a first straight line and a second straight line parallel to each other and a third straight line and a fourth straight line orthogonal to the first straight line and the second straight line. ,
Select a correction target pixel located inside the foreign object shadow,
A first pixel passing through the correction target pixel and on a fifth straight line parallel to the third straight line and the fourth straight line, at a position intersecting with the first straight line, and at a position intersecting with the second straight line. Select a second pixel,
A third pixel that passes through the correction target pixel and is on a sixth straight line that is parallel to the first straight line and the second straight line and that intersects the third straight line, and a position that intersects the fourth straight line A pixel selection unit for selecting a fourth pixel;
By linear interpolation using the luminance of the first pixel and the second pixel, the first complementary luminance at the position of the correction target pixel is obtained,
By linear interpolation using the luminance of the third pixel and the fourth pixel, to determine the second complementary luminance at the position of the correction target pixel,
For the first complementary luminance, the weight is strong when the correction target pixel is in the center between the third straight line and the fourth straight line, and the weight decreases as one of the straight lines approaches, and the second With respect to complementary luminance, the first interpolation is performed by a weighted average that has a strong weight when the correction target pixel is in the center between the first straight line and the second straight line, and the weight decreases as the straight line approaches either straight line. A complementary luminance calculation unit for obtaining an average complementary luminance of the luminance and the second complementary luminance,
The correction unit performs a correction process for correcting the luminance of the correction target pixel on the entire pixels of the foreign object shadow using the obtained average complementary luminance.
An image processing apparatus. The image processing apparatus according to claim 1,
The average complementary luminance calculation unit calculates the weighted average,
The first distance from the correction target pixel to the closer of the first pixel and the second pixel is indicated as a ratio to the distance between the first pixel and the second pixel,
When the second distance to the closer of the third pixel and the fourth pixel is expressed as a ratio to the distance between the third pixel and the fourth pixel,
The average complementary luminance is obtained by multiplying the first complementary luminance by the second distance and by multiplying the second complementary luminance by the first distance.
An image processing apparatus. The image processing apparatus according to claim 1, wherein:
The complementary luminance calculation unit
The frame (120) in which the first pixel, the second pixel, the third pixel, and the fourth pixel are respectively used as the first pixel, the second pixel, the third pixel, and the fourth pixel. Using the average luminance of a plurality of pixels, including both neighboring pixels above,
An image processing apparatus. The image processing apparatus according to claim 3,
The correction unit is
Calculating a ratio between the secondary luminance obtained by averaging the luminance of a plurality of pixels surrounding the correction target pixel and the average complementary luminance, and correcting the luminance of the correction target pixel according to the ratio;
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 4,
In the case where the image to be processed is an image that is captured by an image sensor through red, green, and blue color filters and nonlinearly corrected
Performing the correction process on each of the red, green and blue images;
An image processing apparatus.   A camera comprising the image processing apparatus according to claim 1.   An image processing program for causing a computer to execute the function of the image processing apparatus according to claim 1.

Images