JP5834542B2 - Imaging apparatus, image processing apparatus, program, and recording medium - Google Patents

Description

  The present invention relates to an imaging device, an image processing device, a program, and a recording medium.

  2. Description of the Related Art Conventionally, an imaging element in which a plurality of pixels for focus detection are arranged on a part of a light receiving surface in which a plurality of imaging pixels are arranged in a two-dimensional manner is known (Patent Document 1). The plurality of imaging pixels have spectral characteristics corresponding to each of the plurality of color components, and the focus detection pixels have spectral characteristics different from the plurality of imaging pixels. A signal for generating an image is read from a plurality of imaging pixels to determine an imaging pixel value, and a focus detection signal is read from a focus detection pixel for focus detection. Pixel values are determined. When performing pixel interpolation, the pixel value of the missing color component is interpolated among the imaging pixel values, and the imaging pixel value corresponding to the position of the focus detection pixel is interpolated.

  In the invention described in Patent Document 1, in order to interpolate the imaging pixel value corresponding to the position of the focus detection pixel, interpolation of the focus detection pixel is performed using the pixel value of the imaging pixel near the focus detection pixel. A pixel value is generated, an evaluation pixel value that is a pixel value when the neighboring imaging pixel has the same spectral characteristics as the focus detection pixel is calculated, and the pixel value of the focus detection pixel and the evaluation pixel value Are used to calculate the high-frequency component of the image, and the high-frequency component is added to the interpolated pixel value to calculate the imaging pixel value corresponding to the position of the focus detection pixel.

JP 2009-303194 A

  However, when flare occurs due to the reflected light from the wiring area between the pixels around the light receiving portion, color mixing occurs due to crosstalk in which incident light between the microlenses leaks into adjacent pixels around the light receiving portion. When such a phenomenon occurs, not only the focus detection pixels but also the surrounding pixels are affected, and the focus detection pixels that have undergone color mixing by the above-described pixel interpolation and the surrounding pixels give an unnatural appearance. There was a drawback that it appeared as a residual interpolation.

  SUMMARY An advantage of some aspects of the invention is that it provides an imaging apparatus, an image processing apparatus, a program, and a recording medium that perform pixel interpolation so that the influence of color mixture due to flare does not occur. .

One embodiment of an imaging device that exemplifies the present invention includes an imaging element that has an imaging pixel that generates an image signal, a focus detection pixel that generates a focus detection signal, and an image obtained from the imaging element. Based on the flare determination means for determining whether or not flare has occurred, and when the flare determination means determines that flare has occurred, the pixel values of the imaging pixels around the focus detection pixel are corrected by a weighting factor. A suppression unit that performs a flare suppression process for smoothing the pixel value of the corrected imaging pixel, and a pixel of the imaging pixel in the vicinity of the focus detection pixel among the pixel values of the imaging pixel smoothed by the suppression unit Pixel interpolation processing means for generating an interpolation pixel value of the focus detection pixel using the value.

  As suppression means, the surrounding imaging pixel correction means for correcting the pixel value of the imaging pixels around the focus detection pixel by a weighting factor, and the pixel values of the imaging pixels corrected by the surrounding imaging pixel correction means are smoothed. Smoothing means.

  As the surrounding image pickup pixel correcting means, among the image pickup pixels located in the vicinity of the focus detection pixel, the image pickup pixel located in the vicinity of the target image pickup pixel from the pixel value of the target image pickup pixel is selected. The pixel value of the target imaging pixel may be corrected by subtracting the weighted sum of the pixel value and the weight coefficient.

  As the suppression means, the processing performed by the surrounding imaging pixel correction means and the smoothing means is performed a plurality of times, for example, twice by changing the weighting coefficient for the imaging pixel value, and the weight count is set to “0” in the second and subsequent processing. It is desirable to perform only the processing of the smoothing means.

As the smoothing means, a means for calculating the clip range using the pixel values of the adjacent AF pixels and a position close to the pixel value of the distal imaging pixel located far from the focus detection pixel array arranged in a line. Means for calculating a prediction error from the difference from the pixel value of the proximal imaging pixel, means for determining whether or not the prediction error exceeds the clip range, and a prediction error that is out of the clip range. And means for adding the prediction error after clipping to the pixel value of the proximal imaging pixel.

  Further, the present invention may be an invention of a computer-readable program and a recording medium recording the computer-readable program.

  According to the present invention, when flare occurs, it is possible to suppress the influence of color mixture due to flare on the imaging pixels around the focus detection pixels.

It is a block diagram which shows the outline of the electronic camera which is an imaging device of this invention. It is explanatory drawing which shows the arrangement | sequence of the pixel for a focus detection (AF pixel) provided in the image pick-up element. It is explanatory drawing which shows the array of the pixel of an image pick-up element. It is a block diagram which shows the outline of an AF pixel interpolation part. It is a flowchart which shows the operation | movement procedure of an AF pixel interpolation part. It is a flowchart which shows the procedure of a 2nd pixel interpolation process. It is explanatory drawing which shows a pixel value when the vertical 5 pixel image structure containing a convex structure is cut | disconnected vertically. It is a flowchart which shows the procedure of a 3rd pixel interpolation process.

  An electronic camera 10 to which the present invention is applied includes a CPU 11 as shown in FIG. A non-volatile memory 12 and a work memory 13 are connected to the CPU 11, and the non-volatile memory 12 stores a control program that is referred to when the CPU 11 performs various controls. In addition, the nonvolatile memory 12 includes, in detail, position information of focus detection pixels (AF pixels) provided in an image sensor, which will be described later, data such as various threshold values and addition coefficients used for an image processing program obtained in advance, and Various determination tables and the like are stored.

  The CPU 11 controls each part using the work memory 13 as a temporary storage work area according to a control program stored in the non-volatile memory 12 and operates each part (circuit) function constituting the electronic camera 10.

  Subject light incident from the photographing lens 14 forms an image on a light receiving surface of an image sensor 15 such as a CCD or CMOS. The image sensor drive circuit 16 drives the image sensor 15 based on a control signal from the CPU 11. The image sensor 15 is a Bayer array type single-plate image sensor, and a primary color transmission filter 17 is attached to the front surface.

  The primary color transmission filter 17 is, for example, a primary color Bayer in which the resolution of G (green) is N / 2 and the resolutions of R (red) and B (blue) are N / 4 with respect to the total number N of pixels of the image sensor 15. It is an array.

  The subject image formed on the light receiving surface of the image sensor 15 is converted into an analog image signal. The image signal is sequentially output to a CDS 18 and an AMP 19 constituting an AFE (Analog Front End) circuit, subjected to predetermined analog processing by the AFE circuit, and then digitally processed by an A / D (Analog / Digital converter) 20. It is converted into image data and sent to the image processing unit 21.

  The image processing unit 21 includes a separation circuit, a white balance processing circuit, a pixel interpolation (demosaicing) circuit, a matrix processing circuit, a non-linear transformation (γ correction) processing circuit, an edge enhancement processing circuit, and the like. Thus, processing such as white balance, pixel interpolation, matrix, nonlinear conversion (γ correction), and contour enhancement is performed. The separation circuit separates a signal output from an imaging pixel, which will be described in detail later, and a signal output from the focus detection pixel. The pixel interpolation circuit converts Bayer array data of one color per pixel into normal color image data composed of three colors per pixel.

  The three-color image data output from the image processing unit 21 is stored in the SDRAM 23 through the bus 22. The image data stored in the SDRAM 23 is read out under the control of the CPU 11 and sent to the display control unit 24. The display control unit 24 converts the input image data into a predetermined display signal (for example, an NTSC color composite video signal) and outputs it as a through image to the display unit 25.

  The image data acquired in response to the shutter release is read from the SDRAM 23 and then sent to the compression / decompression processing unit 26. After being subjected to the compression process, the memory card 28 serving as a recording medium is provided via the media controller 27. To be recorded.

  A release button 29, a power switch 30, and an ISO sensitivity setting unit 32 are connected to the CPU 11, and temperature information is input from a temperature detector 31 that detects the temperature of the image sensor 15. This information is sent to the image processing unit 21 and is used when determining noise, which will be described in detail later.

  The AE / AF detection unit 33 detects a defocus amount and a defocus direction by a pupil division type phase difference detection method based on a focus detection pixel signal. The CPU 11 controls the driver 34 based on the defocus amount and the defocus direction obtained by the AE / AF detection unit 33 and drives the focusing motor 35 to advance and retract the focus lens in the optical axis direction to adjust the focus. I do.

  The AE / AF detection unit 33 calculates a light value (Lv = Sv + Bv) using the luminance value (Bv) calculated based on the signal of the imaging pixel and the Sv value set by the photographer using the ISO sensitivity setting unit 32. calculate. Then, the AE / AF detection unit 33 determines the aperture value and the shutter speed so that the exposure value (Ev = Av + Tv) becomes the light value (Lv). And according to this, the electronic shutter and the electronic iris of the image sensor 15 are controlled to obtain an appropriate exposure amount. Note that a mechanical shutter also serving as an aperture may be provided, and the aperture may be varied by the mechanical shutter.

  The imaging element 15 is provided with a primary color transmission filter 17 of any of R (red), G (green), and B (blue) in a Bayer array type in each of a plurality of imaging pixels on the light receiving surface. A CCD or CMOS semiconductor image sensor with a microlens array provided thereon is appropriately selected and used. Furthermore, the image sensor 15 of the present embodiment has a plurality of focus detection pixels (AF pixels) arranged one-dimensionally in the horizontal scanning direction in a partial region on the light receiving surface. The primary color transmission filter 17 is not installed in those AF pixels. In addition, there are two types of AF pixels that receive light beams that pass through the left or right side of the pupil of the optical system of the imaging lens 1. The imaging element 15 can individually read out pixel signals from the imaging pixel group and the AF pixel group.

  As shown in FIG. 2, each AF pixel 36 has sensor openings 36a and 36b that are biased to one side from the cell center (center of the microlens), and are arranged one-dimensionally along the direction of the bias. . The adjacent sensor openings 36a and 36b are biased in opposite directions, and the bias distance is the same. The AF pixel 36 having the sensor opening 36a is placed in place of the G pixel in the RGB primary color Bayer array, and the AF pixel 36 having the sensor opening 36b is placed in place of the B pixel in the RGB primary color Bayer array. It is. The pupil division phase difference AF method is realized by the AF pixel 36 having such sensor openings 36a and 36b. That is, in the light flux passing through the exit pupil, two partial light fluxes that are symmetrical with respect to the optical axis of the photographic lens 14 are separated by the AF pixel 36 having the sensor opening 36a and the AF pixel 36 having the sensor opening 36b. If each light is received, the direction of defocus and the amount of defocus are known from the phase difference between the signals output from these two pixels 36. This enables quick focusing.

  Accordingly, each of the AF pixels 36 in the present embodiment outputs a detection signal obtained by dividing the left or right pupil according to the luminance of white light. FIG. 3 shows a part of the image data centered on the area where the AF pixel 36 is arranged in the image data captured by the image sensor 15. Each cell represents one pixel. Symbols R, G, and B at the head of each cell indicate an imaging pixel having each primary color transmission filter 17. On the other hand, symbols X and Y indicate AF pixels that are sensitive to the light flux from the left side or the right side, and they are alternately arranged one-dimensionally in the horizontal scanning direction. The two-digit number following these symbols indicates the pixel position.

  The pixel interpolating unit interpolates the pixel value of the AF pixel 36 (AF pixel value) to the imaging pixel value, and after interpolating the AF pixel value to the pixel value of the imaging pixel (imaging pixel value). And a pixel interpolation unit for performing color interpolation by a linear interpolation method from the Bayer array to RGB.

  As shown in FIG. 4, the AF pixel interpolation unit 40 includes a noise determination unit 41 and a flare determination unit 42, and performs different AF pixel interpolation processes based on these determinations. The noise determination unit 41 determines whether the noise is a condition that generates a lot of noise based on the shooting condition at the time of shooting. The shooting conditions are the temperature of the image sensor 15, ISO sensitivity, shutter speed at the time of shooting, and the like. The temperature information of the image sensor 15 is obtained from the CPU 11. In addition to the temperature information, information on the ISO sensitivity set at the time of shooting and the shutter speed at the time of shooting can also be obtained from the CPU 11.

  The noise determination unit 41 determines whether there is a lot of noise based on information on the temperature, ISO sensitivity, and shutter speed of the image sensor 15. Note that a temperature detector may be provided on the main board on which the image sensor 15 is mounted, and the temperature of the main board or the temperature around the image sensor 15 may be used instead of the temperature of the image sensor 15. Furthermore, the information used for the noise determination is not limited to the three pieces of information of the temperature of the image sensor 15, the ISO sensitivity, and the shutter speed, and may be any one or two pieces of information.

  When the noise determination unit 41 determines that there is a lot of noise, a first pixel interpolation process that performs, for example, simple average interpolation is performed using the surrounding imaging pixel values without using the AF pixel values. When it is determined that there is little noise, the flare determination unit 42 performs flare determination, and the second or third pixel interpolation processing different from the first pixel interpolation processing depending on whether or not flare has occurred. I do.

  The flare determination unit 42 extracts a high brightness area (high brightness area) based on the brightness histogram of the image data, and then determines whether, for example, a magenta color exists in the extracted high brightness area. If a color exists, the average edge amount and variance value of the luminance component in the magenta color area (magenta area) are calculated, and “total area of magenta area”, “dispersion value of magenta area / total area of magenta area” are calculated. ”And“ average edge amount of luminance component in the magenta area ”are respectively determined as thresholds to determine whether or not flare has occurred.

  For flare determination, a posture detection unit such as a gyro sensor or an acceleration sensor is provided, and the CPU 11 calculates an elevation angle with respect to the horizontal of the photographic lens 14 by calculation based on an output value obtained from the posture detection unit. The information on the subject brightness, the shooting mode, etc. is sent to the flare determination unit, and the flare determination unit determines whether it is outdoor or indoor based on the information on the elevation angle, the subject distance, the subject brightness, the shooting mode, It may be determined whether or not flare occurs by distinguishing between day and night and whether or not a sky is included as a subject in the shooting angle of view when the camera is pointed upward.

  If the AF pixel interpolation unit 40 determines that no flare has occurred, the AF pixel interpolation unit 40 performs a second pixel interpolation process (pixel interpolation unit) that performs pixel interpolation using the AF pixel value. In the second pixel interpolation process, the AF pixel value is interpolated by predicting the AF pixel value (white (W) component) with a weighted sum based on the surrounding imaging pixel values.

  When the flare determination unit 42 determines that a flare has occurred, a third pixel interpolation process is executed. In the third pixel interpolation process, flare suppression means and pixel interpolation processing means are sequentially performed. The flare suppression unit includes: an ambient imaging pixel correction unit that corrects an imaging pixel value around the AF pixel using a weighting factor; and a smoothing unit that smoothes the imaging pixel value corrected by the ambient imaging pixel correction unit. Has been. The surrounding imaging pixel correction means weights the imaging pixel value positioned near the target imaging pixel and the weighting coefficient from the target imaging pixel value among the imaging pixels positioned near the AF pixel. By subtracting the sum, the target pixel value for imaging is corrected.

  As the flare suppression means, the processing performed by the surrounding imaging pixel correction means and the smoothing means is performed twice, for example, by changing the weighting coefficient for the imaging pixel value, and the weight count is set to “0” in the second processing. Thus, only the processing by the smoothing means is performed. Thereafter, the second pixel interpolation processing (pixel interpolation processing means) is executed. Thereby, the influence of the color mixture in the flare can be suppressed on the imaging pixels around the AF pixel. Therefore, in performing the second pixel interpolation process, the influence of the color mixture is also suppressed on the pixel value generated by using the AF pixel as the imaging pixel.

  Next, the operation of the AF pixel interpolation unit 40 will be described with reference to FIG. In the present embodiment, since the primary color transmission filter 17 installed in each of the imaging pixels has a Bayer pattern, the green (G) imaging pixel is positioned at the position of the AF pixel of the symbol X shown in FIG. The pixel value is interpolated, and the blue (B) imaging pixel value is interpolated at the pixel position of the AF pixel of symbol Y. In the following description, a case will be described in which the blue pixel value for Y44 and the green pixel value for X45 are interpolated. The procedure for interpolating imaging pixel values in other AF pixels is the same.

[Noise judgment]
The CPU 11 sends the image data sent from the A / D 20 to the noise determination unit 41. Further, the CPU 11 sends information on the temperature, ISO sensitivity, and shutter speed of the image sensor 15 at the time of shooting to the noise determination unit 41. Then, the CPU 11 controls the noise determination unit 41 to determine whether the noise determination unit 41 has more or less noise than the image data (S-1).

  The determination by the noise determination unit 41 is determined with reference to the noise determination table. A plurality of noise determination tables are prepared for each temperature range of the image sensor 15, and these are stored in advance in the nonvolatile memory 12. The CPU 11 sends a noise determination table corresponding to the temperature of the image sensor 15 when the image data is acquired to the noise determination unit 41.

  As the noise determination table, for example, the table described in [Table 1] is selected when the temperature of the image sensor 15 is less than 40 ° C, and the table described in [Table 2] is selected when the temperature is in the range of 40 ° C to less than 50 ° C. To do. In each table, noise prediction results determined by the shutter speed and ISO sensitivity are determined based on experiments performed in advance.

  When it is determined that there is a lot of noise, the first pixel interpolation process is performed using only the pixel values of the surrounding pixels without using the pixel values of the AF pixels (S-2).

[First pixel interpolation processing]
As the first pixel interpolation processing, for example, an image pickup pixel value corresponding to the AF pixel is obtained by simply performing average interpolation from surrounding pixels. Specifically, in FIG. 3, the pixel value of the AF pixel Y42 placed in place of the B pixel is expressed by the equation described in [Equation 1], and the pixel value of the AF pixel Y44 is expressed by [Equation 2]. Further, the pixel value of the AF pixel Y46 is obtained from the equation described in [Equation 3] from the equation.

[Equation 1]
Y42 = (B22 + B62) / 2
[Equation 2]
Y44 = (B24 + B64) / 2
[Equation 3]
Y46 = (B26 + B66) / 2
Further, the pixel value of the AF pixel X43 placed in place of the G pixel is obtained from the equation described in [Equation 4], and the pixel value of the AF pixel X45 is obtained from the equation described in [Equation 5].

[Equation 4]
X43 = (G32 + G34 + G52 + G54) / 4
[Equation 5]
X45 = (G34 + G36 + G54 + G56) / 4
In this way, when there is a lot of noise, the AF pixel value is predicted only from the surrounding imaging pixel values without using the AF pixel value, and therefore, the predicted AF pixel value varies and interpolation more than expected is performed. Therefore, it is possible to suppress as much as possible the occurrence of a color that does not actually exist called false color and the occurrence of a non-existing structure called false structure. Note that the image data obtained by interpolating the AF pixel value to the imaging pixel value is subjected to color interpolation by the linear interpolation method from the Bayer array to RGB in the image processing unit 21 and stored in the SDRAM 23 as image data for each RGB.

[Flare determination]
When the noise determination unit 41 determines that there is little noise, the CPU 11 controls the flare determination unit 42 to determine whether the flare determination unit 42 has flare (S-3). The AF pixel interpolation unit 40 performs the second pixel interpolation process (S-4) when the flare determination unit 42 determines that flare does not occur, and the third pixel interpolation process when it determines that flare occurs. (S-5) is executed alternatively.

[Second pixel interpolation processing]
Using the imaging pixel values around the AF pixel, the direction in which the variation value, which is the change rate of the pixel value, is minimized is obtained. Then, the AF pixel value is interpolated using the imaging pixel value in the direction of the smallest fluctuation.

(Calculate the direction of the minimum fluctuation value)
As shown in FIG. 6, the AF pixel interpolating unit 40 interpolates the imaging pixel values at the X45 and Y44 AF pixels, as shown in FIG. The values of the direction fluctuations H1 to H4 that are the rate of change of the above are obtained using the equations described in [Equation 6] to [Equation 9], respectively (S-6). In this embodiment, the four directions are 45 degrees and 135 degrees with respect to the horizontal scanning direction, the vertical scanning direction, and the horizontal scanning direction.

[Equation 6]
Direction variation H1 in the horizontal scanning direction =
2 × (| G34-G36 | + | G54-G56 |) + | R33-R35 | + | R53-R55 | + | B24-B26 | + | B64-B66 |
[Equation 7]
Direction variation H2 in the vertical scanning direction =
2 × (| G34-G54 | + | G36-G56 |) + | R33-R53 | + | R35-R55 | + | B24-B64 | + | B26-B66 |
[Equation 8]
45 degree direction change with respect to the horizontal scanning direction H3 =
2 × (| G27-G36 | + | G54-G63 |) + | R35-R53 | + | R37-R55 | + | B26-B62 | + | B28-B64 |
[Equation 9]
Directional change H4 = 135 degrees with respect to the horizontal scanning direction =
2 × (| G23-G34 | + | G56-G67 |) + | R33-R55 | + | R35-R57 | + | B22-B66 | + | B24-B68 |
(AF pixel values are interpolated using surrounding imaging pixel values corresponding to the direction of the minimum variation value)
The AF pixel interpolation unit 40 selects the direction of the smallest direction variation among the direction variations H1 to H4 obtained in step (S-6), and uses the imaging pixel value in that direction to select the AF pixel X45. using the selected equation corresponding to the direction of the imaging pixel value _{B Y44} of B at the position of the imaging pixel value _{G X45} and AF pixel Y44 [number 10] - [number 13] G at position (S-7). As a result, by using the imaging pixel values in the direction of small fluctuation, it becomes possible to more accurately interpolate the imaging pixel values at the positions of AF pixels such as X45 and Y44.

[Equation 10]
When the direction change H1 is the minimum, B _Y44 = (B24 + B64) / 2
G _X45 = (G34 + G36 + G54 + G56) / 4
[Equation 11]
When the direction change H2 is the minimum B _Y44 = (B24 + B64) / 2
G _X45 = (G25 + G65) / 2
[Equation 12]
When direction change H3 is the minimum, B _Y44 = (B26 + B62) / 2
G _X45 = (G36 + G54) / 2
[Equation 13]
When the direction change H4 is the minimum, B _Y44 = (B22 + B66) / 2
G _X45 = (G34 + G56) / 2
The AF pixel interpolating unit 40, in the horizontal scanning direction, which is the AF pixel arrangement direction, changes the direction variation H5 of the pixel value of the AF pixel, for example, pixel values W44 and W45 of white light of Y44 and X45 of the AF pixel, and [ It is calculated using the formula described in [Formula 14].

[Formula 14]
H5 = | W44-W45 |
The AF pixel interpolation unit 40 determines whether or not the value of the direction variation H5 exceeds the threshold th1 (S-8). If the directional fluctuation H5 is greater than the threshold value th1 (YES side), AF pixel interpolation unit 40, the imaging pixel value at Y44 and X45 interpolated values of _{B Y44} and _{G X45} calculated in step (S-7), Update the image data. The image processing unit 21 performs pixel interpolation on the updated image data in three colors to generate three-color image data, and records the three-color image data in the SDRAM 23 via the bus 22 (S-9). ).

  On the other hand, when the direction variation H5 is equal to or less than the threshold th1 (NO side), the image processing unit 21 proceeds to (S-10). The threshold th1 may be set to a value of about 512 when processing a 12-bit image, for example.

The AF pixel interpolation unit 40 determines whether or not the value of the direction variation H2 obtained in step (S-6) exceeds the threshold th2 (S-10). If the directional fluctuation H2 is greater than the threshold value th2 (YES side), AF pixel interpolation unit 40, the imaging pixel value at Y44 and X45 interpolated values of _{B Y44} and _{G X45} calculated in step (S-7), Update the image data. The image processing unit 21 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 23 via the bus 13 (S-9). ).

  On the other hand, when the direction variation H2 is equal to or less than the threshold th2 (NO side), the image processing unit 21 proceeds to (S-11). Note that the threshold th2 may be set to a value of about 64 when processing a 12-bit image, for example.

  Thereafter, the AF pixel interpolating unit 40 uses the average pixel value <W44> of white light in the AF pixel such as Y44 having sensitivity to the light beam from the right side, and the imaging pixel values of the color components R, G, and B in the vicinity. (S-11). Specifically, in step (S-6), for example, when the image processing unit 21 determines that the direction variation H2 is the minimum, the imaging pixel value of B is B24 in the equation described in [Equation 10]. And B64 are used. On the other hand, for the R and G pixel values, the R and G pixel values at the positions of the B imaging pixels B24 and B64 are interpolated using the four equations described in [Equation 15].

[Equation 15]
(1) R _B24 = (R13 + R15 + R33 + R35) / 4
(2) G _B24 = (G14 + G23 + G25 + G34) / 4
(3) R _B64 = (R53 + R55 + R73 + R75) / 4
(4) G _B64 = (G54 + G63 + G65 + G74) / 4
Then, the AF pixel interpolating unit 40 uses the R, G, and G weighting coefficients WR, WG, and WB transferred from the CPU 11 for the white light pixel values W24 and W64 at the positions of the imaging pixels B24 and B64. , [Equation 16]. Note that how to obtain the weighting factors WR, WG and WB will be described later.

[Equation 16]
W24 = WR × R _B24 + WG × G _B24 + WB × B24
W64 = WR × R _B64 + WG × G _B64 + WB × B64
Then, the image processing unit 21 calculates the average pixel value <W44> = (W24 + W64) / 2 of white light in Y44.

  The AF pixel interpolating unit 40 calculates the average pixel value <W45> of white light in the AF pixel such as X45 having sensitivity to the light beam from the left side, as in the case of step (S-11), and the adjacent color components. Calculation is performed using the R, G, and B imaging pixel values (S-12). In step (S-7), when the image processing unit 21 determines that the direction variation H2 is the minimum, the G imaging pixel value uses G25 and G65 in the equation described in [Equation 10]. On the other hand, for the R and B pixel values, the R and B pixel values at the positions of the G imaging pixels G25 and G65 are interpolated using the four equations described in [Equation 17].

[Equation 17]
(1) R _G25 = (R15 + R35) / 2
(2) B _G25 = (B24 + B26) / 2
(3) R _G65 = (R55 + R75) / 2
(4) B _G65 = (B64 + G66) / 2
Then, the AF pixel interpolation unit 40 calculates the pixel values W25 and W65 of the white light at the positions of the imaging pixels G25 and G65 by the weighted sum of the formula described in [Equation 18].

[Equation 18]
W25 = WR × R _G25 + WG × G25 + WB × B _G25
W65 = WR × R _G64 + WG × G25 + WB × B _G65
Then, the image processing unit 21 calculates the average pixel value <W45> = (W25 + W65) / 2 of white light at X45.

  The AF pixel interpolation unit 40 obtains the high-frequency component of the pixel value of white light in each AF pixel of the image sensor 2 using the average pixel value of white light obtained in (S-11) and (S-12) ( S-13). The AF pixel interpolation unit 40 first obtains an average pixel value of white light at the pixel position of each AF pixel from each AF pixel value of the image sensor 2. That is, each AF pixel value is a value obtained by dividing the light flux from the left side or the right side into pupils. Therefore, in order to obtain the pixel value of the white light at the position of each AF pixel, it is necessary to add the pixel values of the light beams from the left side and the right side to each other. Therefore, the AF pixel interpolation unit 40 of the present embodiment uses, for example, each AF pixel value and the adjacent AF pixel value, and sets the average pixel value of white light at the position of the AF pixel Y44 or X45 to [Equation 19], for example. Calculate using the formula described.

[Equation 19]
<W44>'= W44 + (W43 + W45) / 2
<W45>'= W45 + (W44 + W46) / 2
Note that in [Equation 19] described in step (S-13), the pixel value of white light at the position of each AF pixel is calculated using the AF pixel values adjacent in the AF pixel arrangement direction. If there is a strong fluctuation, the calculation of the high-frequency component becomes inaccurate, and the resolution of the white light pixel values in the arrangement direction may be lost. Therefore, in step (S-8) described above, when there is a strong variation in the arrangement direction, the addition of high frequency components is stopped.

  Thereafter, the AF pixel interpolation unit 40 obtains high-frequency components HFY44 and HFX45 of white light at the positions of Y44 and X45 from the equation described in [Equation 20].

[Equation 20]
HFY44 = <W44>'-<W44>
HFX45 = <W45>'-<W45>
The AF pixel interpolating unit 40 determines that the ratio of the high-frequency component HF of the white light pixel value at the position of each AF pixel obtained in step (S-13) to the white light pixel value is the threshold th3 (in the present embodiment). For example, about 10%) is determined (S-14). If the threshold th3 high-frequency component HF is smaller (YES side), AF pixel interpolation unit 40 the interpolated values of _{B Y44} and _{G X45} calculated in step S12 and the imaging pixel values at Y44 and X45, updating the image data To do. The image processing unit 21 performs three-color pixel interpolation on the updated image data to generate three-color image data, and stores the three-color image data in the SDRAM 23 via the bus 13 (S-9). .

  On the other hand, when the high frequency component HF is greater than or equal to the threshold th3 (NO side), the AF pixel interpolation unit 40 proceeds to step (S-15). Note that the value of the threshold th3 will be described together with the description of the weighting factors WR, WG, and WB later.

  The AF pixel interpolation unit 40 calculates the color fluctuations VR, VGr, VB, and VGb of the imaging pixel values of the respective color components R, G, or B in the vicinity of Y44 and X45 (S-15). Here, the color variations VGr and VGb indicate the G color variation at the position of the R or B imaging pixel. The AF pixel interpolation unit 40 obtains the color fluctuations VR and VGr based on the two equations described in [Equation 21].

[Equation 21]
(1) VR = | R33-R53 | + | R35-R55 | + | R37-R57 |
(2) VGr = | (G32 + G34) / 2− (G52 + G54) / 2 | + | (G34 + G36) / 2− (G54 + G56) / 2 | + | (G36 + G38) / 2− (G56 + G58) / 2 |
Note that the AF pixel interpolation unit 40 of the present embodiment calculates the value of VGr after obtaining the average value of the G pixel values at the positions R33, R35, R37, R53, R55, and R57 of the R imaging pixels.

  On the other hand, the AF pixel interpolation unit 40 obtains the color fluctuations VB and VGb based on the two equations described in [Equation 22].

[Equation 22]
(1) VB = | B22−B62 | + | B24−B64 | + | B26−B66 |
(2) VGb = | (G21 + G23) / 2− (G61 + G63) / 2 | + | (G23 + G25) / 2− (G63 + G63) / 2 | + | (G25 + G27) / 2− (G65 + G67) / 2 |
It should be noted that the AF pixel interpolation unit 40 of the present embodiment calculates the value of VGb after obtaining the average value of the G pixel values at the positions B22, B24, B26, B62, B64, and B66 of the B imaging pixels.

  The AF pixel interpolation unit 40 calculates the color variation rates KWG and KWB for the white light of the color components G and B using the color variations VR, VGr, VB, and VGb calculated in step (S-15) (S- 16). First, the AF pixel interpolation unit 40 obtains the color variations VR2, VG2, and VB2 from the three formulas described in [Equation 23] from the color variations VR, VGr, VB, and VGb.

[Equation 23]
(1) VR2 = (VR + α) × (VGb + α)
(2) VB2 = (VB + α) × (VGr + α)
(3) VG2 = (VGb + α) × (VGr + α)
Here, α is an appropriate constant for stabilizing the value of the color variation rate. For example, when processing a 12-bit image, α may be set to about 256.

  Then, the image processing unit 21 uses the color variations VR2, VG2, and VB2 to calculate the color variation VW in white light according to the equation described in [Equation 24].

[Equation 24]
VW = VR2 + VG2 + VB2
Therefore, the AF pixel interpolation unit 40 calculates the color variation rates _KWG and _KWB from [Equation 25].

[Equation 25]
K _WB = VG2 / VW
K _WB = VB2 / VW
The AF pixel interpolation unit 40 uses the high-frequency component HF of the white light pixel value at the position of each AF pixel obtained in step (S-13), and the color variation rates K _WG and K _WB calculated in step (S-16). Are used to calculate the high-frequency components of the pixel values of the color components G and B at the position of each AF pixel from the equation described in [Equation 26] (S-17).

[Equation 26]
HFB _Y44 = HF _Y44 × K _WB
HFG _X45 = HF _X45 × K _WG
The AF pixel interpolation unit 40 adds the high-frequency component of each color component in each AF pixel obtained in step (S-17) to the imaging pixel value obtained by interpolation in step (S-7) (S-18). ). For example, the CPU 11 calculates the imaging pixel values B ′ and G ′ of Y44 and X45 based on the equation described in [Equation 27], respectively.

[Equation 27]
B ′ _Y44 = B _Y44 + HF _Y44
G ′ _X45 = G _X45 + HF _X45
The AF pixel interpolating unit 40 updates the image data using the pixel values such as B ′ _Y44 and G ′ _X45 obtained by interpolation at the positions of the AF pixels such as Y44 and X45 as the imaging pixel values at the respective positions. . The image processing unit 21 converts the updated image data into image data of three colors per pixel and stores it in the SDRAM 23 (S-9).

  Even if there is no change in the arrangement direction of the AF pixels, the high-frequency component of the pixel value of the white light is caused by the deviation between the weighted sum of the spectral characteristics of the imaging pixels of each color component and the spectral characteristics of the AF pixels. Has a slight error. When there is no large fluctuation of the image in the vertical scanning direction (direction intersecting the AF pixel arrangement direction), the accuracy of the interpolation value is sufficient without adding a high-frequency component. There is a risk that a false structure resulting from Therefore, in step (S-10), in such a case, addition of a high frequency component is suppressed. In addition, when the calculated high frequency component is sufficiently small, the accuracy of the interpolation value is sufficient even if it is not added, and there is a possibility that a false structure due to an error may be caused by adding the high frequency component. For this reason, in (S-10), addition of a high frequency component is suppressed in such a case.

  Next, how to obtain the weighting factors WR, WG and WB will be described together with the threshold value th3. In obtaining such an addition coefficient and threshold value, an image sensor 2 incorporated in a product or an image sensor having the same performance as the image sensor 2 is prepared. The imaging device 2 is irradiated with illumination with substantially uniform illuminance while changing the wavelength band in various ways, and captured image data for each wavelength band is acquired. Then, for the captured image data n in each wavelength band, the pixel value Wn of white light is calculated by adding the pixel values of AF pixels with different pupil divisions as in the equation described in [Equation 19]. At the same time, the pixel values Rn, Gn, and Bn of the image pickup pixels of the respective color components in the vicinity of the AF pixel are also extracted.

  Then, the square error E is defined as [Equation 28] as a function of the unknown weighting factors WR, WG and WB.

[Equation 28]
E = Σn (WR × Rn + WG × Gn + WB × Bn−Wn) ²
Then, weighting coefficients WR, WG, and WB that minimize E are obtained (weighting coefficients WR, WG, and WB that obtain a value obtained by partially differentiating E from WR, WG, or WB are set to 0). In this way, by obtaining the weighting factors WR, WG, and WB, a weighting factor that represents the spectral characteristic of the AF pixel by the weighted sum of the spectral characteristics of the imaging pixels of the respective color components R, G, and B is obtained. The weighting factors WR, WG and WB thus obtained are recorded in the nonvolatile memory 12. Further, an error rate Kn is obtained for each captured image data n based on the obtained weighting factors WR, WG, and WB by the equation described in [Equation 29].

[Equation 29]
Kn = | WR × Rn + WG × Gn + WB × Bn−Wn | / Wn
Then, the maximum value of Kn is obtained and recorded in the nonvolatile memory 12 as the threshold value th3.

  FIG. 7 shows an example of an image structure in which the effect of the present embodiment is exhibited. FIG. 7 is a vertical section of an image structure of five vertical pixels including a convex structure (bright lines or dots). The horizontal axis is the vertical scanning direction (y coordinate), and the vertical axis is the light quantity or pixel value. The convex structure is just on the AF pixel row arranged in the horizontal scanning direction.

  The circles in FIG. 7 are pixel values captured by the G imaging pixels. However, since there is no G imaging pixel at the position of the AF pixel, the G pixel value at that position cannot be obtained. Therefore, when there is a convex structure at the position of the AF pixel, the convex structure in FIG. 7 cannot be reproduced only from the pixel values of the G imaging pixels in the vicinity of the AF pixel. In fact, in (S-7), the G pixel value (marked with ● in FIG. 7) obtained by interpolation at the position of the AF pixel using the pixel value of the G imaging pixel in the vicinity of the AF pixel is a convex structure. Is not reproduced.

  On the other hand, the pixel value of white light is obtained at the position of the AF pixel. However, normal pixels receive light that passes through the entire pupil region, whereas AF pixels receive only light that passes through the right or left side of the pupil, so adjacent AF pixels with different pupil divisions are added. Thus, the pixel value of the normal white light (of the light that has passed through the entire pupil region) is calculated ([Equation 19]).

  In addition, it is sufficient in many cases by interpolating and generating other color components R and G at the position of the G imaging pixel in the vicinity of the AF pixel to obtain a weighted sum of the color components R, G, and B. The pixel value of white light can be obtained with high accuracy ([Equation 16] and [Equation 18]).

  The squares in FIG. 7 are distributions of pixel values of white light obtained in this way. In many cases, since the high-frequency component of the white light pixel value is proportional to the high-frequency component of the color component G pixel value, the high-frequency component calculated from the white light pixel value is the convex structure component of the G pixel value. Have information. Therefore, the high-frequency component of the G pixel value is obtained based on the high-frequency component of the white light pixel value, and the value is added to the data of the ● mark to obtain the G pixel value of the ☆ mark and reproduce the convex structure ([Equation 26]).

[Third pixel interpolation processing]
The AF pixel interpolation unit 40 selects and executes the third pixel interpolation process when it is determined that there is little noise based on the determination result in the noise determination unit 41 and flare in the flare determination unit 42 is likely to occur.

  In the third pixel interpolation processing, the imaging pixel value around the AF pixel is corrected with the weighting factor, and the corrected imaging pixel value is smoothed twice by changing the weighting factor for the imaging pixel value. Thereafter, the second pixel interpolation process described above is executed. Hereinafter, the third pixel interpolation process for the two columns of the AF pixel X43 and the AF pixel Y44 in FIG. 3 will be described.

(Correcting pixel values around the AF pixel row with weighting coefficients)
As shown in FIG. 8, the AF pixel interpolating unit 40 determines whether or not the imaging pixel values arranged around the AF pixel row are equal to or greater than the threshold MAX_RAW, and is set based on the determination result. The weighting coefficient is used for correction (S-21). Here, the threshold value MAX_RAW is a threshold value for determining whether or not the pixel value is saturated.

  The AF pixel interpolation unit 40 does not correct the imaging pixel value when the imaging pixel value is equal to or greater than the threshold value MAX_RAW. On the other hand, when the imaging pixel value is less than the threshold value MAX_RAW, the AF pixel interpolation unit 40 corrects the imaging pixel value by subtracting the weighted sum value using the weighting coefficient from the original pixel value. .

  The AF pixel interpolation unit 40 corrects the imaging pixel value of the R color component using [Expression 30] to [Expression 33].

[Equation 30]
R13 ′ = R13− (R3U — 0 × R33 + R3U — 1 × G34 + R3U — 2 × B24)
[Equation 31]
R33 ′ = R33− (R1U — 0 × R33 + R1U_1 × G34 + R1U_2 × B24)
[Formula 32]
R53 ′ = R53− (R1S_0 × R53 + R1S_1 × G54 + R1S_2 × B64)
[Equation 33]
R73 ′ = R73− (R3S — 0 × R53 + R3S — 1 × G54 + R3S — 2 × B64)
Here, R1U_0, R1U_1, R1U_2, R1S_0, R1S_1, R1S_2, R3U_0, R3U_1, R3U_2, R3S_0, R3S_1, and R3S_2 are weighting coefficients. In the weighting factor, the character S indicates that it is positioned above the AF pixel, and the character U indicates that it is positioned below the AF pixel.

  The AF pixel interpolating unit 40 corrects the imaging pixel value of the G color component using [Equation 34] to [Equation 39]].

[Formula 34]
G14 ′ = G14− (G3U — 0 × R33 + G3U_1 × G34 + G3U_2 × B24
[Equation 35]
G23 ′ = G23− (G2U — 0 × R33 + G2U_1 × G34 + G2U_2 × B24)
[Equation 36]
G34 ′ = G34− (G1U — 0 × R33 + G1U_1 × G34 + G1U_2 × B24)
[Equation 37]
G54 ′ = G54− (G1S_0 × R53 + G1S_1 × G54 + G1S_2 × B64)
[Equation 38]
G63 ′ = G63− (G2S — 0 × R53 + G2S_1 × G54 + G2S_2 × B64)
[Equation 39]
G74 ′ = G74− (G3S_0 × R53 + G3S_1 × G54 + G3S_2 × B64)
Here, G1U_0, G1U_1, G1U_2, G1S_0, G1S_1, G1S_2, G2U_0, G2U_1, G2U_2, G2S_0, G2S_1, G2S_2, G3U_0, G3U_1, G3U_2, G3S_1, G3S_1, G3S_1, and G3S_1 are weights.

  Also, the AF pixel interpolation unit 40 corrects the pixel value of the B color component imaging pixel using [Equation 40] and [Equation 41].

[Equation 40]
B24 ′ = B24− (B2U — 0 × R33 + B2U_1 × G34 + B2U_2 × B24)
[Equation 41]
B64 ′ = B64− (B2S_0 × R53 + B2S_1 × G54 + B2S_2 × B64)
Here, B2U_0, B2U_1, B2U_2, B2S_0, B2S_1, and B2S_2 are weighting coefficients.

(Calculation of clip amount using adjacent AF pixel values)
The AF pixel interpolation unit 40 reads the pixel values of the adjacent AF pixels X43 and Y44 (AF pixels having the sensor openings 36a and 36b described in FIG. 2), and obtains the clip amount th_LPF using [Expression 42] ( S-22).

[Formula 42]
th_LPF = (X43 + Y44) × K_TH_LPF
Here, K_TH_LPF is a coefficient, and a large value of about “127”, for example, is applied. As the coefficient K_TH_LPF is larger, the effect of the smoothing process is higher.

(Calculate prediction error for each color component)
The AF pixel interpolation unit 40 uses [Equation 43] and [Equation 44], and among the imaging pixels having the same color component arranged in the same column, the imaging pixel (far-distant pixel) located far from the AF pixel. The difference between the position imaging pixel) value and the imaging pixel (proximal imaging pixel) value near the AF pixel is calculated as a prediction error (S-23).

[Equation 43]
deltaRU = R13′−R33 ′
deltaRS = R73′−R53 ′
[Equation 44]
deltaGU = G14'-G34 '
deltaGS = G74'-R54 '
(Determines whether the prediction error exceeds the clip range)
The AF pixel interpolating unit 40 calculates a clip range (− (th_LPF to th_LPF) is determined (S-24).

(Clip processing)
The AF pixel interpolation unit 40 performs a clipping process on a prediction error out of the clip range among the prediction errors deltaRU, deltaRS, deltaGU, and deltaGS (S-25). Here, the clipping process is to perform clipping so that a prediction error value outside the clip range is included in the clip range.

(Add prediction error to pixel value for proximal imaging)
The AF pixel interpolation unit 40 adds the prediction error to the proximal imaging pixel value of each column by [Equation 45] (S-26). Here, the prediction error is a value obtained by [Equation 43] and [Equation 44] or a clipped value.
[Equation 45]
R33 "= R33 '+ deltaRU
R53 "= R53 '+ deltaRS
G34 "= G34 '+ deltaGU
G54 "= G54 '+ deltaGS
Thereby, the pixel values of the distal imaging pixel and the proximal imaging pixel, which are the pixel values of the imaging pixels around the AF pixel column, are corrected, respectively, and further, the prediction error is used to correct the proximal imaging pixel. Pixel values are corrected.

(The corrected imaging pixel value is stored in SDRAM)
The AF pixel interpolation unit 40 stores the distal imaging pixel corrected by the weighting factor and the proximal imaging pixel corrected by the prediction error in the SDRAM 23 (S-27).

(Second processing)
When the first process is completed, the second process is executed.

(Correcting pixel values around the AF pixel row with weighting coefficients)
The AF pixel interpolating unit 40 uses the pixel values of the imaging pixels corrected in the first process to determine whether or not these pixel values for imaging are equal to or greater than a threshold value MAX_RAW. Based on the determination result, correction is performed using the set weight coefficient (S-28). Here, the threshold value MAX_RAW is a threshold value for determining whether or not the pixel value is saturated, and the same value as in the first process (S-21) is used.

The AF pixel interpolation unit 40 does not correct the imaging pixel value when the pixel value of the imaging pixel is equal to or greater than the threshold value MAX_RAW. When the pixel value of the imaging pixel is less than the threshold value MAX_RAW, the AF pixel interpolation unit 40 corrects all the weighting coefficients in [Expression 30] to [Expression 41] described above by changing them to “0”. That is, when this processing is performed, the pixel values of the imaging pixels arranged around the AF pixel column remain the original values.
[Calculation of clip amount using adjacent AF pixel values]
The AF pixel interpolation unit 40 reads the pixel values of the adjacent AF pixels X43 and Y44, and obtains the clip amount th_LPF using the above-described [Equation 42] (S-29). Here, the same value as the first process is used as the value of K_TH_LPF.

(Calculate prediction error for each color component)
The AF pixel interpolation unit 40 uses the above-described [Equation 43] and [Equation 44], and among the imaging pixels having the same color component arranged in the same column, the imaging pixel at a position far from the AF pixel. The difference between the pixel value of (distal imaging pixel) and the pixel value of the imaging pixel (proximal imaging pixel) located near the AF pixel is calculated as a prediction error (S-30).

(Determines whether the prediction error exceeds the clip range)
The AF pixel interpolating unit 40 calculates the clip range based on the clip amount obtained by [Equation 42] in which each value of the prediction error deltaRU, deltaRS, deltaGU, and deltaGS obtained by [Equation 43] and [Equation 44] described above It is determined whether it is included in (-th_LPF to th_LPF) (S-31).

(Clip processing)
The AF pixel interpolation unit 40 performs a clipping process on a prediction error out of the clip range among the prediction errors deltaRU, deltaRS, deltaGU, and deltaGS (S-32).

(Add prediction error to pixel value for proximal imaging)
The AF pixel interpolation unit 40 adds the value to the proximal imaging pixel value of each column using [Equation 45] described above (S-33). Thus, in the second process, the proximal imaging pixel value is further corrected using the prediction error.

(The corrected imaging pixel value is stored in SDRAM)
The AF pixel interpolation unit 40 stores the distal imaging pixel corrected by the weighting factor and the proximal imaging pixel corrected by the prediction error in the SDRAM 23 (S-34).

  As described above, in the third pixel interpolation process, the above-described correction process is repeatedly executed twice. After this correction process is repeatedly executed twice, the second pixel interpolation process is executed.

(Second pixel interpolation processing)
The AF pixel interpolation unit 40 executes the above-described second pixel interpolation process using the imaging pixel values stored in the SDRAM 23 (S-35). Thereby, an imaging pixel value corresponding to the AF pixel is calculated. That is, the AF pixel value is interpolated.

(The interpolated AF pixel value is stored in SDRAM)
The AF pixel interpolation unit 40 stores the AF pixel value interpolated by the second pixel interpolation process (S-35) in the SDRAM 23.

  In the third pixel interpolation process described above, the smoothing process is effectively performed on the imaging pixel values in the vicinity of the AF pixel column by repeatedly executing the correction process twice. By effectively performing the smoothing process, it is possible to reduce the influence of color mixing due to the flare that occurs on the imaging pixels adjacent to the AF pixel. In addition, since the interpolation processing for the AF pixel is performed using the pixel value of the imaging pixel in which the influence of the color mixture is reduced, a pixel value in which the influence of the color mixture due to the generated flare is reduced also in the AF pixel. That is, an image with reduced flare effects can be obtained.

  As described above, according to the embodiment, when flare occurs, the pixel values of the imaging pixels around the focus detection pixel among the pixel values constituting the flare-determined image are corrected by the weighting factor. Since the process of smoothing the pixel value of the corrected imaging pixel is performed a plurality of times, the influence of color mixture due to flare can be suppressed on the imaging pixels around the focus detection pixel. As a result, the influence of the color mixture can be suppressed even in the subsequent generation of the interpolation pixel value of the focus detection pixel.

  In the above embodiment, the arrangement direction of the AF pixels is the horizontal scanning direction. However, the present invention is not limited to this, and the AF pixels may be arranged in the vertical scanning direction or other directions.

  In each of the above embodiments, each AF pixel is a focus detection pixel that divides the light beam from the left or right side into pupils, but the present invention is not limited to this, and each AF pixel has a light beam from the left and right sides. Focus detection pixels having pixels that divide the pupil may be used.

  In the above embodiment, the flare suppression process is performed twice, but it is not always necessary to rotate the flare twice. For example, the flare suppression process may be performed once to correct the periphery of the flare, and the clip amount may be increased compared to the case where the flare suppression process is performed twice.

  Note that the present invention can also be applied to a program for realizing processing in an image processing apparatus including AF pixel interpolation processing by a computer.

  In each of the above embodiments, the present invention is described as the electronic camera 10, but may be an imaging device such as a camera-equipped mobile phone or a smartphone.

  Furthermore, the present invention is not limited to an image pickup apparatus, and a program for taking in image data such as RAW data not interpolating AF pixel values and determining flare and performing AF pixel interpolation, and a recording medium recording the same It is good.

DESCRIPTION OF SYMBOLS 10 Electronic camera 15 Image pick-up element 36 AF pixel 40 AF pixel interpolation part 42 Flare determination part

Claims (12)

An imaging device having an imaging pixel that generates a signal for an image and a focus detection pixel that generates a signal for focus detection ;
Flare determination means for determining whether flare has occurred based on an image obtained from the image sensor;
When the flare determination means determines that a flare has occurred, the pixel value of the imaging pixels around the focus detection pixel among the pixel values constituting the image is corrected by a weighting factor, and the corrected imaging Suppression means for performing flare suppression processing for smoothing the pixel value of the pixel;
Pixel interpolation processing means for generating an interpolation pixel value of a focus detection pixel using a pixel value of an imaging pixel in the vicinity of the focus detection pixel among pixel values of the imaging pixel smoothed by the suppression means;
Have that imaging device provided with a. The imaging device according to claim 1,
The suppressing means comprises a plurality of times rows cormorants imaging device the flare suppression processing. The imaging device according to claim 1 or 2,
The suppression means is
A surrounding imaging pixel correction unit that corrects a pixel value of an imaging pixel around the focus detection pixel by a weighting factor;
Smoothing means for smoothing the pixel values of the imaging pixels corrected by the surrounding imaging pixel correction means;
Have that imaging device provided with a. The imaging device according to claim 3.
The surrounding imaging pixel correction unit is configured to detect an imaging pixel positioned in the vicinity of the target imaging pixel from a pixel value of the target imaging pixel among imaging pixels positioned in the vicinity of the focus detection pixel. by subtracting the weighted sum of the pixel values and the weighting factors, correct the pixel value of the imaging pixel to be the target imaging device. In the imaging device according to claim 3 or 4,
The suppression means is
The processing performed by said smoothing means and said peripheral imaging pixel correction means performs 2 times by changing the weighting factor for an imaging pixel value, said smoothing means by the weighting count to "0" in the second treatment process only those rows cormorants imaging device in. In the imaging device according to any one of claims 3 to 5,
The smoothing means includes
Means for calculating a clip range using pixel values of adjacent AF pixels;
Means for calculating a prediction error from the difference between the pixel value of the proximal imaging pixel located at a position close to the pixel value of the distal imaging pixel located at a position far from the line of focus detection pixels arranged in a line;
Means for determining whether or not the prediction error exceeds a clip range;
Means for clipping a prediction error outside the clip range to be included in the clip range;
Means for adding the prediction error after clipping to the pixel value of the proximal imaging pixel;
Have that imaging device provided with a.   An imaging device having an imaging pixel that generates a signal for an image and a focus detection pixel that generates a signal for focus detection;
  The occurrence of flare is determined based on the signal from the image sensor, and when flare occurs, the pixel value of the imaging pixel is corrected, and the corrected pixel value of the imaging pixel is used for focus detection. A pixel interpolation processing unit that generates an interpolated pixel value that is a signal for an image in a pixel;
  An imaging apparatus comprising: Flare determination means for determining whether or not flare is generated based on an image acquired from an image sensor having an imaging pixel that generates an image signal and a focus detection pixel that generates a focus detection signal ; ,
When the flare determination means determines that a flare has occurred, the pixel value of the imaging pixels around the focus detection pixel among the pixel values constituting the image is corrected by a weighting factor, and the corrected imaging Suppression means for performing flare suppression processing for smoothing the pixel value of the pixel;
Pixel interpolation processing means for generating an interpolation pixel value of a focus detection pixel using a pixel value of an imaging pixel in the vicinity of the focus detection pixel among pixel values of the imaging pixel smoothed by the suppression means;
It has that images processing device provided with a. The image processing apparatus according to claim 8 .
The suppressing means comprises a plurality of times rows planted image processing apparatus the flare suppression processing. A flare determination step for determining whether or not a flare is generated based on an image acquired from an image pickup element having an image pickup pixel that generates an image signal and a focus detection pixel that generates a focus detection signal ; ,
When the flare determination step determines that a flare has occurred, the pixel value of the imaging pixels around the focus detection pixel among the pixel values constituting the image is corrected by a weighting factor, and the corrected imaging A suppression step for performing flare suppression processing to smooth the pixel value of the pixel;
A pixel interpolation processing step of generating an interpolation pixel value of the focus detection pixel using a pixel value of the imaging pixel in the vicinity of the focus detection pixel among the pixel values of the imaging pixel smoothed by the suppression step;
A program that causes a computer to execute. The program according to claim 10 , wherein
The suppression step is a program for causing a computer to execute processing for performing the flare suppression processing a plurality of times. The computer-readable recording medium which recorded the program of Claim 10 or 11 .

Images