JP6478670B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus and a control method thereof, and more particularly to a pixel value correction technique.

2. Description of the Related Art Conventionally, there has been known an image sensor in which some pixels are configured as focus detection pixels and phase difference detection type focus detection is possible using the output of the focus detection pixels.
However, since the focus detection pixel has a structure different from that of a normal pixel (imaging pixel), it may affect the pixel values around the focus detection pixel and cause image quality degradation. In Patent Literature 1, there is a method of correcting a value of an imaging pixel around a focus detection pixel based on a value of a focus detection pixel in the vicinity of the imaging pixel using a crosstalk ratio according to the aperture value. Proposed.

JP 2009-124573 A

  In the technique described in Patent Document 1, the position and the number of focus detection pixels that affect the correction target pixel in advance are identified in advance, and the correction target pixel value is assumed to be affected by the crosstalk at a specific ratio therefrom. Is corrected. However, the effect of actual crosstalk that occurs in each pixel to be corrected cannot be modeled so simply, and correction with high accuracy may not always be possible.

  The present invention has been made in view of the above-described problems of the prior art, and provides an image processing apparatus and a control method thereof that can accurately correct image quality degradation that occurs in adjacent pixel values of focus detection pixels. With the goal.

The above-described object is present in the vicinity of at least one of the plurality of correction target pixels and the first acquisition unit that acquires the values of the plurality of imaging pixels adjacent to the focus detection pixels as the values of the plurality of correction target pixels. And second acquisition means for acquiring values of a plurality of imaging pixels having the same color as the plurality of correction target pixels, not adjacent to the focus detection pixels, at least one of the values of the plurality of correction target pixels, A determination unit that determines whether at least one of the values of the plurality of correction target pixels needs to be corrected based on the values of the plurality of imaging pixels acquired by the second acquisition unit; A first correction unit that decreases the value of the correction target pixel determined to be necessary, and a second correction unit that increases the value of the plurality of correction target pixels including the value corrected by the first correction unit by the offset correction value. And having correction means It is achieved by an image processing apparatus according to symptoms.

  With such a configuration, according to the present invention, it is possible to provide an image processing apparatus and its control method capable of accurately correcting image quality degradation that occurs in adjacent pixel values of focus detection pixels.

1 is a block diagram illustrating a functional configuration example of a lens interchangeable digital single-lens reflex camera (camera system) as an example of an image processing apparatus according to an embodiment of the present invention. The figure which shows the example of arrangement | positioning of the focus detection area | region in 1st Embodiment, and the pixel arrangement | sequence in an image pick-up element. The figure which shows typically the structural example of the pixel in embodiment, and the generation principle of crosstalk Schematic diagram for explaining the principle of occurrence of crosstalk Flowchart for explaining crosstalk discrimination / correction processing in the first embodiment The figure which shows the example of the pixel array of the image pick-up element in 2nd Embodiment. Flowchart for explaining crosstalk discrimination / correction processing in the second embodiment

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the following embodiments, an example in which the present invention is applied to an imaging apparatus, specifically, a lens interchangeable digital single-lens reflex camera (DSLR) will be described. However, the present invention is not limited to shooting and recording functions. A specific configuration is not essential to the present invention. The present invention can be implemented in any electronic device that can acquire image data obtained by photographing and information on the position of focus detection pixels included in the image sensor used for photographing. The imaging device is not limited to a device having a main function of photographing such as a digital camera, but means any electronic device having a photographing function such as a mobile phone with a camera.

● (first embodiment)
FIG. 1 is a block diagram illustrating a functional configuration example of a lens interchangeable digital single-lens reflex camera (camera system) that is an example of an image processing apparatus according to an embodiment of the present invention.
The camera system includes a camera body 100 and a lens unit 200 that is detachably attached to the camera body 100. The camera body 100 and the lens unit 200 are connected to each other by a mount part, and an electrical contact group 210 is provided on the mount part. The contact group 210 communicates control signals, status signals, data signals, etc. between the camera body 100 and the lens unit 200, supplies power to the photographic lens from the camera body 100, and indicates whether the lens unit 200 is connected. It can be detected by the main body 100. Note that the contact group 210 may transmit a signal other than an electric signal, for example, an optical signal, as long as communication between the camera body 100 and the lens unit 200 is possible. For convenience, the photographing lens 201 built in the lens unit 200 is illustrated as a single lens, but actually includes a plurality of lenses.

  The light flux from the subject is guided to the quick return mirror 102 via the photographing lens 201 and the diaphragm 202. The quick return mirror 102 is movable in the direction of the arrow, and in the illustrated state (down state), the central portion is formed in a half mirror so that a part of the light beam incident from the lens unit 200 is transmitted. The light beam that has passed through the half mirror part is reflected in the direction of entering the AF sensor unit 104 by the sub mirror 103 provided on the back surface of the quick return mirror 102.

  The AF sensor unit 104 is composed of a field lens, a reflection mirror, a secondary imaging lens, a diaphragm, a line sensor, and the like disposed in the vicinity of the imaging surface, and is used for phase difference type automatic focus detection (phase difference AF). A pair of image signals to be used is generated. The focus detection circuit 105 uses a pair of image signals generated by the AF sensor unit 104 to acquire a defocus amount and a defocus direction. Based on the defocus amount and the defocus direction, the system controller 120 drives and controls the focus lens of the lens unit 200 to adjust the focus of the lens unit 200.

  On the other hand, among the light beams incident from the lens unit 200, the light beam reflected by the quick return mirror 102 in the down state is emitted through the pentaprism 101 and the eyepiece lens 106. The photographer can confirm the photographing range by observing the emitted light. A photometric sensor for obtaining luminance information of the subject is provided in the vicinity of the eyepiece lens 106, and the output of the photometric sensor is supplied to the system controller 120 via the photometric circuit 107. The system controller 120 performs automatic exposure control (AE) using the luminance information of the subject.

  At the time of shooting, the quick return mirror 102 moves upward (up state), and the light beam incident from the lens unit 200 enters the image sensor 112 through the focal plane shutter 108 and the optical filter 109 that are mechanical shutters. When the quick return mirror 102 is raised, the sub mirror 103 is folded.

  The optical filter 109 has both functions of an infrared cut filter and an optical low-pass filter. The focal plane shutter 108 has a front curtain and a rear curtain, and transmits or blocks the light flux from the lens unit 200 according to the control of the system controller 120.

  The system controller 120 includes a programmable processor such as a CPU or MPU, a non-volatile memory (ROM) for storing programs, setting values, GUI data, and the like, and a volatile memory (RAM) for expanding programs and using them as work areas. ). The system controller 120 executes the program and controls the operations of accessories such as the camera body 100, the lens unit 200, and the external flash, thereby realizing the functions of the entire camera system. Note that at least a part of the operation realized by the system controller 120 in software may be realized by a hardware circuit such as an ASIC.

  The system controller 120 is communicably connected to the lens controller 207 in the lens unit 200 via the contact group 210 described above. The lens controller 207 controls operations of the lens control circuit 204 and the aperture control circuit 206 under the control of the system controller 120 and transmits information on the lens unit 200 to the system controller 120. The information of the lens unit 200 may include, for example, the position of the focus lens, the set aperture value, the focal length of the photographing lens 201, and the like.

  In addition, the lens controller 207 is provided with a rewritable nonvolatile memory that stores information unique to the lens unit 200 (for example, focal length, wide aperture, individual identification information (lens ID)) and information received from the system controller 120. ing.

  The lens driving mechanism 203 drives the focus lens included in the photographing lens 201 in the optical axis direction under the control of the lens control circuit 204. The aperture drive mechanism 205 drives the aperture 202 according to the control of the aperture control circuit 206.

  The system controller 120 is connected to a shutter charge / mirror drive mechanism 110 that controls the up / down drive of the quick return mirror 102 and the shutter charge of the focal plane shutter 108. The system controller 120 is connected to a shutter control circuit 111 for controlling the traveling of the front curtain and rear curtain of the focal plane shutter 108. The EEPROM 122 stores parameters that need to be adjusted when the system controller 120 controls the camera system, individual identification information (camera ID) of the camera body 100, AF correction data and AE correction data adjusted by the reference lens, and the like. .

  The system controller 120 performs exposure control during shooting according to the aperture value (Av) and the shutter speed (Tv) determined by the AE control described above. Specifically, the size of the aperture of the diaphragm 202 is controlled via the lens controller 207, and the operation of the focal plane shutter 108 is controlled via the shutter control circuit 111.

  The front and rear curtains of the focal plane shutter 108 are spring driven and require a spring charging operation before operation. The shutter charge / mirror drive mechanism 110 controls the spring charge operation and also controls the up / down operation of the quick return mirror 102.

  The image data controller 115 is configured by, for example, a DSP (digital signal processor) and controls driving and reading operations of the image sensor 112. The image data controller 115 corrects or processes the image data digitized by the A / D converter 113 based on the control of the system controller 120. The correction / processing performed by the image data controller 115 includes, for example, color interpolation and white balance adjustment.

  Note that the image data controller 115 can acquire, for example, luminance information for each divided region of the image from the image data and supply the luminance information to the system controller 120. As described above, the system controller 120 can obtain the luminance information of the subject even when the output of the photometric sensor cannot be obtained in the mirror-up state.

  The timing pulse generation circuit 114 outputs a pulse signal necessary for driving the image sensor 112. The A / D converter 113 converts an analog signal corresponding to the subject image output from the image sensor 112 into a digital signal in accordance with the timing pulse from the timing pulse generation circuit 114. The DRAM 121 is an example of a storage device for temporarily storing image data (digital data) before, for example, processing or data conversion into a predetermined format.

  The image codec 119 is a circuit for encoding the image data stored in the DRAM 121 in a predetermined format (for example, JPEG format) and decoding the encoded image data. The encoded image data is recorded on the recording medium 400 as an image file. The recording medium 400 is a built-in memory or a removable storage medium, and is typically a memory card, but is not limited to this. Further, it may be recorded in an external device using wireless communication or the like.

  The image data controller 115 converts the image data on the DRAM 121 into an analog signal by the D / A converter 116 and outputs the analog signal to the encoder circuit 117. The encoder circuit 117 converts the output of the D / A converter 116 into a video signal (for example, NTSC signal) necessary for driving the image display circuit 118 which is generally a liquid crystal display panel.

  The operation display circuit 123 views information on the operation mode of the camera system, exposure information (shutter speed, aperture value, etc.) through the first display device 124 provided on the housing surface of the camera body 100 and the eyepiece 106. Display on the second display device 125 that can.

  The system controller 120 is connected to buttons and switches that constitute an operation unit for the user to perform various instructions and settings on the camera body 100. As these buttons and switches, the following are shown in FIG. A shooting mode selection button 130, a main electronic dial 131, a determination SW 132, a distance measuring point selection button 133, an AF mode selection button 134, a photometry mode selection button 135, a release SW1 136, a release SW2 137, and a viewfinder mode selection SW138. These are merely examples.

  Release SW1 136 and release SW2 137 are switches that are turned on when the release button is half-pressed and fully pressed. Turning on the release SW1 136 corresponds to an instruction to start a shooting preparation operation (AF, AE, etc.). Also, turning on the release SW2 137 corresponds to an instruction to start a shooting operation for recording.

  The viewfinder mode selection SW 138 is a switch for selecting whether or not the image display circuit 118 functions as an electronic view finder. If the image display circuit 118 does not function as an electronic viewfinder, the user can check the shooting range from the eyepiece 106 (optical viewfinder mode).

  Various external accessories may be attached to the camera body 100. In FIG. 1, an external flash 300 is detachably attached via a mount portion (for example, a hot shoe). An electrical contact group 310 is provided on the mount portion. The contact group 310 enables control signals, status signals, data signals, and the like to be communicated between the camera body 100 and the external flash 300, and allows the camera body 100 to detect whether the external flash 300 is connected. Communication between the camera body 100 and the external flash 300 is not limited to electrical communication, and optical communication or the like may be used. Further, the external accessory and the camera body 100 may communicate wirelessly, and thus the external accessory may not be mechanically connected to the camera body 100.

  A light emission control circuit 303 configured with an IGBT or the like controls light emission of the xenon (Xe) tube 301. A reflection shade 302 is disposed in the vicinity of the Xe tube 301. The charging circuit 304 generates a voltage of about 300 V in order to supply power to the Xe tube 301. The power source 305 is, for example, a dry battery. The flash controller 306 communicates with the system controller 120 to control light emission and charging of the external flash 300.

FIG. 2A is a diagram illustrating an example of a focus detection area (AF area) in the present embodiment. A total of 25 focus detection areas of horizontal 5 × vertical 5 are set inside the imaging range 250 of the image sensor 112. The focus detection pixels provided in the image sensor 112 are grouped for each focus detection area, and focus detection in a certain focus detection area is performed using the output of the focus detection pixel group in the focus detection area.
Note that the size and position of the focus detection area may be dynamically determined, for example, depending on the result of face detection.

  FIG. 2B is a diagram illustrating an example of a pixel arrangement of the image sensor 112 in the present embodiment. The imaging element 112 has normal pixels (imaging pixels) and focus detection pixels, and the imaging pixels are provided with primary color Bayer array color filters. In FIG. 2B, R, G, and B indicate imaging pixels corresponding to red, green, and blue color filters. Hereinafter, the pixel corresponding to the red color filter may be referred to as a red pixel or an R pixel. The same applies to pixels corresponding to color filters of other colors.

  PA and PB are focus detection pixels, and PA and PB are configured to receive light beams emitted from different partial areas out of the light beams emitted from the exit pupil of the lens unit 200. Focus detection is performed using a phase difference between a pair of image signals composed of an image (A image) generated from the output of the PA group and an image (B image) generated from the output of the PB group. Hereinafter, PA may be referred to as an A image pixel 351, and PB may be referred to as a B image pixel 352. In this embodiment, a green color filter is provided for both the A image pixel 351 and the B image pixel.

  Note that the configuration of the color filter provided in the image sensor and the arrangement of the focus detection pixels are merely examples, and other configurations may be used.

FIG. 3A is a diagram schematically showing the structure of one imaging pixel (R pixel, B pixel, G pixel) viewed from the direction indicated by the arrow in FIG.
A microlens 404, a color filter 403, a wiring layer 402, and a photodiode 401 are shown from the incident direction of the light beam. The wiring layer 402 corresponds to the lowest layer of the wiring having a layer configuration. As described below, the configuration of the opening of the wiring layer 402 differs between the focus detection pixel and the imaging pixel. Of the pixel configuration, those not related to the following description are not shown.

  Of the exit pupil 405 of the lens unit 200, the light beam emitted from the region 406 is guided to the light receiving region of the photodiode 401 by the microlens 404. As shown in FIG. 3A, in the imaging pixel, the curvature of the microlens 404 and the opening of the wiring layer 402 are formed so that the photodiode 401 receives the light beam emitted from almost the entire area of the exit pupil 405. ing.

FIGS. 3B and 3C are diagrams schematically showing the structure of the focus detection pixel, as in FIG. FIG. 3B shows the structure of an A image pixel 351 (PA), for example, and FIG. 3C shows the structure of a B image pixel 352 (PB), for example.
The A image pixel 351 and the B image pixel 352 differ in areas 506 and 516 in the exit pupil 405 from which the light beam incident on the photodiode 401 is emitted. Specifically, the regions 506 and 516 are controlled by making the center positions in the horizontal direction of the openings of the wiring layers 502 and 512 different between the A image pixel 351 and the B image pixel 352. Note that the openings of the wiring layers 502 and 512 are smaller than the openings of the wiring layer 402.

  Of the exit pupil 405 of the lens unit 200, the center of the region 506 corresponding to the light receiving range of the photodiode 401 of the A image pixel 351 is shifted to the left in the horizontal direction from the optical axis of the lens unit 200. Similarly, the center of the region 516 corresponding to the light receiving range of the photodiode 401 of the B image pixel 352 is shifted to the right in the horizontal direction from the optical axis of the lens unit 200. In this manner, pupil division is realized by controlling the size and position of the openings of the wiring layers 502 and 512, and a pair of image signals that can be used for phase difference focus detection can be generated.

  In the present embodiment, the color filter 403 of the focus detection pixel is assumed to be green (G), but a configuration in which no color filter is provided may be used.

  Next, image quality degradation due to optical crosstalk will be described with reference to FIGS. FIG. 3D shows a state in which two imaging pixels (G pixel and B pixel) are adjacent to each other. FIG. 3E shows an imaging pixel (G pixel) and a focus detection pixel (B image pixel). The adjacent state is shown in the same manner as in FIGS.

  In FIG. 3D, a light beam 609 incident on the B lens microlens 404b from the lens unit 200 is incident on the photodiode 401b directly below the microlens 404b. On the other hand, when a light beam 610 that does not normally enter from the lens unit 200 enters the microlens 404b, it passes through the color filter 403b of the B pixel and the color filter 403a of the G pixel and enters the photodiode 401a of the adjacent G pixel. . This is optical crosstalk (hereinafter simply referred to as crosstalk), which affects the output value of the G pixel.

  Also in the case of FIG. 3E, when the light beam 610 having an angle that is not normally incident from the lens unit 200 is incident on the micro lens 404c of the focus detection pixel, it is incident on the adjacent G pixel as in the case of FIG. , The output value of the G pixel is affected. However, since the color filter 403c of the focus detection pixel is green (G), the influence of crosstalk received by the output value of the G pixel is different from the case of FIG. More specifically, in the case of FIG. 3E, the influence of crosstalk received by the output value of the G pixel is larger (the output value becomes larger).

  In this way, the difference in the influence of crosstalk that neighboring pixels of the same color receive from neighboring pixels is negligible if they are adjacent to the imaging pixel, but one is the imaging pixel and the other is focus detection. When the pixel is adjacent to the target pixel, the image quality can be increased to such a degree that the image quality can be visually recognized. This problem occurs when the color of the color filter included in the focus detection pixel does not follow the arrangement of the color filters provided in the imaging pixel (including the case where the focus detection pixel does not have a color filter).

  In FIG. 3D and FIG. 3E, the light beam 609 incident at an angle that is not normally incident from the lens unit 200, which causes crosstalk, is, for example, that the light beam incident from the lens unit 200 is reflected in the image sensor. Occurs.

FIG. 4 is a diagram schematically illustrating a state in which the light beam incident from the lens unit is reflected in the imaging element.
The imaging element 112 includes a light receiving unit 701 in which the pixels illustrated in FIG. 3 are arranged, and a cover glass 702 provided in the vicinity of the microlens 404 provided on the surface of the light receiving unit 701. Reference numeral 703 denotes a light shielding mask for cutting off reflected light from the inner surface of the lens barrel of the lens unit 200.

  A part of the light beam incident on the image sensor 112 from the subject 705 via the lens unit 200 is reflected on the surface of the cover glass 702 or the micro lens 404. When a part of the light beam diffusely reflected by the surface of the microlens 404 is re-reflected by the cover glass 702 and enters the microlens 404 again, it can become the light beam 609 that causes the crosstalk described above.

  Since such reflected light increases as the luminance of the incident light beam increases, crosstalk also increases as the luminance of the incident light beam increases. However, since it is not conspicuous in a scene without a large luminance difference, it does not cause much problem. On the other hand, in a scene where there is a high-luminance subject having a large luminance difference from the surroundings, such as the sun or a spotlight, the influence of crosstalk on the pixel value is easily understood. However, even in this case, when there are no focus detection pixels, the variation in the magnitude of the crosstalk is small, so that it is not so conspicuous as visually recognized as image quality degradation. However, if there is a focus detection pixel, there is a difference in the magnitude of the effect of crosstalk, and this is recognized as image quality degradation.

  Further, as shown in the figure, the pixel on which the reflected light that causes crosstalk is incident is a position away from the pixel on which the high-luminance luminous flux that is the source of the reflected light is incident. When the reflected light is incident on a pixel included in a region where the luminance and contrast are low, image quality deterioration due to crosstalk is noticeably observed.

  Therefore, when a focus detection pixel that does not follow the arrangement of the color filters applied to the image pickup pixel is formed on the image sensor, a cross caused by a light flux that enters the focus detection pixel and reaches the photodiode of another pixel is formed. It is necessary to correct so that the influence of the talk is not noticeable.

  FIG. 2C is a diagram showing an extracted region surrounded by a thick frame in FIG. In FIG. 2C, pixels GU, GD, GL, and G * are imaging pixels adjacent to the top, bottom, left, and right of the A image pixel 351 that may be affected by the crosstalk of the A image pixel 351 (all G Pixel). In the present embodiment, as shown in FIG. 2B, the A image pixels 351 are arranged every other pixel in the horizontal direction, so that they are adjacent to the right (left) of a certain A image pixel 351. The pixel is also a pixel adjacent to the left (right) of another A image pixel 351. Therefore, the correction processing for these pixels is prevented from being repeated. In the present embodiment, correction processing is performed once for the pixels GU, GD, and GL adjacent to the upper, lower, and left sides of each A image pixel 351 (the pixel G * adjacent to the right is different). To be corrected in the first correction process). The pixels G1, G2, G3, and G4 are pixels (reference pixels) that are referred to in order to determine whether or not the correction target pixel actually needs correction. The reference pixel is an imaging pixel that is present around the correction target pixel and has the same color as the correction target pixel.

  Note that the four R pixels adjacent to the A image pixel 351 at the corners are also affected by the crosstalk of the A image pixel 351 similarly to the pixels GU, GD, and GL, but two R pixels adjacent to the upper and lower sides of G *. Is not subject to correction in the same manner as G *, and R pixels adjacent to the top and bottom of GL are subject to correction. Since the correction target R pixel can be corrected in the same manner as the pixels GU, GD, and GL except that the reference pixel is different, correction processing for the pixels GU, GD, and GL will be described below.

  FIG. 5 is a flowchart showing a procedure for determining whether correction of a value is necessary for a pixel to be corrected and for correcting the pixel value. In the present embodiment, these processes can be executed by the system controller 120 or the image data controller 115 on the image data stored in the DRAM 121 and before the demosaic (color interpolation) process. Therefore, in the processing described here, a pixel provided with a color filter has a value for one color component corresponding to the color of the color filter. Here, the description will be made assuming that the system controller 120 executes processing.

  In step S901, the system controller 120 sets a setting value C for a variable LoopEnd that sets the number of repetitions of processing and 0 for a variable Loop used as a counter for the number of processing times, and advances the processing to step S902. Actually, since correction target pixels related to one focus detection pixel are processed in one process, the number of times equal to the number of focus detection pixels is set to C, but in the following description, in order to facilitate understanding and description , C is 1. Further, the system controller 120 (first acquisition unit) acquires values of the correction target pixels GU, GD, and GL.

In step S902, the system controller 120 (second acquisition unit) acquires values of reference pixels (here, pixels G1, G2, G3, and G4), calculates an average value G_ave, and advances the process to step S903.
In step S903, the system controller 120 detects the maximum value G_max of the correction target pixels GU, GD, and GL, and advances the processing to step S904. As shown in FIG. 3E, the crosstalk to be corrected is generated in a pixel that exists in a direction opposite to the incident direction of the reflected light. For example, when reflected light enters from the right side of the focus detection pixel, the value of the pixel adjacent to the left side of the focus detection pixel is affected by crosstalk. In a general shooting scene, it is unlikely that reflected light that causes crosstalk will enter one focus detection pixel from a plurality of directions. For this reason, it is considered that the pixel having the largest value (high output) among the plurality of adjacent same-color pixels of the focus detection pixels is most likely to be affected by crosstalk. Therefore, in the present embodiment, the maximum value of the correction target pixel is detected as the pixel value to be compared with the value of the reference pixel.

  In step S904, the system controller 120 calculates a difference Sub between G_max and G_ave, and advances the process to step S905. Since the reference pixel is the same color pixel in the vicinity of the correction target pixel, the value of the difference Sub does not increase unless the correction target pixel is affected by crosstalk.

In step S905, the system controller 120 calculates the absolute value Edge of the difference between the maximum value and the minimum value of the reference pixels G1 to G4, which is expressed by the following equation, as a reference value for the high-frequency component in the surrounding pixels, and performs processing in step S906. Proceed to
Edge = | MAX (G1-G4) -MIN (G1-G4) |
In a portion having a high spatial frequency, such as a contour portion of a subject, the value of Edge is large.

  In S906, the system controller 120 calculates a value (A × √ (G_ave)) obtained by multiplying the square root of G_ave by a positive number A as the upper limit value σ, and the process proceeds to S907. In this embodiment, A is 5.

  In step S907, the system controller 120 (determination unit) determines whether it is necessary to correct the value of the correction target pixel having the maximum value G_max. Specifically, if the sum of Sub and Edge is greater than the lower limit value α and less than the upper limit value σ (if the condition of α <Sub + Edge <σ is satisfied), the system controller 120 determines that correction is necessary and performs processing in S908. Proceed to On the other hand, if the sum of Sub and Edge is equal to or lower than the lower limit value α or equal to or higher than the upper limit value σ, the system controller 120 determines that correction is unnecessary and advances the process to S910.

  If crosstalk to be corrected has occurred, the value of Sub increases and exceeds the lower limit threshold value α. α may be an arbitrary fixed value or function. However, when the number of pixels used for sub calculation is small as in the present embodiment, if α is too small, the above condition is accidentally satisfied by the influence of random noise, and erroneous correction is performed. The potential increases. Since it is known that the magnitude of the random noise of a pixel depends on the signal amount (luminance) of the pixel, it is more accurate to use α as a function of the signal amount or luminance of the pixel than to set α to a fixed value. Judgment is possible. However, in the present embodiment, α is a fixed value (50 LSB) for ease of explanation and understanding. Note that 0LSB corresponds to the output of the A / D converter 113 corresponding to the optical black level, and is different from the absolute value of the digital value.

  Since the value of Sub also increases in the bright spot portion and contour portion of the subject, correction processing is not performed in a range where Sub + Edge is a value equal to or greater than the upper limit value σ in order to suppress erroneous correction. The value of the coefficient A that determines the upper limit value σ can be determined experimentally, for example.

  In step S908, the system controller 120 (first correction unit) is in the vicinity of the pixel to be corrected (correction target pixel having the maximum value G_max) and is not adjacent to the focus detection pixel and has the same color as the pixel to be corrected. The value of the imaging pixel is acquired. Then, the system controller 120 calculates the average value of the acquired imaging pixel values as the correction value G_ref, and advances the processing to S909. For example, the system controller 120 can use the average value G_ave of the reference pixels G1 to G4 as the correction value G_ref.

Alternatively, the system controller 120 can calculate, as the correction value G_ref, the average value of a plurality of imaging pixels having the same color as the pixel to be corrected and the closest position. For example, when the pixel to be corrected is GU, the system controller 120 calculates an average value of four G pixels G1, G2, GL, and G * closest to the pixel GU as the correction value G_ref. In this way, by using an average value of a plurality of pixels having the same color and closest position as the pixel to be corrected as a correction value, it is possible to prevent a reduction in resolution due to the correction.

In step S909, the system controller 120 (first correction unit) calculates a weighted average value (β × G_ref + (1−β) × G_max ) between the correction value G_ref and the pixel value G_max to be corrected as a corrected value. To do. Then, the system controller 120 (first correction unit) reduces the value of the pixel GU by replacing the value of the pixel GU to be corrected with the calculated value, and advances the process to S910. The mixing rate β (0 ≦ β ≦ 1) may be a predetermined fixed value, for example, β = 0.5. Alternatively, the dispersion of the pixel values of the four G pixels G1, G2, GL, and G * used for obtaining the correction value G_ref is obtained, and the value of the mixing ratio β is increased as the dispersion is smaller, and the mixing is performed as the dispersion is larger. The rate β may be decreased. This is because the smaller the variation of these four pixel values (the smaller the variance), the smaller the influence of random noise and the higher the reliability of the correction value G_ref. Further, β = 1 may be used to replace the pixel value with the correction value G_ref for correction.

  In S910, the system controller 120 calculates a value (B × √ (G_ave)) obtained by multiplying the square root of G_ave by a positive number B as the offset correction value Offset for the correction target pixels GU, GD, and GL, and the process proceeds to S911. . The value of B can be obtained experimentally, but is 0.25 in this embodiment.

In step S911, the system controller 120 (second correction unit) increases the pixel value by adding the offset correction value Offset to each value of the correction target pixels GU, GD, and GL, and advances the process to step S912.
By determining whether or not correction is necessary using the lower limit α in S907, the possibility of erroneously correcting the value of a pixel whose value has increased due to random noise is suppressed, but it is difficult to completely eliminate the erroneous correction. Therefore, in consideration of the possibility of erroneous correction, if the corrected value is erroneously corrected, an offset correction value is set so as to make it inconspicuous, and added to the value of the correction target pixel including the corrected pixel. To do.

In step S913, the system controller 120 increments the value of the variable Loop and advances the process to step S914.
In S914, the system controller 120 compares the variable Loop with the variable LoopEnd. The system controller 120 repeats the processing from S903 if the value of the variable Loop has not reached the value of the variable LoopEnd, and ends the processing if the value of the variable Loop has reached the value of the variable LoopEnd.

  In the present embodiment, a pixel adjacent to the focus detection pixel is subjected to the crosstalk correction by determining whether or not the influence of the crosstalk that needs to be corrected has occurred. Can be corrected with high accuracy. In addition, since offset correction is performed on neighboring pixels of the same color including the corrected pixel, it is possible to suppress the influence of erroneous correction caused by random noise, and it is possible to realize more accurate correction.

  In particular, when the focus detection pixels are provided at a high density such as every other pixel, the pixels affected by the crosstalk are also present at a high density, so that even if the influence of the individual crosstalk is small, the pixel is easily noticeable. . Further, it is difficult to completely discriminate whether the increase factor of the pixel value is random noise or crosstalk, and it is also difficult to completely eliminate erroneous correction. The influence of erroneous correction is also conspicuous for the same reason as the influence of crosstalk, particularly when focus detection pixels are arranged at a high density. Therefore, assuming that a certain amount of erroneous correction occurs, the offset correction is performed so that the average value of the pixels in the row in which the focus detection pixels are provided is equal to the pixel in the row in which the adjacent focus detection pixels are not provided. Even if an error correction occurs near the average value, it can be made inconspicuous.

● (Second Embodiment)
Next, a second embodiment of the present invention will be described. The present embodiment relates to crosstalk correction in the case where an image sensor having a different focus detection pixel arrangement from that of the first embodiment is used. Except for the focus detection pixel arrangement and the crosstalk determination / correction process, the process may be the same as that of the first embodiment. Therefore, the configuration and processes unique to the present embodiment will be described below.

  FIG. 6 is a diagram showing an example of the pixel arrangement of the image sensor 112 ′ in the present embodiment in the same manner as FIG. Similar to the first embodiment, the image sensor 112 ′ has normal pixels (imaging pixels) and focus detection pixels, and the image pixels are provided with a primary color Bayer array color filter. In the present embodiment, the A image pixel 351 and the B image pixel 352 are not adjacent to the pixel row adjacent to the direction orthogonal to the pupil division direction (here, the horizontal direction) and not to the direction orthogonal to the pupil division direction ( (The positions in the pupil division direction are different). In the example of FIG. 6A, in particular, the A image pixel 351 and the B image pixel 352 are arranged at the R pixel position in the primary color Bayer array and the B pixel position in the adjacent row, respectively.

  Also in this embodiment, the focus detection pixel is provided with a green color filter. However, a transparent filter may be provided or a filter may not be provided. Therefore, a green color filter is provided in each of the pixels included in two adjacent pixel rows where the A image pixel 351 and the B image pixel 352 are arranged.

  FIG. 6B is a diagram showing a region surrounded by a thick line in FIG. The pixels GA, GB, GC, and GD are imaging pixels adjacent to at least one of the A image pixel 351 and the B image pixel 352, and are affected by the crosstalk of at least one of the A image pixel 351 and the B image pixel 352. there is a possibility. In the present embodiment, one correction process is performed with the pixels GA, GB, GC, and GD as correction targets. In this embodiment, since the imaging pixel adjacent to the A image pixel 351 and the imaging pixel adjacent to the B image pixel 352 are to be corrected by one correction process, correction is performed for a maximum of two pixels. Run. The pixels G1, G2, G3, and G4 are pixels (reference pixels) that are referred to in order to determine whether or not the correction target pixel actually needs correction. The reference pixel is an imaging pixel that is present around the correction target pixel and has the same color as the correction target pixel.

  Note that the two R pixels adjacent to the A image pixel 351 at the corners and the two B pixels adjacent to the B image pixel 352 at the corners are also affected by the crosstalk of the A image pixel 351 and the B image pixel 352. However, the R pixel adjacent to the upper right corner of the A image pixel 351 and the B pixel adjacent to the lower left corner of the B image pixel 352 are not subject to correction, and the R pixel 1002 adjacent to the upper left corner of the A image pixel 351, A B pixel 1003 adjacent to the lower right corner of the image pixel 352 is a correction target. Since the correction target R pixel 1002 and B pixel 1003 can be corrected in the same manner as the pixels GA to GD except that the reference pixels are different, the correction processing of the pixels GA to GD will be described below.

In the arrangement of the focus detection pixels according to the present embodiment, the influence of crosstalk generated in the correction target pixel differs depending on the direction adjacent to the focus detection pixel.
The pixel GA adjacent on the A image pixel 351 may receive crosstalk of the A image pixel 351 from below.
The pixel GB adjacent to the lower side of the A image pixel 351 and to the right of the B image pixel 352 may be subjected to crosstalk of the A image pixel 351 from the upper direction and crosstalk of the B image pixel 352 from the lateral direction.
The pixel GC adjacent to the left of the A image pixel 351 and the B image pixel 352 may receive crosstalk of the A image pixel 351 from the horizontal direction and crosstalk of the B image pixel 352 from the lower direction.
The pixel GD adjacent below the B image pixel 352 may receive crosstalk of the B image pixel 352 from above.
It is therefore necessary to detect and correct each of these possible conditions.

  FIG. 7 is a flowchart showing a procedure for determining whether correction of a value is necessary for a pixel to be corrected and for correcting the pixel value. Steps for performing the same processing as in FIG. 5 are given the same reference numerals, and descriptions thereof are omitted.

In S1203, the system controller 120 detects the maximum value G_max1 and the second largest value G_max2 of the correction target pixels GA to GD, and advances the process to S1204.
In step S1204, the system controller 120 calculates a difference Sub1 between G_max1 and G_ave and a difference Sub2 between G_max2 and G_ave, and advances the process to step S1205.

In step S905, the system controller 120 calculates the absolute value Edge of the difference between the maximum value and the minimum value of the reference pixels G1 to G4, which is expressed by the following formula, as a reference value for the high-frequency component in the surrounding pixels, and performs the processing in step S1206. Proceed to
Edge = | MAX (G1-G4) -MIN (G1-G4) |

In step S <b> 1206, the system controller 120 determines that the value of the pixel GD adjacent below the focus detection pixel with respect to the value of the pixel GA adjacent above the focus detection pixel (the magnitude relationship between GA and GD) The direction (up / side / down) in which the talk is considered to occur is determined. Then, the system controller 120 (direction discriminating means)
If the ratio is less than 0.25 (GA / GD <0.25), go to S1207.
If the ratio is 0.25 or more and less than 0.75 (0.25 ≦ GA / GD <0.75), go to S1209.
If the ratio is 0.75 or more (0.75 ≦ GA / GD), go to S1210.
Each process is branched.

In S1207, the system controller 120 sets a value obtained by multiplying the square root of G_ave by a positive number A1 (A1 × √ (G_ave)) as the upper limit value σ1, and a value obtained by multiplying the square root of G_ave by a positive number A2 as the upper limit value σ2 (S1207). A2 × √ (G_ave)) is calculated.
Also in S1209 and S1210, the system controller 120 calculates the upper limit values σ1 and σ2 using the positive numbers A3, A4, A5, and A6 in the same manner as in S1207.

  When the upper limit values σ1 and σ2 are calculated, the system controller 120 advances the process to S1211. The purpose of the upper limit values σ1 and σ2 is to prevent crosstalk correction from being performed on pixel values that are large due to factors other than crosstalk, such as the bright spot portion and contour portion of the subject. Therefore, the positive numbers A1 to A6 can be obtained experimentally using real images corresponding to the directions of the crosstalk based on such a purpose.

  In step S1210, the system controller 120 determines whether it is necessary to correct the value of the correction target pixel having the maximum value G_max1. Specifically, if the sum of Sub1 and Edge is greater than the lower limit value α1 and less than the upper limit value σ1 (if α1 <Sub1 + Edge <σ1), the system controller 120 determines that correction is necessary and advances the process to S1212. On the other hand, if the sum of Sub1 and Edge is equal to or lower than the lower limit value α1 or equal to or higher than the upper limit value σ1, the system controller 120 determines that correction is unnecessary including the correction target pixel having the second largest value G_max2, and the process proceeds to S1217. Proceed. As in the first embodiment, the lower limit value α1 may be an arbitrary fixed value or a function. However, similarly to the first embodiment, α1 is set to 50 LSB in the present embodiment.

  In step S1212, the system controller 120 (first correction unit) is present in the vicinity of the pixel to be corrected (correction target pixel having the maximum value G_max1) and is not adjacent to the focus detection pixel and has the same color as the pixel to be corrected. The values of a plurality of imaging pixels are acquired. Then, the system controller 120 calculates the average value of the acquired imaging pixel values as the correction value G_ref1, and advances the processing to S1213. For example, the system controller 120 can use the average value G_ave of the reference pixels G1 to G4 as the correction value G_ref1.

  Alternatively, the system controller 120 can calculate, as the correction value G_ref1, an average value of a plurality of pixels having the same color as the pixel to be corrected and the closest positions. For example, when the pixel to be corrected is GA, the system controller 120 calculates the average value of the two G pixels G1 and G2 closest to the pixel GA as the correction value G_ref. Note that the two G pixels adjacent to the left and right lower corners of the pixel GA (pixels adjacent to the left and right of the A image pixel) are also closest to the pixel GA, but may be affected by the crosstalk of the focus detection pixel. Excluded. However, even pixels adjacent to the focus detection pixels that are determined not to be affected by the crosstalk from the crosstalk direction determined in S1206 may be used for the calculation in S1212. In this way, by using an average value of a plurality of pixels having the same color and closest position as the pixel to be corrected as a correction value, it is possible to prevent a reduction in resolution due to the correction.

In step S1213, the system controller 120 (first correction unit) calculates a weighted average value (β × G_ref1 + (1−β) × G_max1 ) of the correction value G_ref1 and the pixel value G_max1 to be corrected as a corrected value. To do. Then, the system controller 120 replaces the value of the pixel GA to be corrected with the calculated value, and advances the process to S1214. The mixing rate β (0 ≦ β ≦ 1) may be a predetermined fixed value, for example, β = 0.5. Alternatively, β = 1 may be used to replace the pixel value with the correction value G_ref1 for correction.

  In steps S1214 to S1216, the system controller 120 determines whether or not correction is necessary and corrects the pixel value of the pixel having the value G_max2, similarly to steps S1211 to S1213, and advances the processing to step S1217. The lower limit σ2 can be determined in the same manner as the lower limit σ1, but in this embodiment, the lower limit σ2 is 50 LSB equal to σ1.

In step S <b> 1217, the system controller 120 sets a value obtained by multiplying the square root of G_ave by positive numbers B <b> 1 to B <b> 4 as offset correction values Offset_A to Offset_D for each of the correction target pixels GA to GD, Offset_A = B1 × √ (G_ave).
Offset_B = B2 × √ (G_ave)
Offset_C = B3 × √ (G_ave)
Offset_D = B4 × √ (G_ave)
Is calculated, and the process advances to step S1218. The values of B1 to B4 can be experimentally determined in advance. Note that the values of B1 to B4 may be determined for each crosstalk direction (up / horizontal / down) determined in S1206, and a value corresponding to the direction determined in S1206 may be used in S1217. . For example, among the correction target pixels GA to GD, the offset correction value of the pixel at the position that is not affected by the crosstalk in the direction determined in S1206 is larger than the offset correction value of the pixel at the position that can be affected by the crosstalk. B1 to B4 can be determined to be smaller.

  In addition, as the values of the positive numbers A1 to A6 used in S1207, S1209, and S1210 are larger, the conditions used for determining whether correction is necessary in S1211, S1214 become looser, and the probability that the pixel value is corrected is higher. The difference between the pixel values before and after (that is, the correction amount) can be large. That is, the larger the positive numbers A1 to A6 are, the greater the influence of erroneous correction is. Therefore, among the positive numbers A1 to A6, B1 to B4 can be determined so that the offset correction value increases as the larger positive number is used. Alternatively, the offset correction value can be increased when the condition used for determining whether or not correction is necessary is the second condition, which is looser than the first condition. Then, according to the crosstalk direction determined in S1206 (which step of S1207, S1209, and S1210 has been executed), the corresponding values of B1 to B4 can be used.

  In step S1218, the system controller 120 (second correction unit) adds the offset correction value Offset_A to the value of the correction target pixel GA. Similarly, the system controller 120 adds Offset_B to Offset_D to the values of the other correction target pixels GB to GD, and advances the processing to S912. Since the subsequent steps are the same as those in the first embodiment, description thereof is omitted.

  As described above, even when the focus detection pixels are arranged at a higher density than in the first embodiment, the image quality degradation caused by the crosstalk of the focus detection pixels is reduced as in the first embodiment. It is possible to correct with high accuracy.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

DESCRIPTION OF SYMBOLS 100 ... Camera body, 112 ... Image sensor, 115 ... Image data controller, 120 ... System controller, 200 ... Lens unit

Claims (13)

First acquisition means for acquiring values of a plurality of imaging pixels adjacent to the focus detection pixels as values of a plurality of correction target pixels;
Second acquisition means for acquiring values of a plurality of imaging pixels having the same color as the plurality of correction target pixels that are present in the vicinity of at least one of the plurality of correction target pixels and are not adjacent to the focus detection pixels; ,
It is necessary to correct at least one of the values of the plurality of correction target pixels based on at least one of the values of the plurality of correction target pixels and the values of the plurality of imaging pixels acquired by the second acquisition unit. Determination means for determining whether or not there is,
First correction means for reducing the value of the correction target pixel determined to be corrected by the determination means;
Second correction means for increasing the value of the plurality of correction target pixels including the value corrected by the first correction means by an offset correction value;
An image processing apparatus comprising:   The image processing apparatus according to claim 1, wherein at least one of the values of the plurality of correction target pixels is selected from among the values of the plurality of correction target pixels.   The determination unit is configured to determine the plurality of correction target pixels based on a difference between at least one of the plurality of correction target pixel values and an average value of the plurality of imaging pixel values acquired by the second acquisition unit. The image processing apparatus according to claim 1, wherein it is determined whether at least one of the values needs to be corrected. The determination means includes
A difference between at least one of the values of the plurality of correction target pixels and an average value of the plurality of imaging pixels acquired by the second acquisition unit;
The difference between the maximum value and the minimum value of the plurality of imaging pixels acquired by the second acquisition means;
Is larger than a predetermined lower limit value and smaller than an upper limit value based on an average value of the plurality of imaging pixels acquired by the second acquisition unit, the value of the plurality of correction target pixels The image processing apparatus according to claim 3, wherein it is determined that at least one correction is necessary.   The image processing apparatus according to claim 4, wherein the second correction unit increases the offset correction value as the upper limit value increases.   The first correction unit applies the value of the correction target pixel determined to be corrected by the determination unit to the focus detection pixel that is in the vicinity of the correction target pixel and has the same color as the correction target pixel. 6. The image processing apparatus according to claim 1, wherein the image processing apparatus is decreased by using a correction value based on values of a plurality of imaging pixels that are not adjacent to each other.   The correction value is the closest to the correction target pixel that is determined to need to be corrected by the determination unit, and has the same color as the correction target pixel and is not adjacent to the focus detection pixel. The image processing apparatus according to claim 6, wherein the image processing apparatus is based on a value.   The said 2nd correction | amendment means uses the value based on the average value of the value of the some imaging pixel which the said 2nd acquisition means acquired as said offset correction value, Any one of Claim 1 to 7 characterized by the above-mentioned. The image processing apparatus according to claim 1.   The second correction unit is more effective when the determination unit performs the determination using the second condition that is looser than the first condition than when the determination unit performs the determination using the first condition. The image processing apparatus according to claim 1, wherein the offset correction value is increased.   The condition used by the determination unit for the determination is determined based on values of a plurality of correction target pixels having different directions adjacent to a focus detection pixel among the plurality of correction target pixels. The image processing apparatus according to any one of 1 to 9. An imaging device having focus detection pixels and imaging pixels;
The image processing apparatus according to any one of claims 1 to 10,
An imaging device comprising: A first acquisition unit that acquires values of a plurality of imaging pixels adjacent to the focus detection pixels as values of a plurality of correction target pixels;
The second acquisition unit acquires values of a plurality of imaging pixels having the same color as the plurality of correction target pixels that are present in the vicinity of at least one of the plurality of correction target pixels and are not adjacent to the focus detection pixel. A second acquisition step,
Based on at least one of the values of the plurality of correction target pixels and the values of the plurality of imaging pixels acquired by the second acquisition unit, the determination unit determines at least one of the values of the plurality of correction target pixels. A determination step for determining whether or not correction is necessary,
A first correction step in which a first correction unit decreases a value of a correction target pixel determined to be corrected by the determination unit;
A second correction step, wherein a second correction unit increases a value of the plurality of correction target pixels including the value corrected by the first correction unit by an offset correction value;
A control method for an image processing apparatus, comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 10.

Images