JP2019114912A - Image processing device and image processing method, and imaging apparatus - Google Patents

Abstract

To provide an image processing device capable of reducing fluctuation of the amount of light in a generated viewpoint image when LF (Light Field) data is used.SOLUTION: The image processing device is configured to correct the brightness of a viewpoint image which is generated from signals obtained by part of a plurality of photoelectric conversion parts each of which receives a light flux passing through a different partial area in a pupil region of imaging optical system. The correction amount of the brightness is generated by using projection signals of a pick-up image which is generated from signals obtained by all photoelectric conversion parts and projection signals which are determined as valid in the projection signals of the viewpoint image.SELECTED DRAWING: Figure 12

Description

  The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus.

  As one of focus detection methods of an imaging apparatus, an imaging surface phase difference method is known in which phase difference focus detection is performed using a signal obtained by an imaging device. Patent Document 1 discloses an imaging device using a two-dimensional imaging element in which one microlens and a photoelectric conversion unit divided into a plurality of pixels are formed for one pixel. The photoelectric conversion unit divided into a plurality of parts is configured to receive light fluxes that have passed through different regions of the exit pupil of the photographing lens via one microlens, and so-called pupil division can be realized. Since the image shift amount can be calculated from the parallax between a plurality of viewpoint signals obtained by pupil division, focus detection of the phase difference method can be performed by converting the obtained image shift amount into the defocus amount. it can. Further, Patent Document 2 discloses that an imaging signal is generated by adding a plurality of viewpoint signals received by a plurality of divided photoelectric conversion units.

  Note that, as described above, the photoelectric conversion unit divided into a plurality of pieces receives the light flux that has passed through different regions of the exit pupil of the photographing lens, so that it is possible to acquire spatial distribution and angular distribution of light intensity. Therefore, a plurality of viewpoint signals captured using a plurality of divided photoelectric conversion units are equivalent to light field (LF) data including information on spatial distribution of light intensity and angular distribution.

U.S. Pat. No. 4,410,804 Japanese Patent Application Publication No. 2001-083407

  However, in the case of using LF data acquired by the imaging device disclosed in the above-mentioned patent document, due to the deviation between the exit pupil of the imaging lens and the entrance pupil of the imaging device, A bias may occur. For this reason, fluctuation of the light quantity (shading) may occur in the viewpoint image to deteriorate the image quality and the focus detection accuracy of the phase difference method may be deteriorated. In the patent document mentioned above, the fluctuation of the light quantity which arises in a viewpoint picture is not taken into consideration, and when using LF data, the art which reduces the fluctuation of the light quantity of the generated viewpoint picture is called for.

  The present invention has been made in view of the above problems, and when using LF data, an image processing apparatus, an image processing method, and an imaging apparatus capable of reducing the fluctuation of the light quantity of a viewpoint image to be generated. Intended to provide.

  The above object is to provide a viewpoint image generated from signals obtained by a part of photoelectric conversion units of a plurality of photoelectric conversion units that receive light beams passing through different partial regions of a pupil region of an imaging optical system, Image acquisition means for acquiring a captured image generated from signals obtained by all the photoelectric conversion units, correction amount acquisition means for acquiring a luminance correction amount of a viewpoint image using a viewpoint image and a captured image, luminance correction An image processing apparatus comprising: correction means for correcting the luminance of a viewpoint image based on an amount, wherein the correction amount acquisition means comprises: projection means for generating a projection signal of each of the captured image and the viewpoint image; A determination unit that determines the validity of a signal; a generation unit that generates a luminance correction amount using a projection signal that is determined to be effective as the determination unit among projection signals of a viewpoint image and a projection signal of a captured image; To have It is achieved by an image processing apparatus according to symptoms.

  According to the present invention, it is possible to provide an image processing device, an image processing method, and an imaging device capable of reducing the fluctuation of the light amount of a viewpoint image to be generated when using LF data.

A block diagram showing an example of the functional configuration of a digital camera as an example of an image processing apparatus according to the first embodiment A diagram schematically showing a pixel arrangement in the first embodiment A schematic plan view (A) and a schematic cross-sectional view (B) of a pixel in the first embodiment A diagram schematically showing the relationship between pixels and pupil division in the first embodiment A diagram showing an example of light intensity distribution inside a pixel in the first embodiment The figure which illustrates the pupil intensity distribution in a 1st embodiment A diagram schematically showing the relationship between an imaging device and pupil division in the first embodiment A diagram schematically showing the relationship between the defocus amount of the first viewpoint image and the second viewpoint image and the image shift amount in the first embodiment A flowchart (first half) showing a series of operations of correction processing in the first embodiment Flowchart showing a series of operations of correction processing in the first embodiment (second half) A diagram for explaining shading due to pupil shift of the first viewpoint image and the second viewpoint image in the first embodiment Flow chart on validity determination processing in the first embodiment The figure which shows the example of the projection signal of the captured image in 1st Embodiment, the projection signal of a viewpoint image, and a shading curve FIG. 6 illustrates an example of (A) a captured image, (B) a first viewpoint image before shading correction, and (C) a first viewpoint image after shading correction in the first embodiment. The figure which illustrates the (A) 1st viewpoint image before defect correction in a 1st embodiment, and the (1) 1st viewpoint image after defect correction. A diagram exemplifying (A) a second viewpoint image before correction and (A) a second viewpoint image after correction in the first embodiment

  Hereinafter, exemplary embodiments of the present invention will be described in detail based on the drawings. In the following embodiments, the present invention is applied to a digital camera as an example of an image processing apparatus, but the present invention can be applied to any apparatus capable of processing LF data. The optional devices may include, for example, personal computers, tablets, game machines, wearable terminals such as glasses and watches, in-vehicle systems, surveillance camera systems, and medical devices.

First Embodiment
An exemplary functional configuration of a digital camera 100 as an example of an image processing apparatus according to the first embodiment will be described with reference to FIG.

  FIG. 1 is a block diagram showing an example of a functional configuration of an imaging device according to the present embodiment. The first lens group 101 disposed at the tip of the imaging lens (imaging optical system) is held by the lens barrel so as to be movable back and forth in the optical axis direction. The diaphragm / shutter 102 adjusts the light amount at the time of shooting by adjusting the aperture diameter, and also has a function as an exposure time adjustment shutter at the time of still image shooting. The second lens group 103 moves forward and backward in the optical axis direction integrally with the aperture / shutter 102 and has a variable power operation (zoom function) by interlocking with the forward and backward movement of the first lens group 101. The third lens group 105 is a focusing lens that performs focusing by moving back and forth in the optical axis direction. The optical low pass filter 106 is an optical element for reducing false color and moiré of a captured image. The imaging element 107 includes, for example, a two-dimensional CMOS (complementary metal oxide semiconductor) photosensor and a peripheral circuit, and is disposed on the imaging surface of the imaging optical system. Details of the imaging element 107 will be described later.

  The zoom actuator 111 rotates the cam barrel (not shown) to move the first lens group 101 and the second lens group 103 in the optical axis direction to perform a zooming operation. The aperture shutter actuator 112 controls the aperture diameter of the aperture / shutter 102 to adjust the imaging light amount, and performs exposure time control at the time of still image imaging. The focus actuator 114 moves the third lens group 105 in the optical axis direction to perform a focusing operation.

  The electronic flash 115 for object illumination includes a flash illumination device using a xenon tube or an illumination device provided with a continuously emitting LED (Light Emitting Diode), and is used at the time of photographing. An AF (Auto Focus) auxiliary light source 116 projects an image of a mask having a predetermined aperture pattern onto a field via a light projection lens. This improves the focus detection capability for low brightness subjects or low contrast subjects.

  The control unit 121 includes, for example, a CPU (central processing unit), and controls the overall operation of the digital camera 100. The control unit 121 includes an operation unit, a ROM (read only memory), a RAM (random access memory), an A (analog) / D (digital) converter, a D / A converter, a communication interface circuit, and the like. The control unit 121 drives each unit of the digital camera 100 by developing and executing a predetermined program stored in the ROM in the RAM, and executes a series of operations such as AF control, imaging processing, image processing, recording processing, etc. Do.

  The electronic flash control circuit 122 performs lighting control of the electronic flash 115 in synchronization with the photographing operation according to the control command of the control unit 121. The auxiliary light source drive circuit 123 performs lighting control of the AF auxiliary light source 116 in synchronization with the focus detection operation according to the control command of the control unit 121. The imaging device drive circuit 124 controls the imaging operation of the imaging device 107, A / D converts the acquired image signal, and transmits it to the control unit 121. The image processing circuit 125 performs processing such as gamma conversion, color interpolation, JPEG (Joint Photographic Experts Group) compression, and the like on the image signal acquired by the imaging device 107 in accordance with a control command of the control unit 121.

  The focus drive circuit 126 drives the focus actuator 114 based on the focus detection result according to the control command of the control unit 121, moves the third lens group 105 in the optical axis direction, and performs focus adjustment. The aperture shutter drive circuit 128 drives the aperture shutter actuator 112 in accordance with a control command from the control unit 121 to control the aperture diameter of the aperture / shutter 102. The zoom drive circuit 129 drives the zoom actuator 111 according to the zoom operation instruction of the photographer according to the control instruction of the control unit 121.

  The display unit 131 includes, for example, a display device such as an LCD (liquid crystal display device), and information on the shooting mode of the camera, the preview image before shooting and the confirmation image after shooting, the focus state display image at the time of focus detection, etc. indicate. The operation unit 132 includes, for example, operation switches such as a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, and a touch panel, and outputs an operation instruction signal to the control unit 121. The recording medium 133 is a recording medium removable from the digital camera 100, such as a semiconductor memory card. Image data and the like obtained by photographing can be recorded on the recording medium 133.

(Configuration of imaging device)
A schematic view of the arrangement of pixels and sub-pixels of the image sensor 107 according to the present embodiment is shown in FIG. The horizontal direction in FIG. 2 is taken as the x direction (horizontal direction), the vertical direction as the y direction (vertical direction), and the direction orthogonal to the x direction and the y direction (direction perpendicular to the paper) as the z direction (optical axis direction). In the example shown in FIG. 2, the pixel (unit pixel) array of the two-dimensional CMOS sensor (image pickup device) is shown in a range of 4 columns × 4 rows, and the sub-pixel array is shown in a range of 8 columns × 4 rows.

For example, in the pixel group 200 of 2 columns × 2 rows, a pixel 200R having the spectral sensitivity of R (red) of the first color at the upper left position has a spectral sensitivity of G (green) of the second color at the upper left. In the upper right and lower left, the pixel 200B having the spectral sensitivity of the third color B (blue) is arranged in the lower right. Furthermore, each pixel (unit pixel) is divided into two (Nx division) in the x direction and one division (Ny division) in the y direction into the first subpixel 201 of the division number 2 (division number N _LF = Nx × Ny). And a plurality of sub-pixels of the second sub-pixel 202 (the first sub-pixel to the _N.sub.LF sub-pixel).

In the example shown in FIG. 2, by dividing each pixel of the image sensor 107 into two sub-pixels aligned in the horizontal direction, an image signal (LF data) obtained by one shooting is equal to the division number N _LF A number of viewpoint images and a captured image obtained by combining all viewpoint images can be generated. Note that pixels may be divided in two directions, and the number of divisions in each direction is not limited. Therefore, it can be said that the viewpoint image is an image generated from the signals of some of the plurality of sub-pixels, and the captured image is an image generated from the signals of all the sub-pixels. In this embodiment, as an example, the pixel period P in the horizontal and vertical directions of the image sensor 107 is 6 μm, the number of horizontal pixels N _H = 6000, and the number of vertical pixels N _V = 4000. Therefore, the total number of pixels N = N _H × N _V = 24 million. Further, assuming that the horizontal direction period P _S of the sub-pixels is 3 μm, the total number of sub-pixels N _S = N _H × (P / P _S ) × N _V = 48 million.

  The top view at the time of seeing one pixel 200G of the image pick-up element 107 shown in FIG. 2 from the light-receiving surface side (+ z side) of the image pick-up element 107 is shown to FIG. 3 (A). The z-axis is set in a direction perpendicular to the sheet of FIG. 3A, and the near side is defined as the positive direction of the z-axis. In addition, the y-axis is set in the vertical direction orthogonal to the z-axis, with the upper side being the positive direction of the y-axis, and the x-axis set in the horizontal direction orthogonal to the z-axis and y-axis Define as FIG. 3 (B) shows a cross-sectional view as viewed from the −y side along the a-a cutting line in FIG. 3 (A).

As shown in FIGS. 3A and 3B, in the pixel 200G, a micro lens 305 is formed on the light receiving surface side (+ z direction) of each pixel, and the incident light is collected by the micro lens 305. Be done. Furthermore, a plurality of photoelectric conversion units, ie, a first photoelectric conversion unit 301 and a second photoelectric conversion unit 302, each of which is divided into two in the x (horizontal) direction and divided into one in the y (vertical) direction, are formed. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 correspond to the first subpixel 201 and the second subpixel 202 in FIG. 2, respectively. More generally, the photoelectric conversion units of each pixel are divided into Nx in the x direction and Ny in the y direction, and the number of divisions of the photoelectric conversion units N _LF = Nx × Ny. _{The LF} photoelectric conversion unit corresponds to the first to Nth _LF sub-pixels.

  The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are two independent pn junction photodiodes, and are composed of a p-type well layer 300 and an n-type layer 301 and an n-type layer 302 divided into two. Ru. If necessary, the intrinsic layer may be sandwiched and formed as a pin structure photodiode. A color filter 306 is formed between the microlens 305 and the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 in each pixel. If necessary, the spectral transmittance of the color filter 306 may be changed for each pixel, each photoelectric conversion unit, or the like, or the color filter may be omitted.

  The light incident on the pixel 200G is collected by the micro lens 305, and after being dispersed by the color filter 306, the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 respectively receive light. In the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302, electrons and holes (hole) are generated in a pair according to the amount of light received, and after being separated by the depletion layer, the electrons are accumulated. On the other hand, holes are discharged to the outside of the imaging element 107 through a p-type well layer connected to a constant voltage source (not shown). The electrons stored in the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 are transferred to the electrostatic capacitance unit (FD) through the transfer gate and converted into a voltage signal.

  FIG. 4 schematically shows the correspondence between the pixel structure and the pupil division. FIG. 4 is a cross-sectional view of the section taken along line aa of the pixel structure shown in FIG. 3A as viewed from the + y direction, and the exit pupil plane of the imaging optical system in the −z direction. Shows a view from a perspective. In FIG. 4, in order to correspond to the coordinate axes of the exit pupil plane, the x-axis and the y-axis are shown inverted in the cross-sectional view from the state shown in FIG.

  The imaging element 107 is disposed in the vicinity of the imaging surface of the imaging lens (imaging optical system), and the light flux from the subject passes through the exit pupil 400 of the imaging optical system and enters each pixel. Note that the surface on which the imaging element 107 is disposed is taken as an imaging surface.

The first pupil partial region 501 and the second pupil partial region 502 are approximately optically conjugate by the light receiving surfaces of the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 and the microlens. That is, the first pupil partial region 501 and the second pupil partial region 502 are pupil regions which can be received by the first subpixel 201 and the second subpixel 202, respectively. As described above, the first to N-th N _LF pupil partial regions obtained by dividing the pupil region by N ×× Ny become pupil regions that can receive light at corresponding ones of the first to N-th N _L subpixels. The first pupil partial region 501 to which the first subpixel 201 corresponds is decentered on the pupil plane to the + X side, and the second pupil partial region 502 to which the second subpixel 202 corresponds is-on the pupil surface The center of gravity is eccentric to the X side.

In addition, the pupil region 500 is approximately optically conjugated by the micro-lens and the light-receiving surface in which the light-receiving surfaces of all the divided photoelectric conversion units (here, the first and second photoelectric conversion units 301 and 302) are combined. It is. That is, the divided first to N th _LF photoelectric conversion units can receive the pupil region 500 as a whole. Therefore, by combining the signals obtained by the first to N-th N _LF photoelectric conversion units, it is possible to obtain a pixel signal corresponding to the pupil region 500 in which the entire pixel 200 can receive light. The composite signal corresponds to a pixel signal obtained when the photoelectric conversion unit of the pixel 200 is not divided.

  FIG. 5 is a view schematically showing a light intensity distribution when light is incident on a microlens formed in each pixel. FIG. 5A shows the light intensity distribution in a cross section parallel to the optical axis of the microlens. Further, FIG. 5B shows the light intensity distribution in a cross section perpendicular to the optical axis of the microlens at the focal position of the microlens. In FIG. 5, H indicates the surface on the convex side of the micro lens 305, and f indicates the focal length of the micro lens. Further, nFΔ represents the movable range of the focal position by refocusing, which will be described later, and φ represents the maximum angle of the incident light flux. Incident light is collected by the micro lens 305 at a focal position. However, due to the influence of diffraction due to the wave nature of light, the diameter of the focused spot can not be made smaller than the diffraction limit Δ, and has a finite size. The light receiving surface size of the photoelectric conversion unit is about 1 to 2 μm, whereas the focusing spot of the microlens is about 1 μm. Therefore, the first pupil partial region 501 and the second pupil partial region 502 shown in FIG. 4 that are in a conjugate relationship with the light receiving surface of the photoelectric conversion unit via the micro lens are clearly pupil-divided due to diffraction blur. The light receiving rate distribution (pupil intensity distribution) depends on the incident angle of light.

  FIG. 6 shows an example of a light receiving rate distribution (pupil intensity distribution) depending on the incident angle of light. The horizontal axis represents pupil coordinates, and the vertical axis represents light reception rate. A graph line L1 indicated by a solid line in FIG. 6 represents a pupil intensity distribution along the X axis of the first pupil partial region 501 of FIG. The light receiving rate indicated by the graph line L1 sharply rises from the left end and reaches a peak, then gradually decreases and then the change rate becomes gentle and reaches the right end. A graph line L2 indicated by a broken line in FIG. 6 represents a pupil intensity distribution along the X axis of the second pupil partial region 502. The light receiving rate indicated by the graph line L2 sharply rises from the right end and reaches a peak, and then gradually decreases, and then gradually decreases and reaches the left end, contrary to the graph line L1. As shown, it can be seen that the pupil division is performed gently.

  Next, with reference to FIG. 7, the correspondence between the image sensor 107 and the pupil division will be described. The first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 (the first subpixel 201 and the second subpixel 202) pass through the first pupil partial region 501 and the second pupil partial region 502 of the imaging optical system, respectively. Receive the luminous flux. The image signal obtained by the first sub-pixel group and the image signal obtained by the second sub-pixel group are images having different viewpoints. The LF data is data that can generate an image obtained by each sub-pixel group, and includes information of spatial distribution of light intensity and angular distribution. The LF data may be a set of images obtained by each pixel group, or may be data capable of calculating an image obtained by each pixel group by calculation. For example, assuming that an image obtained by the first sub-pixel group is an A image and an image obtained by the second sub-pixel group is an B image, the LF data may be data of the A image and the B image. It may be data of a composite image (denoted as A + B image) with the B image and data of the A image or B image.

  That is, for LF data, a captured image having a resolution of the total number of pixels N is generated by combining the signals of all the sub-pixels (here, the first sub-pixel 201 and the second sub-pixel 202) for each pixel. be able to.

  Further, by extracting a signal obtained in a specific sub-pixel for each pixel of the LF data, it is possible to generate an image (viewpoint image) corresponding to a pupil partial region corresponding to the sub-pixel. For example, by extracting the image signal obtained by the first subpixel 201 for each pixel, a first viewpoint image having a resolution of N pixels corresponding to the first pupil partial region 501 of the imaging optical system is generated. it can. The same applies to the other subpixels.

  As described above, the imaging element 107 according to the present embodiment has a structure in which a plurality of pixels provided with a plurality of sub-pixels for receiving light beams passing through different pupil partial regions of the imaging optical system are arranged. You can get In the above description, the pupil area is divided into two in the horizontal direction, but may be divided in the vertical direction instead of or in addition to the horizontal direction. In addition, the number of divisions may be three or more in at least one of the horizontal direction and the vertical direction.

(Relationship between defocus amount and image shift amount)
Furthermore, referring to FIG. 8, the defocus amount and image shift of the first viewpoint image and the second viewpoint image (first to Nth _LF viewpoint images) that can be generated based on the LF data acquired by the imaging element 107. The relationship with the quantity will be described.

  FIG. 8 schematically shows the relationship between the defocus amount of the first viewpoint image and the second viewpoint image, and the image shift amount between the first viewpoint image and the second viewpoint image. An imaging element (not shown) is disposed on the imaging surface 600, and the exit pupil of the imaging optical system is set in the first pupil partial region 501 and the second pupil partial region 502 in the same manner as in FIGS. × 1 divided.

  The defocus amount d represents the distance from the imaging position of the object image to the imaging surface 600 as the size | d |. A negative sign (d <0) is defined in the front pin state where the imaging position of the subject image is closer to the object than the imaging plane 600, and a positive sign (d> 0) is defined in the opposite rear pin state. Do. In the in-focus state where the imaging position of the subject image is on the imaging plane (that is, the in-focus position), d = 0. For example, the position of the subject 801 indicates the position corresponding to the in-focus state (d = 0), and the position of the subject 802 indicates the position corresponding to the front pin state (d <0). Hereinafter, the front pin state (d <0) and the rear pin state (d> 0) are collectively referred to as a defocus state (| d |> 0).

  In the front pin state (d <0), the light flux that has passed through the first pupil partial area 501 (or the second pupil partial area 502) among the light flux received from the object 802 is once condensed and then the barycentric position of the light flux It spreads in width Γ1 (or Γ2) centering on G1 (or G2). In this case, a blurred image is obtained on the imaging surface 600. The blurred image is received by the first sub-pixel 201 (or the second sub-pixel 202) constituting each pixel unit arranged in the image sensor 107, and a first viewpoint image (or a second viewpoint image) is generated. Therefore, the first viewpoint image (or the second viewpoint image) is stored as image data of an object image (blurred image) having a width Γ1 (or Γ2) at the gravity center position G1 (or G2) on the imaging surface 600 Is stored in The width Γ1 (or Γ2) of the subject image generally increases in proportion to the increase of the magnitude | d | of the defocus amount d. Similarly, when the image shift amount of the subject image between the first viewpoint image and the second viewpoint image is denoted as "p", the size | p | is an increase in the size | d | of the defocus amount d. It increases with it. For example, as shown in FIG. 8, the image shift amount p can be defined as the difference “G1−G2” of the barycentric positions of light beams, and the size | p | thereof increases with | d | Generally increases in proportion. In the back focus state (d> 0), although the image shift direction of the subject image between the first viewpoint image and the second viewpoint image is opposite to that in the front focus state, | p | It tends to be proportional to the quantity | d |.

  Therefore, in the present embodiment, as the defocus amount of the captured image obtained by adding the first viewpoint image and the second viewpoint image or the first viewpoint image and the second viewpoint image increases or decreases, the first viewpoint image and the first viewpoint image The magnitude of the image shift amount between the two viewpoint images increases.

(Correction processing of viewpoint image based on captured image)
In the following embodiments, image processing (correction processing) such as flaw correction and shading correction is performed on the first viewpoint image and the second viewpoint image (first to Nth _LF viewpoint images) based on the captured image, and the output image is displayed An example of generation will be described.

  First, with reference to FIG. 9 and FIG. 10, a series of flows for performing correction processing and generating an output image based on the acquired LF data will be described. The processing shown in FIGS. 9 and 10 is executed by the control unit 121 loading and executing a program stored in the ROM into the RAM. The image sensor 107 and the image processing circuit 125 operate in accordance with an instruction from the control unit 121.

(Captured image and viewpoint image)
First, as pre-processing of S1 shown in FIG. 9, the control unit 121 as an image acquisition unit combines a captured image obtained by combining different pupil partial regions of the imaging optical system from the LF data acquired by the imaging element 107. At least one viewpoint image corresponding to a pupil partial region of the image optical system is generated. The control unit 121 may acquire the generated captured image and the viewpoint image by reading the storage image from the storage medium instead of generating the captured image and the viewpoint image from the LF data. The preprocessing is referred to as S0 for convenience. The pre-processing S0 will be described below.

  The control unit 121 acquires LF data. The LF data may be acquired using the imaging device 107, or may be acquired by reading out LF data stored in advance from, for example, a recording medium.

Then, the control unit 121 generates a captured image by combining a plurality of viewpoint images constituting the LF data. Each pixel signal constituting the LF data is composed of sub-pixel signals of the total number N _{LF in} the horizontal direction Nx and the vertical direction Ny. Here, of the total number N _LF of sub-pixel signals constituting one pixel signal, the i _S (1 ≦ i _s ≦ N x) th in the horizontal direction and the j _s (1 ≦ j _s ≦ Ny) in the vertical direction _{Let the} signal of the sub-pixel be the k-th sub-pixel signal (k = Nx (j _s −1) + i _S ). Here, 1 ≦ k ≦ N _LF , and k is a serial number of the photoelectric conversion area (sub-pixel) of the total number N _LF divided in the horizontal direction Nx and the vertical direction Ny. The captured image is an image corresponding to the pupil region 500 in FIG. 4, and is obtained by combining sub-pixel signals of respective pixels. The j th vertical pixel and the i th horizontal pixel I (j, i), 1 ≦ j ≦ N _V , 1 ≦ i ≦ N _H constituting the captured image I can be obtained according to equation (1) it can.

  Note that this process may be performed when generating LF data. In this case, in order to maintain the S / N of the captured image I favorably, the expression of the equation in the electrostatic capacitance unit (FD) in the image sensor 107 before analog / digital conversion (A / D conversion) of each sub-pixel signal The sub-pixel signals may be combined according to (1). When converting the charge accumulated in the FD in the imaging element 107 into a voltage signal, the sub-pixel signals may be combined according to the equation (1), or after the A / D conversion is performed, the equation (1 The sub-pixel signals may be combined in accordance with.

The present embodiment is a case of _{Nx = 2, Ny = 1,} N LF = 2 a is the x-direction divided into two, using the input image corresponding to the illustrated pixel array in FIG. 2 (LF data). Therefore, the control unit 121 generates a captured image in which all the signals of the first sub-pixel 201 and the second sub-pixel 202 divided into two in the x-direction are combined for each pixel. This captured image is a Bayer-arranged RGB signal having a resolution of the number of pixels N (= the number of horizontal pixels N _H × the number of vertical pixels N _V ).

  In addition, in the correction process of the viewpoint image to be described later, since the captured image is used as a reference image of correction standard, the shading correction process and the defect correction process for each RGB are also performed on the captured image I.

Next, the control unit 121 sets the pixel I _k (j, i) that constitutes the i-th in the horizontal direction and the j-th k-th viewpoint image I _{k in} the vertical direction corresponding to the k-th pupil partial region of the imaging optical system. , According to equation (2).

In the present embodiment, (2 divides the pupil area in the horizontal direction) photoelectric conversion unit 2 divides the x direction _{Nx = 2, Ny = 1,} N LF = LF data obtained by the image sensor 107 of the second configuration The first viewpoint image (k = 1) is generated among the two viewpoint images that can be generated from. Control unit 121 of the LF data corresponding to the illustrated pixel array in Figure 2, to extract the signal of the first sub-pixel 201, to generate a first viewpoint image I _1. First viewpoint image I ₁ is a viewpoint image corresponding to the first pupil partial area 501 of the image forming optical system. First viewpoint image I ₁ is the RGB signals of the Bayer array having a resolution of pixel number N (= the number of pixels N _V in the horizontal pixel number N _H × vertical _direction). Where necessary, it may generate the second viewpoint image I ₂ corresponding to the second pupil partial area 502 of the image forming optical system.

As described above, in the present embodiment, based on the LF data, one of the captured image I corresponding to the pupil region of the imaging optical system and one of the viewpoint images I _k corresponding to different pupil partial regions of the imaging optical system Generate the above. The generated captured image and viewpoint image data may be recorded on the recording medium 133. Incidentally, by subtracting the first viewpoint image I ₁ from the captured image I, can generate a second viewpoint image I _2. The captured image I corresponds to a captured image obtained by a conventional imaging device in which the photoelectric conversion unit of each pixel is not divided. Therefore, the same image processing as in the prior art can be applied to the captured image I. Further, in order to equalize the process to each viewpoint image, the first viewpoint image I _1, and generates a second viewpoint image I _2, may be recorded on the recording medium 133. In this case, by combining the first viewpoint image I ₁ and the second viewpoint image I _2, it is possible to generate a captured image I.

(Shading correction of the viewpoint image (brightness correction))
Next, shading correction (brightness correction) in S1 shown in FIG. 9 will be described. In the present process, the control unit 121 performs shading correction for each of the RGB with respect to the first viewpoint image I ₁ (k th viewpoint image I _k ) based on the captured image I. In addition, although the case where the control part 121 performs also about subsequent processes is demonstrated to an example, the image processing circuit 125 may perform the one part or all part.

First, shading due to pupil shift of the first viewpoint image and the second viewpoint image (first to Nth _LF viewpoint images) will be described. FIG. 11 shows a first pupil partial region 501 received by the first photoelectric conversion unit 301 at a peripheral image height of the imaging device, a second pupil partial region 502 received by the second photoelectric conversion unit 302, and emission of the imaging optical system. The relationship of the pupil 400 is shown. In FIG. 11, the first photoelectric conversion unit 301 and the second photoelectric conversion unit 302 (the first photoelectric conversion unit to the N _LF photoelectric conversion unit) respectively correspond to the first subpixel 201 and the second subpixel 202 (the first subpixel). Pixel to the _N.sub.LF sub-pixel).

  FIG. 11A shows the case where the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the imaging element 107 are the same. In this case, the exit pupil 400 of the imaging optical system is approximately equally pupil-divided by the first pupil partial region 501 and the second pupil partial region 502. On the other hand, FIG. 11B shows the case where the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the imaging device. In this case, at the peripheral image height of the imaging device 107, pupil deviation occurs between the exit pupil of the imaging optical system and the entrance pupil of the imaging device, and the exit pupil 400 of the imaging optical system is divided into pupils unevenly. In the example of FIG. 11B, the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is smaller (brighter) than the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502. It becomes a value. Conversely, at the image height on the opposite side (not shown), the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502 It is a larger (darker) value.

  Similarly, even when the exit pupil distance D1 of the imaging optical system is longer than the set pupil distance Ds of the imaging element shown in FIG. 11C, the exit pupil and imaging of the imaging optical system at the peripheral image height of the imaging element 107 A pupil shift of the entrance pupil of the element occurs. Therefore, the exit pupil 400 of the imaging optical system is divided into non-uniform pupils also in the case shown in FIG. In the example of FIG. 11C, the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is larger than the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502 (dark) It becomes a value. Conversely, at the image height on the opposite side (not shown), the effective aperture value of the first viewpoint image corresponding to the first pupil partial region 501 is the effective aperture value of the second viewpoint image corresponding to the second pupil partial region 502 It becomes smaller (brighter) value.

  Thus, as the pupil division becomes nonuniform, the intensities of the first viewpoint image and the second viewpoint image also become nonuniform (that is, the light amount distribution is biased), and either of the first viewpoint image and the second viewpoint image Shading occurs where one intensity is high and the other intensity is low. And, this shading occurs for each pixel, that is, for each RGB.

Therefore, in order to generate a viewpoint image of good image quality and quality by uniformly bringing the above-mentioned uneven signal intensity distribution due to shading into consideration, shading correction for each RGB is performed. That is, in the present embodiment, shading correction for each RGB is performed on the first viewpoint image I ₁ (k th viewpoint image I _k ) using the corrected captured image I (j, i) as a reference image.

More specifically, the control unit 121 in S1 of FIG. 9, first, detects the effective pixels from the captured image I and the first viewpoint image _{I 1} (S1-1). Effective pixels, in the captured image I, also in the first viewpoint image I ₁ which is a pixel of the non-saturation and non-defective (non-scratch). Specifically, the control unit 121 is an effective pixel for each of the corresponding pixels (pixel I (j, i) and pixel I ₁ (j, i)) of the captured image I and the first viewpoint image I _1. It is determined whether there are any pixels or at least one of them is not a valid pixel. Then, the control unit 121 sets V ₁ (j, i) = 1 if all are determined to be valid pixels, and V ₁ (j, i) = 0 if it is determined that at least one is not a valid pixel. V ₁ can be regarded as a binary image having a value of 1 for valid pixels and 0 for invalid pixels. Alternatively, V ₁ can also be regarded as an image indicating a defective pixel position as 0. For example, the control unit 121 can determine that the pixel is a saturated pixel when the pixel value exceeds the saturation determination threshold IS and a non-saturated pixel when the pixel value does not exceed the saturation determination threshold IS. However, the saturation determination threshold IS _k used for the viewpoint image is less than or equal to the saturation determination threshold IS used for the captured image. Further, the control unit 121 can determine that a pixel registered as a defective pixel in the imaging element 107 is a defective pixel, and a pixel not registered is a non-defective pixel. Note that these determination methods are merely examples, and it can be determined whether a certain pixel is a saturated pixel or a defective pixel by any known method. The control unit 121 detects an effective pixel in the same manner when performing shading correction on the k-th viewpoint image I _k (k ≧ 2).

  In the image sensor 107 of the present embodiment, when any of the plurality of photoelectric conversion units included in each pixel is saturated, the saturation charge leaks to other photoelectric conversion units (sub-pixels) in the same pixel instead of outside the pixel. Are configured. Such leakage of saturated charge is called charge crosstalk. When charge crosstalk occurs in the same pixel, both of the saturated sub-pixel (for example, the second sub-pixel 202) and the sub-pixel for which the saturated charge leaks (for example, the first sub-pixel 201) The linear relationship of the amount of charge can not be maintained. Therefore, shading can not be correctly detected for pixels in which charge crosstalk has occurred.

The amount of charge accumulated in the photoelectric conversion portion is relatively large and the charge crosstalk is relatively likely to occur in the case where the imaging sensitivity is high and the case where the imaging sensitivity is low. Therefore, in the present embodiment, in order to improve the detection accuracy of the saturated pixel, the saturation determination threshold IS of the imaging signal at the first imaging sensitivity is set to the imaging signal at the second imaging sensitivity higher than the first imaging sensitivity. Or less than the saturation determination threshold IS. The same applies to the saturation determination threshold IS _k used for the viewpoint image.

Furthermore, when the exit pupil distance of the imaging optical system is shorter than the first predetermined pupil distance (or longer than the second predetermined pupil distance (> first predetermined pupil distance)), the exit pupil of the imaging optical system And shading due to pupil shift of the entrance pupil of the imaging device. In this case, in the peripheral pixels where the image height is large, the intensity of one of the first viewpoint image and the second viewpoint image is increased and the intensity of the other is decreased, so that charge crosstalk easily occurs. Therefore, in the present embodiment, in order to improve the detection accuracy of the saturated pixel, the saturation determination threshold IS in the case of occurrence of shading due to pupil displacement is made equal to or less than the saturation determination threshold IS in the case of no shading due to pupil displacement Can. The case where shading due to pupil shift does not occur is the case where the exit pupil distance of the imaging optical system is equal to or greater than the first predetermined pupil distance and equal to or less than the second predetermined pupil distance. The same applies to the saturation determination threshold IS _k used for the viewpoint image.

Here, the captured image I of the Bayer array shown in FIG. 2 can be defined for each color (R, Gr, Gb, B) of the Bayer pattern.
That is, RI is a captured image of R
Composed of pixels _{RI (2j 2 -1,2i 2 -1)} = I (2j 2 -1,2i 2 -1).
The Gr captured image GrI is
It consists of pixels GrI (2j ₂ -1, 2i ₂ ) = I (2j ₂ -1, 2i ₂ ).
Gb captured image GrI
Composed of pixels _{GbI (2j 2, 2i 2 -1} ) = I (2j 2, 2i 2 -1).
The captured image BI of B is
It consists of pixels BI (2j ₂ , 2i ₂ ) = I (2j ₂ , 2i ₂ ).
At this time, the integer j ₂ and the integer i ₂ are 1 ≦ j ₂ ≦ N _V / 2 and 1 ≦ i ₂ ≦ N _H / 2, respectively.

Similarly, the k-th viewpoint image I _k of the Bayer array illustrated in FIG. 2 is handled for each of R, Gr, Gb, and B.
That is, the k-th viewpoint image RI _k of R is
Composed of pixels _{RI k (2j 2 -1,2i 2 -1} ) = I k (2j 2 -1,2i 2 -1).
Gr's kth viewpoint image GrI _k
It comprises pixels GrI _k (2j ₂ -1, 2i ₂ ) = I _k (2j ₂ -1, 2i ₂ ).
Gb kth viewpoint image GbI _k is
It consists of pixels GbI _k (2j ₂ , 2i ₂ -1) = I _k (2j ₂ , 2i ₂ -1).
The k th viewpoint image BI _k of B is
It consists of pixels BI _k (2j ₂ , 2i ₂ ) = I _k (2j ₂ , 2i ₂ ).
The integer j ₂ and the integer i ₂ are respectively 1 ≦ j ₂ ≦ N _V / 2 and 1 ≦ i ₂ ≦ N _H / 2.

[Projection processing]
Next, in S1 shown in FIG. 9, the control unit 121 as a projection unit performs projection processing based on the captured image RI, GrI, GbI, and BI (S1-2A). Specifically, each pixel of the captured image RI, GrI, GbI, and BI is projected in the projection direction according to the equations (3A) to (3D), and a plurality of signals are arranged in the direction orthogonal to the projection direction 1 Generate a dimensional projection signal. Here, the projection direction is a direction (y direction) orthogonal to the pupil division direction (x direction). As a result, a projection signal related to the change in luminance in the x direction of the image is obtained. The projection direction may be another direction. Thereby, the control unit 121 generates the pixels RP (2i ₂ -1), GrP (2i ₂ ), GbP (2i ₂ -1), and BP (2i ₂ ) of the respective colors constituting the projection signal P of the captured image. .

Note that the saturation signal value and the defect signal value do not include information for correctly detecting the shading for each RGB of the captured image. Therefore, the control unit 121 performs projection processing after excluding the invalid pixels of the captured image I by taking the product of the corresponding pixels of the captured image I and V _k . This projection processing corresponds to the numerator of the upper equation of the equations (3A) to (3D). Then, the control unit 121 normalizes the projection result by the number of effective pixels used in the projection process. This normalization process corresponds to the denominator of the upper equation of equations (3A) to (3D). When the number of effective pixels used for the projection process is zero, the control unit 121 sets the projection signal of the captured image to zero as shown in the lower part of Expressions (3A) to (3D). Furthermore, the control unit 121 sets the projection signal of the captured image to 0 also when the projection signal of the captured image is a negative signal due to the influence of noise or the like.

Similarly, in step S1-2B, the control unit 121 causes the pixels of the k-th viewpoint images RI _k , GrI _k , GbI _k , and BI _k to be orthogonal to the pupil division direction according to equations (3E) to (3H). Project in (y direction). Thus, the control unit 121 causes the pixels RP _k (2i ₂ -1), GrP _k (2i ₂ ), GbP _k (2i ₂ -1), and BP _{k of} the respective colors that constitute the projection signal P _k of the k-th viewpoint image. Generate (2i ₂ ).

The control unit 121 applies, as necessary, low-pass filter processing to smooth the projection signal P of the captured image and the projection signal P _k of the k-th viewpoint image after the above-described projection processing.

  In step S1-3, the control unit 121 determines whether to use the projection signal of the viewpoint image obtained in step S1-2B for calculation of a shading correction value. The control unit 121 obtains an evaluation value for each of a plurality of pixels (that is, each pixel column) arranged in the projection direction for the viewpoint image. Then, in accordance with the comparison result of the evaluation value and the predetermined threshold value, the control unit 121 determines whether the corresponding projection signal is valid or invalid (whether or not to use for calculation of the shading correction value). judge.

  In order to determine the effectiveness of the projection signal in consideration of the signal-to-noise ratio of the signal, an evaluation value in which both the number of effective pixels and the luminance are considered is used. In general, the signal-to-noise ratio of a signal is improved by adding a plurality of signal values. For example, by adding four signal values, it is possible to double the signal-to-noise ratio of the signal. Further, the SN ratio of the image signal is proportional to 1/2 of the luminance. Therefore, by obtaining the evaluation value by calculation including addition of signal values, it is possible to accurately determine the validity / invalidity of the signal of the viewpoint image with low luminance and therefore low SN ratio. By accurately determining the validity of the projection signal, it is possible to calculate a shading correction value using an effective (reliable) projection signal.

The validity determination of the projection signal of the viewpoint image is performed for each color component (R, Gr, Gb, B). For example, the validity of the red pixel RP _k (2i ₂ -1) that constitutes the projection signal P _k of the k-th viewpoint image is determined based on the evaluation value obtained from the k-th viewpoint image RI _k .

When the captured image I and the viewpoint image I _k are compared, the luminance of the viewpoint image I _k is smaller. This is because the size of the light receiving surface of the photoelectric conversion unit used to generate the viewpoint image I _k is smaller than the size of the light receiving surface of the photoelectric conversion unit used to generate the captured image I. Therefore, in the present embodiment, the validity of the projection signal of the viewpoint image whose luminance is lower than that of the captured image, and therefore the SN ratio is low is determined. Further, in order to accurately perform threshold determination on a viewpoint image whose luminance is lower than that of a captured image, an evaluation value to which signal values are added is generated and compared with the threshold.

  The determination process is applied to each color component (R, Gr, Gb, B) based on the same equation. In the following, processing relating to the R component is representatively described using the flowchart shown in FIG. 12, but the same applies to other color components.

As described above, the k-th viewpoint image RI _k of R is
Composed of pixels _{RI k (2j 2 -1,2i 2 -1} ) = I k (2j 2 -1,2i 2 -1). The integer j ₂ and the integer i ₂ are respectively 1 ≦ j ₂ ≦ N _V / 2 and 1 ≦ i ₂ ≦ N _H / 2.

Control unit 121 in the threshold determination, first, the equation (3I), obtaining the evaluation value _{E i2} of _{i 2} column (S1301). Here, since the evaluation value E _i2 is generated without using the value of the pixel corresponding to the defective pixel position, it is proportional to the number of effective pixels. Further, since the evaluation value _Ei2 is a squared cumulative value of the pixel value (brightness), it is proportional to the pixel value (brightness). Therefore, the evaluation value E _i2 is an evaluation value in consideration of the number of effective pixels and the luminance. Further, by using such an evaluation value, the validity of the projection signal can be determined in consideration of the SN ratio of the image signal.

Next, the control unit 121, by the formula (3J), compares the evaluation value _{E i2} and vertical evaluation threshold E _limit (S1302). Then, the control unit 121, when the evaluation value _{E i2} is larger than the rated threshold E _limit, determines an effective projection signal RP _(2i 2 -1) obtained in S1-3 (S1303). On the other hand, the control unit 121, when the evaluation value _{E i2} is equal to or less than the rated threshold E _limit, is determined to be invalid the projection signal RP _(2i 2 -1) obtained in S1-3 (S1304). The control unit 121 may set the value to 0 so that the value of the projection signal determined to be invalid is not used.

The control unit 121 determines whether or not the validity determination has been performed for all the pixel rows of the parallax image (S1305), and if it is determined that it has been executed, the processing ends, otherwise the processing returns to S1301. A determination process is performed on an undetermined pixel row. The control unit 121 determines the validity of the projection signal RP _k (2i ₂ −1) by calculating the evaluation value E and comparing the evaluation value E with the threshold value E _limit for each pixel row of the parallax image (i To change the value of). The control unit 121 applies the determination process to other color components GrP _k (2i ₂ ), GbP _k (2i ₂ -1), and BP _k (2i ₂ ) in the same manner.

  Equation (3I) is an example for calculating the evaluation value E, and another arbitrary evaluation value calculated in consideration of the number of pixels in the vertical direction can be used for the viewpoint image. In addition, since fixed pattern noise is not considered, the evaluation value E obtained by Formula (3I) can perform more exact determination by setting it as the evaluation value which considered fixed pattern noise. Specifically, since fixed pattern noise does not depend on luminance, it is possible to obtain an evaluation value E in consideration of fixed pattern noise by using equation (3K) obtained by adding the term of fixed value Offset to equation (3I). it can.

In addition, the number of pixels of the captured image I and the parallax image I _k is reduced by addition and thinning before performing effective pixel detection processing in S1-1 for the purpose of reducing the image processing load and improving the SN ratio. Sometimes. In particular, when the pixels are added, the signal-to-noise ratio of the signal is improved as described above, so the validity of the projection signal is accurately determined by comparing the evaluation value with the threshold considering the addition. be able to. For example, it is assumed that the number of pixels in the vertical or vertical direction is reduced to 1 / N (N is an integer of 2 or more) by addition processing before the effective pixel detection processing in S1-1. In this case, the SN ratio of the image signal in the addition direction is improved by N ^1/2 times by the addition process. Therefore, the evaluation value is compared with the corrected threshold value E _limit / (N ^1/2 ). When the pixels are thinned out, there is no change in the signal-to-noise ratio of the signal, so that the correction of the threshold E is unnecessary.

  In the present embodiment, the image is projected in pixel rows (vertical direction) in projection processing to obtain a horizontal one-dimensional image (projection signal), so the threshold E is a threshold in the vertical direction of the image. On the other hand, the threshold can also be set in the horizontal direction of the image. For example, a threshold can be set for the proportion of projection signals determined to be valid in the process shown in FIG. Since the total number of projection signals is equal to the number of pixels in the horizontal direction, the threshold for the proportion of projection signals determined to be valid is the threshold in the horizontal direction. For example, when the ratio or the number of projection signals determined to be effective is less than or equal to the threshold as a result of the effectiveness determination process in color component units, the control unit 121 projects the viewpoint image including the projection signals determined to be effective. Determine all signals as invalid. In this case, a sufficient number of data having high reliability can not be obtained as data for generating a shading function in S1-4 described later. Therefore, the control unit 121 does not perform generation of the shading function and shading correction using the shading function. That is, the threshold in the horizontal direction is a determination threshold whether to perform shading correction.

  The effective pixel detection in S1-1 is based on the presence or absence of saturation or a flaw, and the SN ratio of the signal is not considered. Specifically, even if the output value of the pixel deviates from the true value due to the low SN ratio, the validity is determined based on the output value. On the other hand, since the judgment of the validity of the projection signal in S1-3 is a judgment as to whether or not the projection signal is appropriate for use in the calculation of the shading correction value, allowing a deviation from the true value due to a low SN ratio. If so, the accuracy of shading correction may be reduced. By excluding the projection signal which is deviated from the true value due to the low SN ratio, accurate shading correction can be realized.

FIGS. 13A and 13B show specific examples of projection signals, in which the horizontal axis is the pixel position in the horizontal direction, and the vertical axis is the signal intensity. The FIG. 13A shows an example of the values of the green (G) component, the red (R), and the blue (B) component that constitute the projection signal P of the captured image. In addition, since G component shows Gr component and Gb component collectively, a value exists in each horizontal pixel position. On the other hand, although the R and B components have values at every other pixel position, they are continuously described in the drawing for the sake of convenience. Further, FIG. 13B is an example of values of green (G) component, red (R) component, and blue (B) component constituting the projection signal P _k of the first viewpoint image as in FIG. 13 (A). Is shown. The signal strength of each projection signal has a value corresponding to the color of the subject. Therefore, in order to perform shading correction of the k-th viewpoint image I _k correctly, the amount of deviation between the RGB components of the k-th viewpoint image I _k caused by the pupil deviation (shading component), RGB due to the color of the subject It is necessary to separate the ingredients.

  Before performing S1-4 in FIG. 9, the control unit 121 determines that the ratio or number of projection signals exceeding the threshold in the horizontal direction is effective for all color components by the effectiveness determination process in S1-3. Determine if there is. Then, when there is even one color component for which the ratio or number of projection signals exceeding the threshold in the horizontal direction is not determined to be valid, the processing relating to the shading correction after S1-4 is not performed.

  On the other hand, if it is determined that the ratio or the number of projection signals exceeding the threshold in the horizontal direction is valid for all color components, the control unit 121 executes the processing of S1-4 and thereafter. Although the following processing will be described on the assumption that all projection signals are determined to be valid, the processing is actually skipped for projection signals determined to be invalid.

In S1-4, the control unit 121 performs the shading signal RS _k (2i ₂ −1) for each RGB of the k-th viewpoint image I _k relative to the captured image I according to Equations (4A) to (4D) , GrS _k (2i ₂ ), GbS _k (2i ₂ −1), and BS _k (2i ₂ ). The amount of light received by the pixel in the imaging device needs to be larger than the amount of light received by the sub-pixel obtained by dividing the pixel, and the amount of light received by the sub-pixel needs to be larger than 0 for calculation of the shading component. Therefore, when the control unit 121 satisfies the conditional expression RP (2i ₂ -1)> RP _k (2i ₂ -1)> 0, the projection signal RP of the R component RI _k of the k-th viewpoint image is satisfied according to equation (4A). The ratio (that is, the ratio) of _k (2i ₂ -1) and the projection signal RP (2i ₂ -1) of the R component RI of the captured image is calculated. Then, the control unit 121 multiplies the calculated ratio by the pupil division number N _LF for normalization to generate a shading signal RS _k (2i ₂ −1) of the R component of the k-th viewpoint image I _k . As a result, it is possible to offset the R signal component caused by the subject and to separate the R shading component of the k-th viewpoint image I _k . On the other hand, when the control unit 121 does not satisfy the conditional expression RP (2i ₂ −1)> RP _k (2i ₂ −1)> 0, the shading signal RS _k (2i ₂ −) of the R component of the k th viewpoint image I _k Set 1) to 0.

Similarly, shading signals for other color components are calculated. With regard to the Gr component of the k-th viewpoint image I _k , when the conditional expression GrP (2i ₂ )> GrP _k (2i ₂ )> 0 is satisfied, the projection signal GrP _k of the Gr component of the k-th viewpoint image and 2i _2), calculates the ratio of the projection-back signals GrP (2i ₂₎ of Gr component of the captured image. Then, the control unit 121 multiplies the calculated ratio by the number of pupil divisions N _LF for normalization to generate a shading signal GrS _k (2i ₂ ) of the Gr component of the k-th viewpoint image I _k . As a result, it is possible to offset the Gr signal component caused by the subject and to separate the shading component of the Gr component of the k-th viewpoint image I _k . The control unit 121 sets the shading signal GrS _k (2i ₂ ) of the Gr component of the k-th viewpoint image I _k to 0 when the conditional expression GrP (2i ₂ )> GrP _k (2i ₂ )> 0 is not satisfied. Do.

Similarly, equation (4C) and in accordance with (4D), the shading signals _BS k of the B component of the shading signals _GbS k _(2i 2 -1) and a k-viewpoint image _{I k} of Gb component of the k viewpoint image _{I k} (2i ₂ ) Generate.

Note that, in order to perform the shading correction accuracy with high accuracy, the shading correction may be performed only when there are sufficient effective shading signals. For example, predetermined number of effective shading signals satisfying RS _k (2i ₂ -1)> 0, GrS _k (2i ₂ )> 0, GbS _k (2i ₂ -1)> 0, BS _k (2i ₂ )> 0 In the above case, shading correction may be performed.

Further, the control unit 121 controls the shading functions RSF _k (2i ₂ −1), GrSF _k (2i ₂ ), and GbSF _k (2i) of each of the k-th viewpoint images I _k according to Equations (5A) to (5D). ₂ -1) and BSF _k (2i ₂ ) are calculated. These shading functions are, for example, polynomial functions in a smooth _NSF dimension with respect to position variables in the pupil division direction (x direction). In addition, effective shading signals that satisfy RS _k (2i ₂ −1)> 0, GrS _k (2i ₂ )> 0, GbS _k (2i ₂ −1)> 0, and BS _k (2i ₂ )> 0 described above Be the data points to be input to these shading functions.

The control unit 121 performs parameter fitting by, for example, the least squares method using these data points, and calculates the coefficients RSC _k (μ), GrSC _k (μ), GbSC of each shading function of Equations (5A) to (5D). Calculate _k (μ) and BSC _k (μ). In this manner, the shading functions RSF _k (2i ₂ −1), GrSF _k (2i ₂ ), and GbSF _k (2i ₂ −1) for each RGB of the k-th viewpoint image I _k relative to the captured image , BSF _k (2i ₂ ) can be generated. Since the shading signal can be adjusted on the basis of the statistical shading occurrence tendency according to the position in the x direction by the statistical processing using the shading function, smooth shading correction can be performed.

The functions _obtained by inverting the shading functions RSF _k , GrSF _k , GbSF _k , and BSF _k in the pupil division direction (x direction) are R [RSF _k ], R [GrSF _k ], R [GbSF _k ], R [ BSF _k ]. Assuming that the predetermined allowable value is ε (0 <ε <1), 1−ε ≦ RSF _k + R [RSF _k ] ≦ 1 + ε, 1−ε ≦ GrSF _k + R [GrSF _k ] ≦ 1 + ε, 1−ε ≦ GbSF _k + R [ If each conditional expression of GbSF _k ] ≦ 1 + ε and 1−ε ≦ BSF _k + R [BSF _k ] ≦ 1 + ε is satisfied at each position, it is judged that the detected shading function is appropriate, and the expression (6A Through equation (6D). In other cases, it is determined that the detected shading function is inappropriate, RSF _k ≡ 1, GrSF _k ≡ 1, GbSF _{k 、} 1, BSF _k例外 1, and exception processing is performed as necessary.

13C, the shading functions RSF ₁ (R), GrSF ₁ (G), GbSF ₁ (G), and BSF ₁ (for each RGB) of the first viewpoint image I ₁ relative to the captured image I are shown. An example of B) is shown. The projection signal of the first viewpoint image shown in FIG. 13 (B) and the projection signal of the captured image shown in FIG. 13 (A) include a component caused by the color of the subject and a shading component. There is. In this embodiment, the color component of the subject is canceled by obtaining the ratio of the projection signal of the first viewpoint image and the projection signal of the captured image, and a shading function is generated based on the shading signal separated from the projection signal. Therefore, the first viewpoint image I _1, it is possible to generate a good accuracy, a smooth shading function for each RGB component.

  In the above description, the shading function is generated using a polynomial function. However, the present invention is not limited to this, and other functions may be used to generate a shading function according to the characteristics of shading.

In S1-5, the control unit 121 applies shading correction (brightness correction) to the k-th viewpoint image I _k (j, i) according to Expressions (6A) to (6D). Specifically, the control unit 121 applies shading correction processing to each of the color components of the k-th viewpoint image I _k (j, i) using a corresponding shading function. Hereinafter, the k-th viewpoint image I _k after the shading correction is referred to as a k-th viewpoint (first correction) image M ₁ I _k .

Here, the k-th viewpoint (first correction) image M ₁ I _k of the Bayer array is defined for each of R, Gr, Gb, and B. That is, R k th viewpoint (first correction) image is RM ₁ I _k ( ₂ j ₂ -1, ₂ i ₂ -1) = M ₁ I _k ( ₂ j ₂ -1, ₂ i ₂ -1), Gr k Let the viewpoint (first correction) image be GrM ₁ I _k (2j ₂ -1, 2i ₂ ) = M ₁ I _k (2j ₂ -1, 2i ₂ ). In addition, the Gb k-th viewpoint (first correction) image is represented by GbM ₁ I _k (2j ₂ , 2i ₂ -1) = M ₁ I _k (2j ₂ , 2i ₂ -1), and the B-th k-th viewpoint (first Correction) Let the image be BM ₁ I _k (2j ₂ , 2i ₂ ) = M ₁ I _k (2j ₂ , 2i ₂ ). As necessary, the k-th viewpoint (first correction) image M ₁ I _k (j, i) after the shading correction may be used as an output image.

  As described above, in the present embodiment, by using the projection signals of the captured image and the viewpoint image to offset the signal change due to the color component of the subject, the shading function for each RGB Relationship with). At this time, since the reliability of the projection signal of the viewpoint image is determined and the shading function is generated using the highly reliable projection signal, the shading function with high accuracy can be obtained. The control unit 121 as a correction amount acquisition unit generates the reciprocal of the shading function value as a shading correction amount (brightness correction amount). Then, the control unit 121 as a correction unit performs a shading correction process of correcting the brightness of the viewpoint image based on the brightness correction amount. Therefore, by improving the accuracy of the shading function, the accuracy of the shading correction also improves.

Here, with reference to FIGS. 14A to 14C, the effect of the shading correction process for each of the first viewpoint image I ₁ (j, i) shown in S1 of FIG. 9 will be described. First, FIG. 14A shows an example of a captured image I (after demosaicing) of the present embodiment, and shows an example of a captured image with high image quality. Further, FIG. 14B shows an example of the first viewpoint image I ₁ (after demosaicing) before the shading correction of the present embodiment. That is, shading occurs for each of RGB due to the pupil shift of the exit pupil of the imaging optical system and the entrance pupil of the imaging device described above, and the first viewpoint image I ₁ (j, i) shown in FIG. A decrease in luminance (and modulation of the RGB ratio) occurs at the image height. Further, FIG. 14C shows an example of a first viewpoint (first correction) image M ₁ I ₁ (after demosaicing) after shading correction according to the present embodiment. That is, the reduction in luminance (and modulation of the RGB ratio) is corrected by the above-described shading correction for each RGB, and the first viewpoint (first correction) image M after shading correction with good image quality and quality is the same as in the captured image. ₁ I ₁ (j, i) is generated.

(Defect correction of viewpoint image)
Next, defect correction processing for a viewpoint image performed in S2 of FIG. 9 will be described. In the present embodiment, the control unit 121 performs defect correction of the kth viewpoint (first correction) image M ₁ I _k after shading correction based on the captured image I. In the present embodiment, the case of k = 1 will be described as an example.

Although the captured image I is normal due to a short circuit or the like of the transfer gate due to the circuit configuration of the imaging device and the driving method, a defect in a very small part of the k th viewpoint image I _k (first viewpoint image I ₁ ) May occur, resulting in point defects and line defects. In such a case, if necessary, point defect information and line defect information inspected in a mass production process or the like may be recorded in advance in, for example, the image processing circuit 125 or the like, and the recorded point defect information or line defect information may be used. Thus, defect correction processing of the k-th viewpoint image I _k (first viewpoint image I ₁ ) can be performed. In addition, point defect determination and line defect determination can be performed by inspecting the k-th viewpoint image I _k (first viewpoint image I ₁ ) in real time as needed.

Hereinafter, in the example of the defect correction process in S2, the k-th viewpoint image I _k has a linear line defect in the horizontal direction (x direction) in the odd-numbered row 2j _D −1 or the even-numbered row 2j _D. The case where there is no line defect in the odd-numbered row or even-numbered row in the captured image I will be described as an example.

In S2-1, the control unit 121 performs defect correction of the k-th viewpoint (first correction) image M ₁ I _k using the normal captured image I as a reference image. That is, the control unit 121 compares the signal value of the kth viewpoint (first corrected) image M ₁ I _{k at} a position not determined to be a defect with the signal value of the captured image I at a position not determined to be a defect. Do. However, when performing this comparison, after removing the influence of the shading component for each RGB of the k-th viewpoint image I _k generated by the pupil shift, the subject is the k-th viewpoint image I _k and the captured image I It is necessary to exactly compare the signal components for each of the RGB that it has. Therefore, using the k-th viewpoint (first correction) image M ₁ I _k generated in S1, the k-th viewpoint image in a shading state equivalent to the captured image I is used. That is, in this step, defect correction with high accuracy is performed on the shading-corrected k th viewpoint (first correction) image M ₁ I _k using the captured image I as a reference image.

The control unit 121 transmits a normal signal of the captured image I to a signal determined as a defect in a very small part of the k-th viewpoint (first correction) image M ₁ I _k (j, i) after the shading correction. The normal signal of the k th viewpoint (first correction) image M ₁ I _k is used. An image generated by the defect correction process is a k-th viewpoint (second correction) image M ₂ I _k (j, i). Here, the k-th viewpoint (second correction) image M ₂ I _k of the Bayer array is defined for each of R, Gr, Gb, and B. That is, R k th viewpoint (second correction) image is RM ₂ I _k ( ₂ j ₂ -1, ₂ i ₂ -1) = M ₂ I _k ( ₂ j ₂ -1, ₂ i ₂ -1), Gr k The viewpoint (second correction) image is set to GrM ₂ I _k (2j ₂ -1, 2i ₂ ) = M ₂ I _k (2j ₂ -1, 2i ₂ ). In addition, the Gb k-th viewpoint (second correction) image is represented by GbM ₂ I _k (2j ₂ , 2i ₂ -1) = M ₂ I _k (2j ₂ , 2i ₂ -1) and the B-th k-th viewpoint (second Correction) Let the image be BM ₂ I _k (2j ₂ , 2i ₂ ) = M ₂ I _k (2j ₂ , 2i ₂ ).

Control unit 121, when the first position of the R of the k-th viewpoint (first modification) image _{M 1 I k (2j D -1,2i} D -1) is determined as a defect, the operation according to the formula (7A) go and generates a k-th viewpoint in the first position (second correction) image _{RM 2 I k (2j D -1,2i} D -1). The calculation of the equation (7A), a captured image _RI of the first position (2j _D -1,2i D -1), the k viewpoint (first modification) of the second position of the R which are not determined as defective image The RM ₁ I _k and the captured image RI at the second position are used.

Also, if it is determined that the first position (2j _D -1, 2i _D ) of Gr of the k-th viewpoint (first correction) image M ₁ I _k is a defect, calculation according to equation (7B) is performed, and the first position The second correction image GrM ₂ I _k (2j _D −1, 2i _D ) is generated after the defect correction in. Furthermore, similarly, with regard to Gb and B of the k-th viewpoint (first correction) image M ₁ I _k , viewpoint images after defect correction are generated according to Equations (7C) and (7D), respectively.

At most positions (j, i) not determined as defects, the k-th viewpoint (second correction) image M ₂ I _k (j, i) is the k-th viewpoint (first correction) image M ₁ I _k ( The same (or substantially the same) signal value as j, i), that is, M ₂ I _k (j, i) = M ₁ I _k (j, i). Note that, as necessary, the k-th viewpoint (second correction) image M ₂ I _k (j, i) after defect correction may be used as an output image.

Further, the effects of the above-described defect correction processing will be described with reference to FIGS. 15 (A) and 15 (B). FIG. 15A illustrates an example of the first viewpoint (first correction) image M ₁ I ₁ before the above-described defect correction. The main image is subjected to shading correction processing and demosaicing processing. This example shows the case where a line-shaped line defect in the horizontal direction (x direction) is generated at the central portion of the first viewpoint (first corrected) image M ₁ I ₁ (j, i). On the other hand, FIG. 15B shows an example of the first viewpoint (second correction) image M ₂ I ₁ (after demosaicing, after demosaicing) after the above-described defect correction. By the defect correction based on the normal captured image I, the line-like line defect in the horizontal direction (x direction) is corrected, and the image quality is good as in the captured image.

  In the present embodiment, the signal value of the viewpoint image at the first position determined to be a defect, the signal value of the captured image at the first position, and the signal values of the viewpoint image and the captured image at the second position not determined to be a defect Make corrections based on At this time, since the signal value of the viewpoint image subjected to shading correction (brightness correction) is used, the correction accuracy of the viewpoint image can be enhanced.

Furthermore, in step S2-2, the control unit 121 follows the equations (8A) to (8D) with respect to the defect-corrected k-th (second) image M ₂ I _k (j, i). Shading processing is performed to generate a _k-th ( _third ) corrected image M ₃ I _k (j, i). Here, the k-th viewpoint (third correction) image M ₃ I _k of the Bayer array is defined for each of R, Gr, Gb, and B. That is, the k th viewpoint (third correction) image of R is RM ₃ I _k ( ₂ j ₂ -1, ₂ i ₂ -1) = M ₃ I _k ( ₂ j ₂ -1, ₂ i ₂ -1), Gr k Let the viewpoint (third correction) image be GrM ₃ I _k (2j ₂ -1, 2i ₂ ) = M ₃ I _k (2j ₂ -1, 2i ₂ ). In addition, the Gb k-th viewpoint (third correction) image is represented by GbM ₃ I _k (2j ₂ , 2i ₂ -1) = M ₃ I _k (2j ₂ , 2i ₂ -1), and the B-th k-th viewpoint (third Correction) Let the image be BM ₃ I _k (2j ₂ , 2i ₂ ) = M ₃ I _k (2j ₂ , 2i ₂ ).

In S3, saturation signal processing is performed on the captured image I (j, i) and the k-th viewpoint (third corrected) image M ₃ I _k (j, i). In the present embodiment, the case where k = 1 and N _LF = 2 will be described as an example.

  First, in S3-1, the control unit 121 performs saturation signal processing on the captured image I (j, i). That is, according to the equation (9), saturation signal processing is performed with the maximum value of the imaging signal as Imax to generate a corrected captured image MI (j, i). Here, the maximum value Imax of the imaging signal and the saturation determination threshold IS of the imaging signal satisfy Imax ≧ IS.

Next, in step S3-2, the control unit 121 performs saturation signal processing on the k-th viewpoint (third corrected) image M ₃ I _k (j, i). That is, saturation signal processing is performed according to the shading state in which the shading function of the Bayer array is SF _k (j, i) according to the equation (10), and the k th viewpoint (fourth correction) image M ₄ I _k (j, i) Generate. Here, the shading function RSF _k (2i ₂ −1) generated for each of R, Gr, Gb, and B by the shading function SF _k (j, i) of the Bayer array is expressed by Expression (5A) to Expression (5D) _{, GrSF k (2i 2),} GbSF k (2i 2 -1), the _{BSF k (2i 2), SF} k (2j 2 -1,2i 2 -1) = RSF k (2i 2 -1), SF k (2j ₂ -1, 2i ₂ ) = GrSF _k (2i ₂ ), SF _k (2j ₂ , 2i ₂ -1) = GbSF _k (2i ₂ -1), SF _k (2j ₂ , 2i ₂ ) = BSF _k (2i ₂ )

Next, in S4-1, the control unit 121 causes the corrected captured image MI (j, i) generated in S3-1 and the first viewpoint (fourth correction) image M ₄ I generated in S3-2. _{From 1} (j, i), a second viewpoint image I ₂ (j, i) is generated according to equation (11).

Note that the maximum signal value at the time of saturation of the first viewpoint (third correction) image M ₃ I ₁ (j, i) depends on the driving method of the imaging device and the circuit configuration of A / D conversion. , I) may become the same maximum signal value as the maximum signal value Imax at the time of saturation. In such a case, when the first viewpoint (third correction) image is subtracted from the captured image by applying the equation (11) without performing the saturation signal processing, a saturation signal value is generated in the second viewpoint image to be generated. What should be done may be a false signal value of zero. In order to prevent such an operation, in S3, a saturated signal previously adjusted to the shading state for the captured image I (j, i) and the k-th viewpoint (third correction) image M ₃ I _k (j, i) Do the processing. Thus, the corrected captured image MI (j, i) after the saturation signal processing and the first viewpoint (fourth corrected) image M ₄ I ₁ (j, i) are generated. In S4-1, by generating the second viewpoint image I ₂ (j, i) according to Equation (11), it is possible to generate a second viewpoint image I ₂ corresponding to a more correct saturation signal value.

In S5, the control unit 121 performs shading correction on the first viewpoint (fourth corrected) image M ₄ I ₁ (j, i) and the second viewpoint image I ₂ (j, i). In S5-1, the shading functions RSF ₁ , GrSF ₁ , GbSF _{1 that} have been generated according to Equations (5A) to (5D) for the first viewpoint (fourth corrected) image M ₄ I ₁ (j, i) Apply BSF ₁ . Then, the shading correction process is performed according to the equations (6A) to (6D). At this time, the first viewpoint (fifth correction) image M ₅ I ₁ (j, i) is generated.

Next, in S5-2, the control unit 121 applies the modified captured image MI (j, i) to the second viewpoint image I ₂ (j, i) in the same manner as in the equations (3A) to (6D). Shading correction processing is performed to generate a second viewpoint (fifth correction) image M ₅ I ₂ (j, i). Here, also for the second viewpoint projection signal of the image I _2, similarly to the projection signal of the first viewpoint image I ₁ performs validity determination, generates a shading function with reliable projection signal.

Next, in S6, the control unit 121 sets the first viewpoint (fifth correction) image M ₅ I ₁ (j, i) and the second viewpoint (fifth correction) image M ₅ I ₂ (j, i) On the other hand, saturation signal processing is performed respectively. In S6-1, saturation signal processing is performed on the first viewpoint (fifth corrected) image M ₅ I ₁ (j, i) according to equation (12), and the corrected first viewpoint image MI ₁ (j, i) Generate On the other hand, in S6-2, saturation signal processing is performed on the second viewpoint (fifth corrected) image M ₅ I ₂ (j, i) according to equation (12), and the corrected second viewpoint image MI ₂ (j, i) Generate. Here, the maximum value Imax _{/ N LF} k-th viewpoint image, the saturation determination threshold value IS _k of the k viewpoint image satisfies Imax _{/ N LF} ≧ IS _k.

Hereinafter, with reference to FIGS. 16A and 16B, the effect of the shading correction process for each of the RGB with respect to the second viewpoint image I ₂ (j, i) shown in S5 will be described. FIG. 16A shows an example of the second viewpoint image I ₂ (after demosaicing) before the shading correction. In this example, since the shading for each of RGB occurs due to the pupil shift of the exit pupil of the imaging optical system and the entrance pupil of the imaging device, the luminance decreases on the left side of the second viewpoint image I ₂ (j, i) And modulation of the RGB ratio). On the other hand, FIG. 16B shows an example of a corrected second viewpoint image MI ₂ (after demosaicing) after shading correction according to the present embodiment. In this example, the reduction in luminance (and the modulation of the RGB ratio) is corrected by the shading correction for each RGB based on the captured image, and the corrected second viewpoint image MI ₂ (j (j , I) are generated.

Finally, the control unit 121 outputs each of the corrected first viewpoint image MI ₁ (j, i) and the corrected second viewpoint image MI ₂ (j, i). Alternatively, the control unit 121 outputs a combined image in which the corrected first viewpoint image MI ₁ (j, i) and the corrected second viewpoint image MI ₂ (j, i) are combined into one.

  As described above, in the present embodiment, the shading correction based on the shading function generated using the captured image and the viewpoint image is applied to the first viewpoint image and the second viewpoint image. At that time, by performing the reliability determination on the projection signal of the viewpoint image and generating the shading function using the highly reliable projection signal, it is possible to generate the shading function with high accuracy, and the shading on the viewpoint image The accuracy of the correction can be increased.

(Focus detection process of phase difference method)
Next, from the first viewpoint image and the second viewpoint image (first to Nth _LF viewpoint images), defocusing is performed by the phase difference method based on the correlation (degree of signal coincidence) of the first viewpoint image and the second viewpoint image. The focus detection process for detecting an amount will be described. In the following description, an example in which the control unit 121 executes a process will be described, but the image processing circuit 125 may execute it.

First, from the k-th viewpoint image Ik (k = 1 to N _LF ), which is a RGB signal of Bayer arrangement, the control unit 121 matches the color centers of RGB of each color for each position (j, i). The viewpoint luminance signal Yk is generated according to equation (13).

Next, the control unit 121 sets the shading correction amount S _k (i) of the k-th viewpoint luminance signal Y k (k = 1 to N _LF ) to the position i in the pupil division direction (x direction) according to equation (14). It is calculated as a smooth N _S-order polynomial function. Each coefficient SC _k (μ | F, based on the optical characteristic of the image sensor 107 (pupil intensity distribution for each k th sub-pixel) and the optical characteristic of the imaging optical system (diaphragm value F, exit pupil distance D1) Calculate Dl) and save it in a ROM not shown.

Furthermore, the control unit 121 performs a shading correction process on the kth viewpoint luminance signal Yk (j, i) using the shading correction amount S _k (i) according to the equation (15) Generate signal MYk (j, i).

Control unit 121, to the first modification viewpoint luminance signal MY ₁ produced, in the pupil division direction (horizontal direction), the one-dimensional band-pass filter process to generate a first focus detection signal Dya. Similarly, the second modified viewpoint luminance signal MY _2, the pupil division direction (horizontal direction), the one-dimensional band-pass filter process to generate a second focus detection signal DYB. In addition, as a one-dimensional band pass filter, for example, first derivative filters [1, 5, 8, 8, 8, 8, 5, 1 -1, -5, -8, -8, -8,- 8, -5, -1] and the like can be used. If necessary, the pass band of the one-dimensional band pass filter may be adjusted.

Next, the control unit 121 relatively shifts the first focus detection signal dYA and the second focus detection signal dYB in the pupil division direction (horizontal direction) to calculate a correlation amount representing the degree of signal coincidence. The image shift amount M _DIS is generated based on the calculated correlation amount.

For example, focus detection position _{(j AF, i} AF) around a _second vertical _j, horizontal _{i 2} th first focus detection signal _dYA a pupil division direction _{(j AF + j 2, i} AF + i 2 And the second focus detection signal is dYB (j _AF + j ₂ , i _AF + i ₂ ). However, j ₂ is −n ₂ ≦ j ₂ ≦ n ₂ , and i ₂ is −m ₂ ≦ i ₂ ≦ m ₂ . Furthermore, assuming that the shift amount is s (−ns _s s n _s ), the correlation amount COR _EVEN (j _AF , i _AF , s) at each position (j _AF , i _AF ) and the correlation amount COR _ODD (j _{AF 1} , i _AF , s) can be calculated by Equation (16A) and Equation (16B), respectively.

The correlation amount COR _ODD (j _AF , i _AF , s) is a shift amount between the first focus detection signal dYA and the second focus detection signal dYB with respect to the correlation amount COR _EVEN (j _AF , i _AF , s) By a half phase minus one shift.

The control unit 121 performs the sub-pixel calculation from the correlation amount COR _EVEN (j _AF , i _AF , s) and the correlation amount COR _ODD (j _AF , i _AF , s), and the correlation amount becomes the minimum value. Calculate the shift amount of real numbers. Furthermore, an average value of the calculated shift amount, focus detection position _{(j AF, i} AF) image shift in weight _Dis (j _{AF, i} AF) detected.

Next, the control unit 121 multiplies the image shift amount Dis (j _AF , i _AF ) for each image height position of the focus detection area by converting the image shift amount K from the image shift amount to the defocus amount, and detects each focus detection position _{(j AF, i} AF) defocus amount _{M Def (j AF, i AF} ) in detecting the. The conversion coefficient K is calculated according to, for example, the optical characteristics of the image pickup element (pupil intensity distribution for each k th subpixel) and lens information (the aperture value F of the imaging optical system, the exit pupil distance Dl, etc.) Save to ROM not shown.

Finally, the control unit 121 drives the lens to the in-focus position according to the defocus amount M _Def (j _AF , i _AF ) detected at the focus detection position (j _AF , i _AF ) to perform focus detection processing. Finish.

  When performing automatic focus detection using the first viewpoint image and the second viewpoint image (a plurality of viewpoint images), high-speed shading correction is required to perform automatic focus detection in real time processing with excellent responsiveness. There is a case. Therefore, in the present embodiment, when automatic focus detection is performed, shading correction may be performed at high speed using shading correction related data calculated in advance and stored in a ROM (not shown) or the like. In this way, it is possible to use a phase difference type focus detection signal of good signal quality, and it is possible to realize automatic focus detection with high accuracy and excellent response. The shading correction related data can be calculated based on the optical characteristic of the image pickup element (pupil intensity distribution for each k th subpixel) and the optical characteristic of the imaging optical system (aperture value F, exit pupil distance Dl) . In this case, when automatic focus detection is performed, shading correction may be applied only to a specific viewpoint image (for example, a first viewpoint image) to perform automatic focus detection at higher speed.

  In the present embodiment, when outputting a viewpoint image having a high image quality or a composite image of these, an accurate shading correction for each RGB is performed based on a captured image. On the other hand, in the case of automatic focus detection in real-time processing where excellent responsiveness is required, an example of switching shading correction methods and performing high-speed shading correction using shading correction related data calculated in advance and stored on a recording medium explained. As a result, it is possible to achieve both image output of a viewpoint image with high image quality and high-accuracy automatic focus detection with excellent responsiveness.

  As described above, the captured image and the viewpoint image are generated on the basis of the LF data obtained from the imaging device having a structure in which the photoelectric conversion unit in the unit pixel is divided, and shading on the viewpoint image Bias) was corrected. Furthermore, defective pixels are corrected using a viewpoint image that has been subjected to shading correction. By doing this, it is possible to generate a viewpoint image of good image quality. In other words, when using the LF data, it is possible to reduce the fluctuation of the light amount of the generated viewpoint image.

  The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or storage medium, and one or more processors in a computer of the system or apparatus read and execute the program. Can also be realized. It can also be implemented by a circuit (eg, an ASIC) that implements one or more functions.

  107: Imaging device, 121: Control unit, 125: Image processing circuit

Claims (11)

Among the pupil regions of the imaging optical system, viewpoint images generated from signals obtained by partial photoelectric conversion units of a plurality of photoelectric conversion units that receive light beams passing through different partial regions, and all photoelectric conversion units Image acquisition means for acquiring a captured image generated from the signal obtained by
Correction amount acquisition means for acquiring the luminance correction amount of the viewpoint image using the viewpoint image and the captured image;
Correction means for correcting the brightness of the viewpoint image based on the brightness correction amount;
An image processing apparatus having
The correction amount acquisition unit
Projection means for generating projection signals of the captured image and the viewpoint image;
A determination unit that determines the validity of the projection signal of the viewpoint image;
Generation means for generating the luminance correction amount using a projection signal determined to be effective by the determination means among projection signals of the viewpoint image, and a projection signal of the captured image;
An image processing apparatus comprising: The projection unit projects the captured image and the viewpoint image in a predetermined projection direction to generate a one-dimensional projection signal.
The determination means determines the validity of the signal by comparing an evaluation value based on the values of a plurality of pixels aligned in the projection direction of the viewpoint image with a threshold.
The image processing apparatus according to claim 1,   The image processing apparatus according to claim 2, wherein the evaluation value is generated without using a value of a pixel corresponding to a defective pixel position among the plurality of pixels.   The image processing apparatus according to claim 2, wherein the evaluation value is a squared cumulative value of values of pixels not corresponding to a defective pixel position among the plurality of pixels.   The image processing apparatus according to claim 2, wherein the evaluation value is an evaluation value in consideration of an SN ratio of the viewpoint image.   The determination means determines that a projection signal corresponding to the plurality of pixels is valid if the evaluation value based on the plurality of pixels is larger than the threshold, and the plurality of pixels is determined if the evaluation value is equal to or less than the threshold The image processing apparatus according to any one of claims 2 to 5, wherein the projection signal corresponding to is determined to be invalid.   When the number or ratio of the signals determined to be effective by the determination unit among the projection signals is equal to or less than a threshold, generation of the luminance correction amount and correction of the luminance of the viewpoint image based on the luminance correction amount are not performed. The image processing apparatus according to any one of claims 2 to 6, wherein   The correction amount acquiring unit acquires the luminance correction amount based on a ratio of a projection signal determined to be effective by the determination unit among projection signals of the viewpoint image to a projection signal of the corresponding captured image. The image processing apparatus according to any one of claims 2 to 7, characterized in that: An image processing apparatus according to any one of claims 1 to 8.
An imaging device in which a plurality of pixels including the plurality of photoelectric conversion units are arranged;
An imaging apparatus characterized by having: An image processing method performed by an image processing apparatus, comprising:
Among the pupil regions of the imaging optical system, viewpoint images generated from signals obtained by partial photoelectric conversion units of a plurality of photoelectric conversion units that receive light beams passing through different partial regions, and all photoelectric conversion units An image acquisition step of acquiring a captured image generated from a signal obtained by
A correction amount acquisition step of acquiring a luminance correction amount of the viewpoint image using the viewpoint image and the captured image;
Correcting the luminance of the viewpoint image based on the luminance correction amount;
Have
In the correction amount acquisition process,
A projection step of generating projection signals of the captured image and the viewpoint image;
A determination step of determining the validity of the projection signal of the viewpoint image;
A generation step of generating the luminance correction amount using a projection signal determined to be effective in the determination step among projection signals of the viewpoint image and a projection signal of the captured image;
An image processing method comprising:   The program for functioning a computer as each means of the image processing apparatus of any one of Claims 1-8.