JP6497977B2 - Image processing apparatus, imaging apparatus, image processing method, image processing program, and storage medium - Google Patents

Description

  The present invention relates to an image processing method for improving the quality of a captured image.

  When imaging is performed by an imaging device such as a camera, part of light incident on the optical system may be reflected by the lens interface or a member holding the lens and reach the imaging surface as unnecessary light. Unnecessary light reaching the imaging surface appears in the captured image as unnecessary components such as ghosts and flares. In a telephoto lens, when a diffractive optical element is used to correct axial chromatic aberration and lateral chromatic aberration, light from a high-intensity object such as the sun outside the imaging angle of view strikes the diffractive optical element. Unnecessary light may appear across the screen. Such unnecessary light also appears in the captured image as an unnecessary component. Therefore, conventionally, a method for removing unnecessary components by digital image processing is known.

  Patent Document 1 and Patent Document 2 disclose a method for detecting a ghost by comparing a plurality of viewpoint images obtained by monocular stereoscopic imaging. According to the methods of Patent Literature 1 and Patent Literature 2, since a plurality of parallax images can be obtained by one imaging, it is possible to cope with still image imaging and moving image imaging of a moving subject.

JP 2011-205531 A JP 2014-171236 A

  However, the method of Patent Document 1 is based on the premise that the parallax component is corrected in advance, and a ghost cannot be accurately detected when the parallax component remains.

  On the other hand, Patent Document 2 discloses a method for detecting a ghost when a parallax component remains. When the amount of parallax component is relatively small in a distant view such as a landscape photograph, the parallax component can be removed by applying a low-pass filter to the parallax image. However, if the amount of parallax component is large, such as when focusing on a subject at a short distance, it is not sufficient to apply a low-pass filter to remove the parallax component. On the other hand, if the low-pass filter is strongly applied, the ghost to be detected is also reduced, and it is difficult to detect the ghost appropriately.

  Therefore, in this embodiment, an image processing apparatus and an imaging apparatus capable of determining a first unnecessary component of a captured image without performing a plurality of imaging operations and determining a second unnecessary component obtained by reducing the subject parallax component from the first unnecessary component. An image processing method, an image processing program, and a storage medium are provided.

An image processing apparatus according to one aspect of the present invention includes a first determination unit that determines a first unnecessary component based on difference information of a plurality of parallax images, and a second unnecessary component by reducing the parallax component from the first unnecessary component. have a second determining means for determining a component, the second determining means, said first required component, based on the mask created based on the first required component, the second required component To decide .

An imaging apparatus according to another aspect of the present invention includes an imaging unit that photoelectrically converts an optical image formed through an optical system and outputs a plurality of parallax images, and a first unit based on difference information between the plurality of parallax images. 1 possess a first determining means for determining a required component, and a second determining means for determining a second required component by reducing the parallax component from the first unwanted component, the second determining means, said first The second unnecessary component is determined based on the unnecessary component and a mask created based on the first unnecessary component .

An image processing method according to another aspect of the present invention includes a step of determining a first unnecessary component based on difference information of a plurality of parallax images, and reducing a parallax component from the first unnecessary component to obtain a second unnecessary component. possess a step of determining, the second required component, said first required component is determined based on the mask created based on the first unnecessary components.

An image processing program according to another aspect of the present invention is configured to cause a computer to execute the image processing method .

  A storage medium according to another aspect of the present invention stores the image processing program.

  Other objects and features of the present invention are illustrated in the following examples.

  According to the present invention, an image processing apparatus and an imaging apparatus capable of determining a first unnecessary component of a captured image without performing a plurality of imaging operations and determining a second unnecessary component obtained by reducing a subject parallax component from the first unnecessary component. An image processing method, an image processing program, and a storage medium can be provided.

FIG. 3 is a diagram illustrating a procedure of an image processing method in Embodiment 1. 3 is an example of an output image obtained by the image processing method according to the first embodiment. In the imaging system of Examples 1-4, it is a related figure of the light-receiving part of an image pick-up element, and the pupil of an optical system. It is a schematic diagram of the imaging system in Examples 1-4. It is a block diagram of the imaging device in each embodiment. It is explanatory drawing of the structure of the optical system in each Example, and the unnecessary light which generate | occur | produces in an optical system. It is explanatory drawing of the unnecessary light which permeate | transmits the aperture_diaphragm | restriction of the optical system in Examples 1-4. 3 is a flowchart illustrating an image processing method according to the first exemplary embodiment. 10 is a flowchart illustrating an image processing method according to the second exemplary embodiment. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 2. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 2. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 2. 10 is a flowchart illustrating an image processing method according to the third exemplary embodiment. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 3. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 3. 10 is a flowchart illustrating an image processing method according to a fourth embodiment. 14 is an example of an output image obtained by the image processing method according to the fourth embodiment. FIG. 9 is a diagram illustrating an image sensor in Example 5. FIG. 10 is an explanatory diagram of unnecessary light that passes through a diaphragm of an optical system in Example 5. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 5. FIG. 10 is a diagram illustrating a procedure of an image processing method in Embodiment 5. FIG. 10 is a diagram illustrating an imaging system in Embodiment 6. FIG. 10 is a diagram illustrating an imaging system in Embodiment 6. FIG. 10 is a diagram illustrating an imaging system in Embodiment 6. It is a figure which shows the conventional image pick-up element. It is a figure which shows the image obtained by the imaging system of FIG. It is a figure which shows the image obtained by the imaging system of FIG. 23 and FIG. FIG. 10 is a diagram illustrating an example of an imaging apparatus according to a sixth embodiment. FIG. 10 is a diagram illustrating an example of an imaging apparatus according to a sixth embodiment. FIG. 10 is a configuration diagram of an imaging system in Embodiment 7.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  In each embodiment, an imaging apparatus capable of generating a plurality of parallax images causes a plurality of light beams that have passed through different regions of the pupil of an optical system (imaging optical system) to different light receiving units (pixels) in the imaging device. An imaging system that guides and performs photoelectric conversion is included.

  First, Example 1 of the present invention will be described. FIG. 3 is a diagram illustrating the relationship between the light receiving unit of the image sensor and the pupil of the optical system in the imaging system of the present embodiment. In FIG. 3, ML is a microlens, and CF is a color filter. EXP is an exit pupil (pupil) of the optical system, and P1 and P2 are areas of the exit pupil EXP. G1 and G2 are pixels (light receiving units), and one pixel G1 and one pixel G2 are paired with each other (the pixels G1 and G2 are provided so as to share one microlens ML). . In the imaging device, a plurality of pairs of pixel G1 and pixel G2 (pixel pairs) are arranged. The pair of pixels G1 and G2 have a conjugate relationship with the exit pupil EXP through a common microlens ML (that is, one for each pixel pair). In each embodiment, the plurality of pixels G1 and G2 arranged in the image sensor may be collectively referred to as a pixel group G1 and G2, respectively.

  FIG. 4 is a schematic diagram of the imaging system in the present embodiment, and shows an imaging relationship between the object point OSP, the exit pupil EXP, and the imaging element that are imaging. The pixel G1 receives the light beam that has passed through the region P1 in the exit pupil EXP. The pixel G2 receives the light beam that has passed through the region P2 in the exit pupil EXP. The OSP is an object point being imaged. The object point OSP does not necessarily have an object. The light beam passing through the object point OSP is applied to either the pixel G1 or the pixel G2 according to the position (region P1 or region P2 in this embodiment) in the pupil (exit pupil EXP) through which the light beam passes. Incident. The passage of light beams through different regions in the pupil corresponds to the separation of incident light from the object point OSP by the angle (parallax). That is, among the pixels G1 and G2 provided for each microlens ML, an image generated using the output signal from the pixel G1 and an image generated using the output signal from the pixel G2 are: It becomes a plurality (here, a pair) of parallax images having parallax with each other. In the following description, receiving light beams that have passed through different regions in the pupil by different light receiving units (pixels) may be referred to as pupil division.

  Further, even when the above-described conjugate relationship is not perfect due to the positional deviation of the exit pupil EXP shown in FIGS. 3 and 4, or when the regions P1 and P2 partially overlap (overlap) with each other, In the example, a plurality of obtained images are treated as parallax images. In addition, a minimum element constituting an image is referred to as a pixel (hereinafter referred to as a pixel in order to distinguish it from a pixel on the image sensor), and each pixel represents light intensity and color by a numerical value. The value of each pixel is called a pixel value. If the pixel value is a monochrome image, the pixel value is set to the luminance value. In each embodiment of the present invention, the pixel value will be described as a monochrome image for simple explanation. Therefore, here, the pixel value and the luminance value have the same meaning. In the case of an RGB color image, the same calculation may be performed for each color for the pixel value of each color. The same applies to each of the following embodiments.

  Next, with reference to FIG. 5, an imaging apparatus that executes the image processing method in the present embodiment will be described. FIG. 5 is a block diagram illustrating a configuration of the imaging apparatus 200 in the present embodiment. The optical system 201 (imaging optical system) includes a stop 201a and a focus lens 201b, and focuses (condenses) light from a subject (not shown) on the imaging element 202. The imaging element 202 (imaging means) is composed of a photoelectric conversion element such as a CCD sensor or a CMOS sensor, and corresponds to each region of light beams that have passed through different areas in the pupil described with reference to FIGS. Receive light (perform pupil division). As described above, the image sensor 202 photoelectrically converts the subject image (optical image) formed via the optical system 201 and outputs image signals (analog electrical signals) that are a plurality of parallax images. The A / D converter 203 converts the analog electric signal output from the image sensor 202 into a digital signal, and outputs the digital signal to the image processing unit 204.

  The image processing unit 204 performs image processing that is generally performed on the digital signal, and performs unnecessary light (unnecessary component) determination processing and correction processing that reduces or eliminates unnecessary light. The image processing unit 204 performs an unnecessary component region extraction process. In this embodiment, the image processing unit 204 corresponds to an image processing device mounted on the imaging device 200. The image processing unit 204 includes an unnecessary component detection unit 204a (first determination unit), an extraction processing unit 204b (second determination unit), and an unnecessary component reduction unit 204c (reduction unit).

  The unnecessary component detection unit 204a generates (acquires) a parallax image and detects (determines) an unnecessary component from the parallax image. The extraction processing unit 204b reduces the parallax component from the detected unnecessary component and extracts an unnecessary component region such as a ghost. The unnecessary component reduction unit 204c reduces unnecessary components from each parallax image. In the present embodiment, as a method of generating a parallax image, output is generated in the form of being separated into two from the beginning, such as “an image including only the pixel group G1” and “an image including only the pixel group G2”. Can do. Alternatively, the “image composed only of the pixel group G1” and the “composite image of the pixel group G1 and the pixel group G2” are first output, and the image composed only of the pixel group G1 is subtracted from the composite image, so that only the pixel group G2 An image corresponding to the image may be obtained by calculation.

  The output image (image data) processed by the image processing unit 204 is stored in an image recording medium 209 such as a semiconductor memory or an optical disk. Further, the output image from the image processing unit 204 can be displayed on the display unit 205. The storage unit 208 stores an image processing program and various information necessary for image processing by the image processing unit 204.

  A system controller 210 (control means) controls the operation of the image sensor 202, processing in the image processing unit 204, and control of the optical system 201 (aperture 201a and focus lens 201b). The optical system control unit 206 mechanically drives the aperture 201a and the focus lens 201b of the optical system 201 in accordance with a control instruction from the system controller 210. The aperture of the aperture 201a is controlled according to the set aperture value (F number). The position of the focus lens 201b is controlled by an auto focus (AF) system (not shown) or a manual focus mechanism in order to perform focus adjustment (focus control) according to the subject distance. The state detection unit 207 acquires current shooting condition information in accordance with a control instruction from the system controller 210. In this embodiment, the optical system 201 is configured as a part of the imaging apparatus 200 including the imaging element 202 (integrated with the imaging apparatus 200), but is not limited thereto. The present embodiment can also be applied to an imaging apparatus in which an interchangeable optical system (interchangeable lens) is detachably attached to the imaging apparatus body, such as a single-lens reflex camera.

  FIG. 6 is an explanatory diagram of the configuration of the optical system 201 and unnecessary light generated in the optical system 201. FIG. 6A shows a specific configuration example of the optical system 201. In FIG. 6A, STP is a diaphragm (corresponding to the diaphragm 201a), and IMG is an imaging surface. The image sensor 202 shown in FIG. 5 is arranged at the position of the imaging surface IMG. In FIG. 6B, strong light is incident on the optical system 201 from the sun SUN as an example of a high-luminance object, and the light reflected by the lens interface constituting the optical system 201 is converted into unnecessary components A (ghost, flare, etc.). The state of reaching the imaging surface IMG as unnecessary light) is shown. FIG. 6C shows an imaging surface in which strong light is incident as in FIG. 6B, and light reflected at the lens interface different from unnecessary light A is used as unnecessary component B (unnecessary light such as ghost and flare). It shows how the IMG is reached.

  FIG. 7 shows regions P1 and P2 (pupil regions or pupil division regions) through which the light beams incident on the pixels G1 and G2 shown in FIG. The stop STP can be considered as corresponding to the exit pupil EXP of the optical system 201 (virtual image viewed from the image plane position of the optical system 201), but actually the stop STP and the exit pupil EXP are different from each other. The light beam from the high-intensity object (sun SUN) passes through almost the entire area of the stop STP, but the region through which the light beam incident on the pixels G1 and G2 passes is divided into regions P1 and P2 (pupil regions). In the example shown in FIGS. 6B and 6C, the light beam from the high-intensity object passes through a substantially lower half area of the stop STP. Referring to FIG. 4, a part of the light beam is in the area P1. All the remaining light fluxes pass through the region P2. The light beam that has passed through the region P1 enters the pixel G1, and the light beam that has passed through the region P2 enters the pixel G2.

  Next, with reference to FIGS. 1 and 2, a method of determining an unnecessary component that is an image component that appears in a captured image generated by the imaging apparatus 200 by photoelectric conversion of unnecessary light will be described. FIG. 1 is a diagram illustrating a procedure of an image processing method in the present embodiment. FIG. 2 is an example of an output image by the image processing method in the present embodiment. When imaging is performed with the optical system 201 in FIG. 6A, the unnecessary component A generated in the optical path in FIG. 6B and the unnecessary component B generated in the optical path in FIG. . For this reason, in FIG. 1 and FIG. 2, the unnecessary component A and the unnecessary component B are drawn in an overlapping manner. However, even if a plurality of unnecessary components are overlapped or separated, the spirit and basic idea of the present embodiment are not changed, and the method for determining unnecessary components described later is not changed.

  FIG. 2 shows a captured image generated by “imaging without pupil division”, and fine objects are omitted for simplicity. The photographed image includes a gray portion of the background, a triangular subject A that is in focus, a subject B that is located behind the subject A and is out of focus, and two squares (ghost and subject) Unnecessary components A and B) that are higher in luminance than the background and overlap each other are shown. Actually, the subject is transparent to some extent in the background of these unnecessary components. Further, the unnecessary component is a portion where the photographic subject is covered with unnecessary light, and thus the luminance is higher than that of the photographic subject. For this reason, the luminance is shown to be higher than the gray portion of the background. This also applies to the drawings in other embodiments described later.

  FIGS. 1A-1 and 1B-1 show a pair of parallax images obtained as a result of photoelectric conversion of light beams that have passed through the regions P1 and P2 (pupil regions) by the pixel groups G1 and G2, respectively. Show. In the pair of parallax images, there is a difference (subject parallax component) corresponding to the parallax of the image components. For this reason, the subject B in FIGS. 1A-1 and 1B-1 is shifted to the left and right (parallax shift). The pair of parallax images also include an unnecessary component A and an unnecessary component B, which are schematically shown as gray squares with uniform luminance, but the luminance differs between the parallax images. Here, an example in which the unnecessary component A and the unnecessary component B overlap with each other as described above is shown, but a state in which they are separated without overlapping may be used. That is, it is only necessary that the position and luminance of the unnecessary component are different between the parallax images.

  FIGS. 1A-2 and 1B-2 show luminance cross sections of a pair of parallax images along broken line portions in FIGS. 1A-1 and 1B-1, respectively. The numerical values in the graphs of FIGS. 1A-2 and B-2 are luminance values Y of unnecessary components. For example, in FIG. 1A-2, the background luminance value is 50, and the luminance value of the subject B having a parallax shift is 100. FIG. 1C-1 shows an image (parallax composite image) obtained by adding and synthesizing FIGS. 1A-1 and 1B-1. FIG. 1C-2 shows a luminance cross section of the parallax composite image along the broken line portion in FIG. This parallax composite image is equivalent to a captured image (FIG. 2) generated by “imaging without pupil division”. In the present embodiment, by adding and synthesizing a pair of parallax images, the brightness is equivalent to that of a captured image generated by “imaging without pupil division”. You may use the imaging device which becomes the brightness equivalent to the picked-up image produced | generated by "the imaging which does not perform pupil division" by carrying out the averaging of a pair of parallax image. Such a case will be described later in a fifth embodiment.

  FIG. 1D-1 relates to a pair of parallax images, an image in a state where the image of FIG. 1B-1 is subtracted from FIG. 1A-1, and FIG. An image (difference synthesized image, first unnecessary component image) obtained by adding and synthesizing an image obtained by subtracting the image of (A-1) is shown. FIG. 1D-2 shows a luminance cross section along the broken line portion in FIG. 1D-1 is an image obtained by removing the subject and the background from FIG. 1C-1, and the unnecessary component and subject included in FIG. Contains only the parallax component.

  In this embodiment, in order to calculate FIG. 1 (D-1), the difference is taken twice as described above, and then addition synthesis is performed. However, the absolute value of the difference is obtained as in the following equation (1). Even if an operation for obtaining is performed, it is equivalent.

FIG1D1 (x, y) = | FIG1A1 (x, y) −FIG1B1 (x, y) | (1)
In Expression (1), FIG. 1D1 (x, y), FIG. 1A1 (x, y), and FIG. 1B1 (x, y) are respectively shown in FIG. 1 (D-1), FIG. 1 (A-1), and FIG. -1) represents the luminance value at each coordinate. As a result, the result of FIG. 1 (D-1) is obtained by one calculation. Thus, the first unnecessary component is determined, and FIG. 1D-1 is a first unnecessary component image obtained by imaging the first unnecessary component.

  In this way, by performing the difference calculation for each parallax image, only the unnecessary component remains (in other words, separated or extracted), and the first unnecessary component (first unnecessary component image) can be determined. However, when the subject parallax component exists as in the present embodiment, the unnecessary component and the subject parallax component are mixed in the first unnecessary component image. For this reason, it is necessary to reduce or remove the subject parallax component in order to determine the unnecessary component with high accuracy. Therefore, in this embodiment, the subject parallax component is reduced from the first unnecessary component. In the present embodiment, each process described later is performed based on the first unnecessary component. However, it is not necessary to store the first unnecessary component image as a so-called “image” that can be displayed and confirmed by the user later. The first unnecessary component image may be used as numerical data in the processing flow.

  Next, a method for removing or reducing the subject parallax component from the first unnecessary component determined as described above will be described. The subject parallax component is a component related to a difference between a plurality of parallax images. For this reason, the subject parallax component is often a linear component such as an edge, or a component having a small luminance value even if the surface is somewhat wide. On the other hand, an unnecessary component that is originally desired to be reduced from a captured image such as a ghost is often a component that has a relatively large surface and a relatively large luminance value. For this reason, in order to reduce the subject parallax component, the luminance value is reduced by blurring a linear component such as an edge, and the contrast of the subject parallax component is increased by performing processing for increasing the contrast of a portion having a small luminance value. It can approach (reduce) to zero. Further, the last remaining noise component can be reduced by performing a certain threshold processing. And it can determine with the component which remained after performing these processes being an unnecessary component. However, unnecessary components may be slightly reduced by blurring the linear component or increasing the contrast of a portion having a small luminance value. Therefore, in the present embodiment, the region enlargement process is performed on the remaining region (hereinafter, these processes are collectively referred to as an extraction process). Thereby, an unnecessary component area | region (1st unnecessary component area | region) can be determined automatically. In this embodiment, the subject parallax component is reduced by performing the extraction process on the first unnecessary component.

  Next, an image processing method (subject parallax component reduction processing) in the present embodiment will be described with reference to FIG. FIG. 8 is a flowchart showing an image processing method. Each step in FIG. 8 is executed by the system controller 210 or the image processing unit 204 in accordance with an image processing program as a computer program.

  First, in step S <b> 101, the system controller 210 controls the image capturing unit configured by the optical system 201 and the image sensor 202 to image the subject. The image processing unit 204 acquires a captured image as an input image.

  Subsequently, in step S102, the image processing unit 204 uses the digital signal output from the image sensor 202 (pixel group G1, G2) and A / D converted by the A / D converter 203 to generate a pair of parallax images. Generate. Here, the image processing unit 204 may perform normal development processing and various image correction processing in order to generate a parallax image.

  Subsequently, in step S103, the image processing unit 204 (unnecessary component detection unit 204a) obtains difference information between the pair of parallax images. That is, the image processing unit 204 includes a difference image obtained by subtracting FIG. 1 (B-1) from FIG. 1 (A-1), and a difference image obtained by subtracting FIG. 1 (A-1) from FIG. 1 (B-1). Are added and combined to generate the difference composite image of FIG. Thus, in the simple difference calculation, the difference value of the unnecessary component takes a positive value and a negative value. For example, in this embodiment, when FIG. 1 (B-1) is subtracted from FIG. 1 (A-1) in order to generate the difference composite image of FIG. 1 (D-1), FIG. The luminance value of the unnecessary component included is larger than the luminance value of the unnecessary component included in FIG. For this reason, the difference value becomes a positive value. Similarly, when FIG. 1A-1 is subtracted from FIG. 1B-1, the difference value becomes a negative value. Here, in this embodiment, in order to simplify the unnecessary component reduction process described later, the negative value is rounded down to a zero value.

  Subsequently, in step S104, the image processing unit 204 (unnecessary component detection unit 204a) determines the component remaining in the difference image (difference composite image) obtained in step S103 as the first unnecessary component. Here, the first unnecessary component image is obtained by imaging the first unnecessary component. Specifically, only a difference value between unnecessary components included in FIGS. 1A-1 and 1B-1 is detected as a positive value. The unnecessary component detection unit 204a determines this as the first unnecessary component and generates a first unnecessary component image (FIG. 1 (D-1)). However, the first unnecessary component image does not need to be generated or stored as so-called “image data” that can be confirmed by the user, and may be used for calculation of processing. If the difference image is calculated with the absolute value of the difference as in the above equation (1), the difference calculation may be performed once. Further, in order to further improve the processing speed, the difference image may be treated as the first unnecessary component to skip step 104 and proceed to step S105 following step S103.

  Subsequently, in step S <b> 105, the image processing unit 204 generates an image (parallax composite image) equivalent to the captured image generated by one “imaging without pupil division” by performing an additive synthesis process on the parallax images. To do. For example, by executing a process of adding the parallax image of FIG. 1A-1 generated in step S102 and the parallax image of FIG. 1B-1 to FIG. The added and synthesized parallax composite image shown is generated. Alternatively, in step S105, the digital signal output from the image sensor 202 (pixel group G1, G2) and A / D converted by the A / D converter 203 without passing through the step (S102) of generating a parallax image is obtained. You may produce | generate by adding. Further, step S105 is not necessarily executed at this position, and the execution position of this step is particularly limited as long as it is executed before step S107 so that the parallax composite image can be used in subsequent step S107. Is not to be done.

  The subsequent step S108 (S120 to S125) is a step (extraction process step) for determining an unnecessary component region (first unnecessary component region as a mask) based on the first unnecessary component determined in step S104. Step S108 is executed by the image processing unit 204 (extraction processing unit 204b). First, in step S120, the image processing unit 204 (extraction processing unit 204b) performs a smoothing process on the first unnecessary component. FIG. 1E-1 shows an image that has been smoothed. FIG. 1E-2 shows a luminance cross section taken along the broken line in FIG. Thereby, the edge portion of the subject parallax component can be blurred, and the overall luminance value of the subject parallax component can be reduced. Specifically, moving average processing, Gaussian filter processing, and the like are performed as the smoothing processing.

  Subsequently, in step S121, the image processing unit 204 (extraction processing unit 204b) performs processing for increasing contrast (contrast increasing processing) on the image blurred by the smoothing processing in step S120. FIG. 1F-1 shows an image with increased contrast. FIG. 1F-2 shows a luminance cross section along the broken line portion in FIG. As a result, the luminance value of the blurred portion becomes smaller, and the luminance value of the portion having a relatively high luminance value such as an unnecessary component becomes higher.

  Subsequently, in step S122, the image processing unit 204 (extraction processing unit 204b) performs threshold processing. Specifically, for example, a process of setting all luminance values in an area where the luminance value is 30 or less to zero is performed. In this embodiment, for the sake of simplicity, a relatively large luminance value of 30 remains. However, in practice, it is preferable that the smoothing process is more aggressively performed and the luminance value of about 2 to 10 is made zero by the threshold process (in the 8-bit image). As a result, it is possible to avoid erroneously removing unnecessary components originally desired to be left in this step (step S122). As a result, the subject parallax component and the noise component can be effectively removed, and finally only the unnecessary component can be left. FIG. 1G-1 shows an image in which only unnecessary components remain. FIG. 1G-2 shows a luminance cross section taken along the broken line in FIG.

  If many subject parallax components still remain after the threshold processing in step S122, the process may return to step S120 and repeat steps S120 and S121. Thereby, the subject parallax component can be reduced stepwise.

  Subsequently, in step S123, the image processing unit 204 (extraction processing unit 204b) performs binarization processing on the remaining unnecessary components. In the binarization processing, the luminance value is left as zero for the region where the luminance value is zero, and all the luminance values are set to 1 (or 8 for the region where the luminance value is 1 or more (or a predetermined threshold value or more). If it is a bit grayscale image, the processing is performed to 255). Thereby, the remaining unnecessary components can be regarded as “regions (regions where unnecessary components exist)”. FIG. 1 (H-1) shows an image after binarization processing. FIG. 1 (H-2) shows a luminance cross section along the broken line in FIG. 1 (H-1).

  Subsequently, in step S124, the image processing unit 204 (extraction processing unit 204b) performs processing (region expansion processing) for expanding the “region where unnecessary components exist” binarized in step S123. FIG. 1 (I-1) shows an image after the area enlargement process. FIG. 1 (I-2) shows a luminance cross section along the broken line in FIG. 1 (I-1). By performing the smoothing process in step S120, the contrast increasing process in step S121, and the threshold process in step 122, the subject parallax component can be removed or reduced. At the same time, however, unnecessary components are reduced to some extent. In this case, there is a possibility that the “region where the unnecessary component exists” that is finally left after binarization does not completely cover the entire region including the unnecessary component. Therefore, in step S124, the region where the unnecessary component reduced by the smoothing process or the like is expanded so as to cover the entire region actually including the unnecessary component. Note that unnecessary components such as ghosts may have sharp edges or blurs, and there are many unnecessary components in which it is difficult to actually define the contour. For this reason, instead of strictly extracting the outline of the unnecessary component, it is more natural to set the unnecessary component region that is somewhat wider than the actual unnecessary component as in this embodiment when the unnecessary component reduction process is performed later. Is obtained. Therefore, the peripheral part of the unnecessary component may be extracted a little more.

  Subsequently, in step S125, the image processing unit 204 (extraction processing unit 204b) determines an unnecessary component region (first unnecessary component region) based on the result of step S124. The first unnecessary component region is used as a mask for reducing or removing the parallax component included in the first unnecessary component.

  Next, in step S106, the image processing unit 204 (extraction processing unit 204b) is based on the first unnecessary component determined in step S104 and the first unnecessary component region (mask) determined in step S125. Thus, the second unnecessary component is determined. Specifically, the extraction processing unit 204b performs a masking process using the first unnecessary component region (mask) on the first unnecessary component, and only the region including unnecessary components such as ghosts from the first unnecessary component. Extracted as a second unnecessary component. For example, for the region having the luminance value of 255 in FIG. 1 (I-1), the luminance value of the same region in FIG. 1 (D-1) is maintained. On the other hand, regarding the region where the luminance value in FIG. 1 (I-1) is zero, the luminance value in the same region in FIG. 1 (D-1) is replaced with zero. Thereby, the image of FIG. 1 (J-1) obtained by reducing the subject parallax component from the first unnecessary component image of FIG. 1 (D-1) is obtained.

  As described above, a component obtained by reducing or removing the subject parallax component from the first unnecessary component (first unnecessary component image) is referred to as a second unnecessary component, and an image obtained by imaging the second unnecessary component is a second unnecessary component image (FIG. 1 (J-1)). In this embodiment, a mask with clearly divided regions is created as shown in FIG. 1 (I-1), but the boundary between the luminance values of zero and 255 in FIG. 1 (I-1) is blurred. Or a gradation mask. In this case, masking processing is performed on FIG. 1D-1 using the gradation portion as a weighting coefficient.

  Finally, in step S107, the image processing unit 204 (unnecessary component reduction unit 204d) reduces unnecessary components based on the second unnecessary component determined in step S106 and the parallax composite image generated in step S105. Process. Specifically, the unnecessary component reduction unit 204d subtracts the second unnecessary component (second unnecessary component image) in FIG. 1 (J-1) from the parallax composite image in FIG. 1 (C-1). Thereby, the unnecessary component reduced image shown in FIG. 1 (K-1) is generated.

  In the present embodiment, the smoothing amount (blurring amount) in the smoothing process, the contrast change amount in the contrast-up process, the threshold value in the threshold process, and the enlargement amount in the area enlargement process can be changed as appropriate. In the present embodiment, the smoothing process, the contrast increasing process, and the threshold process are not limited to these, and other processes may be used as long as they have similar effects. In this embodiment, an example of a gray scale image is shown for the sake of simplicity of explanation, but a color image can be similarly applied. In this case, the above-described processing is performed independently for each color channel, and the respective colors are finally combined into one image.

  According to the present embodiment, it is possible to determine the second unnecessary component in which the subject parallax component is reduced from the first unnecessary component including the subject parallax component. Further, according to the present embodiment, it is possible to generate an unnecessary component reduced image in which the second unnecessary component is reduced from the parallax composite image.

  Next, a second embodiment of the present invention will be described. The present embodiment is different from the first embodiment with respect to the extraction processing method. In this embodiment, the basic configuration of the imaging apparatus is the same as that of the imaging apparatus 200 of Embodiment 1 described with reference to FIG. Further, the image processing method of the present embodiment is different from the first embodiment only in the processing flow and the calculation method, and the result is the same.

  In the present embodiment, processing for extracting and reducing edge components in the first unnecessary component is added in the first half of the extraction processing of the first embodiment. As described in the first embodiment, the subject parallax component is the amount of parallax shift, and the amount of shift is determined according to the distance from the subject in focus. When the subject is close to the focused subject, the amount of parallax deviation is small, and the subject parallax component appearing in the first unnecessary component is linear. On the other hand, when the subject is far from the focused subject, the amount of parallax deviation is large, and the subject parallax component appearing in the first unnecessary component has a planar shape with a little width.

  On the other hand, unnecessary components such as ghosts are often large planes, and thus subject parallax components and unnecessary components often have characteristic differences in their shapes. For this reason, if edge components such as a linear subject parallax component are reduced before performing the smoothing process in step S120 described with reference to FIG. 8 in the first embodiment, the subsequent smoothing process, etc. This is preferable because the load can be reduced. In this embodiment, therefore, processing for extracting and reducing edge components in the first unnecessary component is added.

  With reference to FIG. 10 to FIG. 12, a method for determining an unnecessary component that is an image component that appears in a captured image generated by the imaging apparatus 200 by photoelectric conversion of unnecessary light will be described. 10 to 12 are diagrams illustrating a procedure of the image processing method.

  In the present embodiment, the subject parallax component exists in the blurred subject B. Between the parallax images in FIGS. 10A-1 and 10B-1, the subject B is deviated from side to side. FIGS. 10A-2 and 10B-2 show luminance cross sections of a pair of parallax images along the broken line portions in FIGS. 10A-1 and 10B-1, respectively. The numerical values in the graphs of FIGS. 10A-2 and B-2 are the luminance value Y of the unnecessary component. For example, in FIG. 10A-2, the background luminance value is 50, and the luminance value of the subject B having a parallax shift is 100. FIGS. 10A-3 and 10B-3 show luminance cross sections of a pair of parallax images taken along the alternate long and short dash line in FIGS. 10A-1 and 10B-1, respectively. The numerical values in the graphs of FIGS. 10A-3 and B-3 are the luminance values Y of unnecessary components. For example, in FIG. 10A-3, the luminance value of the subject A is 100, the luminance value of the unnecessary component A is 160, and the luminance value of the unnecessary component B is 130.

  FIG. 10C-1 shows an image (parallax composite image) obtained by adding and synthesizing FIGS. 10A-1 and 10B-1. FIGS. 10C-2 and 10C-3 respectively show the luminance cross sections of the parallax composite image along the broken line portion and the alternate long and short dash line portion in FIG. This parallax composite image is equivalent to a captured image generated by “imaging without pupil division”. In the present embodiment, by adding and synthesizing a pair of parallax images, the brightness is equivalent to that of a captured image generated by “imaging without pupil division”. You may use the imaging device which becomes the brightness equivalent to the picked-up image produced | generated by "the imaging which does not perform pupil division" by carrying out the averaging of a pair of parallax image. Such a case will be described later in a fifth embodiment.

  FIG. 10D-1 relates to a pair of parallax images, an image in a state where the image of FIG. 10B-1 is subtracted from FIG. 10A-1, and FIG. 10B-1 to FIG. An image (difference synthesized image, first unnecessary component image) obtained by adding and synthesizing an image obtained by subtracting the image of (A-1) is shown. 10D-2 and 10D-3 illustrate luminance cross sections along each of the broken line portion and the alternate long and short dash line portion in FIG. 10D-1 is an image obtained by removing the subject and the background from FIG. 10C-1, and the unnecessary component and the subject included in FIG. Contains only the parallax component.

  In this way, by performing the difference calculation for each parallax image, only the unnecessary component remains (in other words, separated or extracted), and the first unnecessary component (first unnecessary component image) can be determined. However, when the subject parallax component exists as in the present embodiment, the unnecessary component and the subject parallax component are mixed in the first unnecessary component image. For this reason, it is necessary to reduce or remove the subject parallax component in order to determine the unnecessary component with high accuracy. Therefore, in this embodiment, the subject parallax component is reduced from the first unnecessary component.

  Next, an image processing method (subject parallax component reduction process) in the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart showing an image processing method. Each step in FIG. 9 is executed by the system controller 210 or the image processing unit 204 in accordance with an image processing program as a computer program.

  Steps S201 to S207 in FIG. 9 are the same as steps S101 to S107 in the first embodiment described with reference to FIG. Further, step S208 (S220 to S225) in FIG. 9 is the same as step S108 (S120 to S125) in the first embodiment described with reference to FIG. For this reason, the description regarding these steps is omitted. The present embodiment is different from the first embodiment in that steps S210 to S213 are inserted between steps S204 and S220 (steps S210 to S213 are added to the extraction process of the first embodiment). .

  In step S210, the image processing unit 204 (extraction processing unit 204b) performs a correction process for reducing or removing unnecessary components from the parallax composite image, and generates an unnecessary component reduced image (first unnecessary component reduced image). FIG. 10E-1 shows the first unnecessary component reduced image. FIGS. 10E-2 and 10E-3 show luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. Specifically, the extraction processing unit 204b can reduce or remove the first unnecessary component by subtracting the image of FIG. 10D-1 from the image of FIG. 10C-1. When this processing is performed, the subject parallax component is also removed, so that the shape of the round subject B having a parallax shift changes (in this example, the subject B is reduced by the amount of the parallax shift).

  Subsequently, in step S211, the image processing unit 204 (extraction processing unit 204b) adds blur to the first unnecessary component reduced image to generate a blur image. FIG. 10F-1 shows a blurred image. FIGS. 10F-2 and F-3 show luminance cross sections along broken line portions and dashed-dotted line portions in FIG. Specifically, the extraction processing unit 204b performs filter processing using a filter having a blurring effect on the first unnecessary component reduced image. In the present embodiment, for example, a Gaussian filter capable of adjusting the blur amount is used. When the blur amount of the subject B is known, the standard deviation of the Gaussian distribution may be set to be sufficiently larger than the subject parallax component in the filter coefficient as the blur amount. On the other hand, the blurring amount can be determined by referring to photographing conditions such as the focal length of the optical system 201 and the aperture value Fno obtained from the state detection unit 207. The blur amount is an amount determined according to a parameter for controlling the intensity of blur, such as a standard deviation amount of a Gaussian distribution, or a parameter for controlling a blur range by a filter size. Since the amount of parallax deviation of the subject (corresponding to the region size of the subject parallax component) differs according to the focal length and aperture value Fno of the optical system 201, an appropriate blur amount can be determined by referring to the shooting conditions. It is possible to reduce the filter size.

  In the present embodiment, the same blur amount is set for the entire screen. However, if the subject distance information is known, the blur amount may be changed according to the distance for each subject. Since the amount of parallax deviation of the subject varies depending on the subject distance, the blur amount appropriate for each subject can be determined by changing the blur amount according to the distance for each subject, and the filter size can be reduced. It becomes possible. By reducing the filter size, the processing load of the blur addition process can be reduced. Here, regarding subject distance information, for example, corresponding points of a subject are extracted by a block matching method generally known for a pair of parallax images, and the amount of parallax is calculated as a pixel position difference of corresponding point pixels between parallax images. Can be calculated. Then, based on the calculated parallax amount, the focal length of the imaging system, which is known information, and the baseline length data obtained by pupil division, distance information (subject distance information) with respect to the photographic subject can be calculated. In this embodiment, such a subject distance information calculation step may be added. In this embodiment, the blur amount (parallax deviation amount) of the subject B is unknown, and the standard deviation of the Gaussian distribution is set to be smaller than the subject parallax component in the filter coefficient as the blur amount.

  Subsequently, in step S212, the image processing unit 204 (extraction processing unit 204b) obtains a difference between the blurred image generated in step S211 and the first unnecessary component reduced image generated in step S210. Generate an image. FIG. 10G-1 shows a blurred difference image. FIGS. 10G-2 and G-3 illustrate luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. Thereby, the edge portion of the first unnecessary component reduced image can be extracted. Subsequently, in step S213, the image processing unit 204 (extraction processing unit 204b) determines an edge reduced image based on a difference between the first unnecessary component image determined in step S204 and the blur difference image generated in step S212. Is generated.

  Here, referring to FIG. 12, a blur image is generated by adding blur to the first unnecessary component reduced image, and an edge is extracted by obtaining a difference between the blur image and the first unnecessary component reduced image. Explain why you can. 12A-1 to 12H-1 are views focusing on the subject B having a parallax shift, and unnecessary components are not drawn in each drawing. 12A to 12H are diagrams focusing on the unnecessary component B, and the subject parallax component is not drawn in each diagram. FIG. 12 corresponds to a luminance cross-sectional view of an image, where the vertical axis represents luminance and the horizontal axis represents coordinates.

  FIGS. 12A-1 and 12B-1 and FIGS. 12A-2 and 12B-2 each show a pair of parallax images (first parallax image and second parallax image). . In particular, FIGS. 12A-1 and 12B-1 illustrate a state in which the subject B is shifted in parallax. 12 (C-1) and (C-2) show the parallax composite image of FIG. 10 (C-1). FIG. 12D-1 shows the first unnecessary component image (first unnecessary component) determined in step S204 of FIG. FIG. 12E-1 shows the first unnecessary component reduced image generated in step S210. FIG. 12E-1 corresponds to the difference between FIG. 12C-1 and FIG. 12D-1. FIG. 12 (F-1) shows the blurred image generated in step S211. Here, for ease of understanding, a blur amount different from the state of FIG. 10F-1 is given. FIG. 12G-1 shows the blurred difference image generated in step S212. FIG. 12G-1 corresponds to the difference between FIG. 12F-1 and FIG. In addition, the negative value at the time of calculating | requiring a difference is substituted by the zero value, and the following is also the same. FIG. 12 (H-1) shows the edge reduced image generated in step S213. FIG. 12H-1 corresponds to the difference between FIG. 12D-1 and FIG. 12G-1. Thereby, the luminance value of the subject parallax component of the first unnecessary component image can be reduced. 12 (D-2) to (H-2) show the results of performing the same processing as that of FIGS. 12 (D-1) to (H-1) on the unnecessary component B, respectively.

  FIG. 10H-1 shows the result of such edge reduction processing. FIGS. 10H-2 and 10H-3 show luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG.

  In the present embodiment, the edge reduction image is generated using the blur difference image. However, the binarization process described in the first embodiment is performed on the blur difference image, and the binarized blur difference image is converted into the binarized blur difference image. Based on this, masking processing may be performed on the first unnecessary component image. According to the present embodiment, by adding steps S210 to S213 as extraction processing, the luminance of the subject parallax component can be reduced without substantially affecting the luminance of the first unnecessary component.

  The subsequent steps S220 to S225 are the same as steps S120 to S125 of the first embodiment described with reference to FIG. 8, and thus detailed description thereof is omitted. FIGS. 10 (I-1) to (L-1) and FIG. 11 (A-1) correspond to the diagrams obtained by the processes of steps S220 to S225, respectively. 10 (I-2) and (I-3) show luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 10 (I-1), respectively. 10 (J-2) and (J-3) show luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 10 (J-1), respectively. FIGS. 10K-2 and 10K-3 illustrate luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. FIGS. 10 (L-2) and (L-3) show luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 10 (L-1). 11A-2 and 11A-3 illustrate luminance cross sections along each of the broken line portion and the alternate long and short dash line portion in FIG.

  Next, in step S206, a second unnecessary component is determined based on the first unnecessary component and the first unnecessary component region. Specifically, the first unnecessary component region is masked with respect to the first unnecessary component, and only a portion including an unnecessary component such as a ghost is extracted as the second unnecessary component. FIG. 11B-1 shows the second unnecessary component (second unnecessary component image). 11B-2 and 11B-3 show luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG.

  Finally, in step S207, the image processing unit 204 (unnecessary component reduction unit 204d) reduces unnecessary components based on the second unnecessary component determined in step S206 and the parallax composite image generated in step S205. Process. Specifically, the unnecessary component reduction unit 204d subtracts the second unnecessary component (second unnecessary component image) in FIG. 11 (B-1) from the parallax composite image in FIG. 10 (C-1). Thereby, an unnecessary component reduced image (second unnecessary component reduced image) is generated. FIG. 11C-1 shows a second unnecessary component reduced image. 11C-2 and 11C-3 illustrate luminance cross sections along each of the broken line portion and the alternate long and short dash line portion in FIG.

  According to the present embodiment, it is possible to determine the second unnecessary component in which the subject parallax component is reduced from the first unnecessary component including the subject parallax component. Moreover, according to the present Example, the unnecessary component reduced image (2nd unnecessary component reduced image) which reduced the 2nd unnecessary component from the parallax synthetic | combination image can be produced | generated.

  In the present embodiment, in order to reduce the noise component and the offset component, the luminance value of each coordinate of the edge extraction result in step S212 is multiplied by a coefficient, or the edge width is expanded to further enhance the edge. The edge reduced image of step S213 may be generated.

  Next, Embodiment 3 of the present invention will be described. The present embodiment is different from the first and second embodiments with respect to the extraction processing method. In this embodiment, the basic configuration of the imaging apparatus is the same as that of the imaging apparatus 200 of Embodiment 1 described with reference to FIG. Further, the image processing method of the present embodiment is different from the first embodiment only in the processing flow and the calculation method, and the result is the same.

  The image processing method (subject parallax component reduction processing) in the present embodiment will be described with reference to FIGS. FIG. 13 is a flowchart showing an image processing method. Each step in FIG. 13 is executed by the system controller 210 or the image processing unit 204 in accordance with an image processing program as a computer program. 14 and 15 are diagrams illustrating the procedure of the image processing method.

  Steps S301 to S307 in FIG. 13 are the same as steps S201 to S207 in the second embodiment described with reference to FIG. Further, steps S311 to S325 in FIG. 13 are the same as steps S211 to S225 in the second embodiment described with reference to FIG. For this reason, the description regarding these steps is omitted. The present embodiment is different from the second embodiment in that there is no step corresponding to step S210 and a blur difference image is generated by adding blur directly to the first unnecessary component (first unnecessary component image). In the present embodiment, since there is no step corresponding to step S210, the first unnecessary component reduced image is not generated as in the second embodiment. However, in order to facilitate the comparison with the second embodiment, the unnecessary component reduced image generated in the last step S307 is referred to as a “second unnecessary component reduced image”. 14A-1 to 14D correspond to FIGS. 10A-1 to 10D-1 of the second embodiment, respectively.

  Referring to FIG. 15, the step corresponding to step S210 of the second embodiment is eliminated, and the subject parallax component is generated by adding the blur directly to the first unnecessary component (first unnecessary component image) to generate the blur difference image. The reason why can be reduced will be described. 15A to 15D correspond to FIGS. 12A-1 to 12D-1 of the second embodiment, respectively. In the second embodiment, the first unnecessary component reduced image is generated as shown in FIG. 12E-1, but in this embodiment, blur is added to the first unnecessary component image in FIG. The blurred image in FIG. 15E is generated. Although the blur addition method of the present embodiment uses Gaussian filter processing as in the second embodiment, other blur addition methods may be used.

  The present embodiment is different from the second embodiment in the method for generating the blurred difference image. In the present embodiment, the blur difference image is generated by subtracting the blur image generated in step S311 from the first unnecessary component image determined in step S304. At this time, the direction in which the difference is taken is opposite to that in the second embodiment. Specifically, the blurred image in FIG. 15E is subtracted from the first unnecessary component image in FIG. As a result, a blur difference image as shown in FIG. 15F is obtained. Further, an edge reduced image is obtained by subtracting the blur difference image from the first unnecessary component image. Specifically, the blur difference image in FIG. 15F is subtracted from the first unnecessary component image in FIG. As a result, an edge reduced image as shown in FIG. 15G is obtained. As a result, the edge portion of the first unnecessary component image can be reduced.

  14 (E-1), (F-1), and (G-1) correspond to FIGS. 15 (E), (F), and (G), respectively, and the processes in steps S311, S312 and S313 in FIG. These images are obtained respectively. 14E-2 and 14E-3 are luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 14 (F-2) and 14 (F-3) are luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 14 (F-1). 14 (G-2) and (G-3) are luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 14 (G-1), respectively.

  Subsequent steps S308 (S320 to S325) are the same as steps S208 (S220 to S225) of the second embodiment described with reference to FIG. 9, and thus detailed description thereof is omitted. 14 (H-1) to (M-1) are images obtained by the processes of steps S320 to S325, respectively. 14 (H-2) and (H-3) are luminance cross sections along the broken line portion and the alternate long and short dash line portion in FIG. 14 (H-1). 14 (I-2), (I-3), FIG. 14 (J-1), (J-2), FIG. 14 (K-1), (K-2), FIG. 14 (L-2), Similarly, (L-3) is a luminance cross section corresponding to each of FIGS. 14 (I-1) to (L-1).

  Next, in step S306, a second unnecessary component is determined based on the first unnecessary component and the first unnecessary component region. Specifically, the first unnecessary component region is masked with respect to the first unnecessary component, and only a portion including an unnecessary component such as a ghost is extracted as the second unnecessary component.

  Finally, in step S307, the image processing unit 204 (unnecessary component reduction unit 204d) reduces unnecessary components based on the second unnecessary component determined in step S306 and the parallax composite image generated in step S305. Process. Specifically, the unnecessary component reduction unit 204d subtracts the second unnecessary component (second unnecessary component image) in FIG. 14 (M-1) from the parallax composite image in FIG. 14 (C-1). Thereby, an unnecessary component reduced image (second unnecessary component reduced image) as shown in FIG. 14 (N-1) is generated.

  According to the present embodiment, it is possible to determine the second unnecessary component in which the subject parallax component is reduced from the first unnecessary component including the subject parallax component. Moreover, according to the present Example, the unnecessary component reduced image (2nd unnecessary component reduced image) which reduced the 2nd unnecessary component from the parallax synthetic | combination image can be produced | generated.

  Next, a fourth embodiment of the present invention will be described. The present embodiment is different from Embodiments 1 to 3 in that the result of the extraction process is displayed on the display unit 205 (display means). In this embodiment, the basic configuration of the imaging apparatus is the same as that of the imaging apparatus 200 of Embodiment 1 described with reference to FIG. In addition, as the extraction process in the image processing method of the present embodiment, the same extraction process as in the first to third embodiments can be applied.

  The image processing method (subject parallax component reduction processing and unnecessary component extraction result display processing) in the present embodiment will be described with reference to FIGS. FIG. 16 is a flowchart showing an image processing method. Each step in FIG. 16 is executed by the system controller 210, the image processing unit 204, or the display unit 205 in accordance with an image processing program as a computer program. FIG. 17 is an example of an output image obtained by the image processing method.

  Steps S401 to S405 in FIG. 16 are the same as steps S101 to S105 in the first embodiment, respectively. Further, step S108 (S120 to S125) of the first embodiment can be applied as step S408 of the extraction process. Instead of this, step S208 (S210 to S225) of the second embodiment or step S308 (S311 to S325) of the third embodiment may be applied. In this embodiment, the second unnecessary component region is determined in step S450 after the first unnecessary component region is determined, and the display unit 205 displays the second unnecessary component region together with the parallax composite image in step S451. , Different from Examples 1-3.

  In step S450, the image processing unit 204 (extraction processing unit 204b) determines a second unnecessary component region (an unnecessary component region displayed on the display unit 205) based on the first unnecessary component region determined in step S408. decide. The simplest method for determining the second unnecessary component region is to use the first unnecessary component region as it is as the second unnecessary component region. Specifically, for example, the regions where the luminance values in FIGS. 1 (I-1), 11 (A-1), and 14 (L-1) of Examples 1 to 3 are 255 are not necessary. The component area. Thereby, a 2nd unnecessary component area | region can be determined easily. Alternatively, the result of processing the first unnecessary component area so that the user can easily discriminate the result when the first unnecessary component area is expanded or reduced, for example, when the first unnecessary component area is displayed on the display unit 205 in the subsequent step. May be determined as

  Subsequently, in step S451, the system controller 210 and the image processing unit 204 display the second unnecessary component region determined in step S450 on the display unit 205 together with the parallax composite image generated in step S405. Specifically, as shown in FIG. 17A, the second unnecessary component region is synthesized on the parallax composite image and displayed on the display unit 205. The method for displaying the second unnecessary component region can be performed by, for example, coloring or blinking the region or displaying the background parallax composite image in a watermark so as to be easily understood by the user. For example, as shown in FIG. 17B, immediately after shooting, the image may be displayed on the rear display 20 (display unit) of the imaging apparatus 200 so that the user can check.

  According to the present embodiment, the unnecessary component region (second unnecessary component region) is displayed on the display unit together with the parallax composite image, thereby allowing the user to recognize that an unnecessary component such as a ghost is generated. is there. In this embodiment, it is sufficient that the second unnecessary component region is configured to be displayed, and it is essential to determine the second unnecessary component obtained by reducing the subject parallax component from the first unnecessary component. is not.

  Next, Example 5 (plural pupil division) of the present invention will be described. In this embodiment, the number of parallaxes is different from those of the first to fourth embodiments. In the image processing method of this embodiment, the processing steps up to the step of generating the parallax image and determining the first unnecessary component are different from those of the above-described embodiments, but the subsequent processing steps are the same as those of the embodiments. Therefore, the description thereof is omitted. In this embodiment, the basic configuration of the imaging apparatus is the same as that of the imaging apparatus 200 of Embodiment 1 described with reference to FIG.

  FIG. 18 is a diagram illustrating an image sensor (light receiving unit) in the present embodiment. In FIG. 18, ML is a microlens, G1, G2, G3, and G4 are light receiving portions (pixels), and each pixel forms a pair. A plurality of pixel sets G1, G2, G3, and G4 are arranged on the image sensor. The pixel group has a conjugate relationship with the exit pupil EXP through a common microlens ML (that is, one pixel group is provided for each pixel group). In this embodiment, when an image equivalent to a captured image generated by “imaging without pupil division” is output, the signals obtained from the four pixel sets G1, G2, G3, and G4 are added and averaged. By doing so, one signal value is generated. In this embodiment, a specific configuration example of the optical system is the same as that of the optical system 201 of Embodiment 1 described with reference to FIG.

  FIG. 19 shows regions P1, P2, P3, and P4 (pupil regions or pupil division regions) through which light beams incident on the pixels G1, G2, G3, and G4 shown in FIG. The stop STP can be considered as corresponding to the exit pupil EXP of the optical system 201 (virtual image viewed from the image plane position of the optical system 201), but actually the stop STP and the exit pupil EXP are different from each other. When the light beam from the high-intensity object (sun SUN) passes through the stop STP and enters each pixel, it is divided into regions P1, P2, P3, and P4 (pupil regions).

  Next, with reference to FIGS. 20 and 21, a method for determining an unnecessary component that is an image component that appears in a captured image generated by the imaging apparatus 200 as a result of photoelectric conversion of unnecessary light will be described. 20 and 21 are diagrams showing the procedure of the image processing method in the present embodiment.

  20 (A-1), (B-1), (C-1), and (D-1) respectively show the luminous fluxes that have passed through the regions P1, P2, P3, and P4 as pixel groups G1, G2, G3, A set of parallax images obtained as a result of photoelectric conversion in G4 is shown. The set of parallax images includes a triangular subject A, a round subject B, and an unnecessary component A and an unnecessary component B, which are schematically shown as squares. In these parallax images, the subject A is in focus and the subject B is located behind the subject A, so that the subject A is shifted by a predetermined amount of parallax. Each unnecessary component between parallax images exists in the same position in FIGS. 20A-1, B-1, C- 1, and D- 1, and their luminances are different from each other. ing.

  20E-1 is synthesized by performing an averaging process on the parallax images shown in FIGS. 20A-1, (B-1), (C-1), and (D-1). It is an image. Specifically, the luminance values of the coordinates in FIGS. 20A-1, B-1, C- 1, and D- 1 are added and divided by 4 to obtain FIG. The luminance value of each coordinate of 1) is calculated. The image in FIG. 20E-1 is equivalent to a captured image generated by “imaging without pupil division” in the imaging apparatus of the present embodiment. In addition, the image in FIG. 20E-1 is a display image displayed on the display unit 205 so that the user can confirm the captured image.

  21 (A-1), (B-1), and (C-1) show a set of parallax images, with FIG. 20 (A-1) as a reference image, and FIGS. C-1) is a difference image obtained by subtracting the images of (D-1). Similar to the first embodiment, these difference images include unnecessary components as difference information. Further, as in the first embodiment, there is a portion where unnecessary components included in FIGS. 21A-1, B-1 and C-1 are calculated as negative values by difference calculation. However, in order to simplify the unnecessary component reduction process in the latter stage, negative values are rounded down and replaced with 0. The same applies to all other difference images. FIG. 21D-1 is the difference information acquired as two-dimensional data, and the difference at each pixel position in the difference image of FIGS. 21A-1, (B-1), and (C-1). This is information obtained by extracting the maximum value between information (difference maximum value information or difference maximum value image).

  21 (A-2), (B-2), and (C-2) show a set of parallax images with FIG. 20 (B-1) as a reference image. C-1) is a difference image obtained by subtracting the images of (D-1). FIG. 21D-2 shows the difference information acquired as two-dimensional data, and the difference at each pixel position in the difference image of FIGS. 21A-2, B-2, and C-2. It is the difference maximum value information between information.

  21 (A-3), (B-3), and (C-3) show a set of parallax images with FIG. 20 (C-1) as a reference image. B-1) is a difference image obtained by subtracting the images of (D-1). FIG. 21 (D-3) is the difference information acquired as two-dimensional data, and the difference at each pixel position in the difference image of FIGS. 21 (A-3), (B-3), and (C-3). It is the difference maximum value information between information.

  21 (A-4), (B-4), and (C-4) show a set of parallax images with reference to FIG. 20 (D-1) as a reference image. B-1) is a difference image obtained by subtracting the images of (C-1). FIG. 21 (D-4) shows the difference information at each pixel position in the difference image of FIGS. 21 (A-4), (B-4), and (C-4), which is difference information acquired as two-dimensional data. It is the difference maximum value information between information. The difference maximum value information is a result of extracting an unnecessary component from each parallax image.

  Here, as described in the first to fourth embodiments, consider the case where the first unnecessary component (the first unnecessary component image obtained by imaging the first unnecessary component) is determined. At this time, as described above, an unnecessary component is extracted for each parallax image as the difference maximum value information. Therefore, it is considered that each difference maximum value information corresponds to the first unnecessary component image as one method. However, it is necessary to perform the subsequent processing as images for the number of parallax images, resulting in a complicated processing process. Therefore, in this embodiment, the subsequent processing is simplified by combining the maximum difference information into one. Specifically, addition averaging processing is performed on FIGS. 21D-1, D-2, D-3, and D-4 to synthesize these images. FIG. 21 (F-1) shows the synthesis result.

  In this embodiment, minimum value information between difference information at each pixel position in the difference image may be extracted. FIG. 21 (E-1) shows the difference information at each pixel position in the difference image of FIGS. 21 (A-1), (B-1), and (C-1), which is difference information acquired as two-dimensional data. It is information (difference minimum value information or difference minimum value image) obtained by extracting the minimum value between information. FIG. 21E-2 shows the difference information acquired as two-dimensional data, and the difference at each pixel position in the difference image of FIGS. 21A-2, B-2, and C-2. It is the difference minimum value information between information. FIG. 21E-3 shows the difference information at each pixel position in the difference image of FIGS. 21A-3, B-3, and C-3, which is difference information acquired as two-dimensional data. It is the difference minimum value information between information. FIG. 21E-4 is the difference information acquired as two-dimensional data. The difference at each pixel position in the difference image of FIGS. 21A-4, B-4, and C-4. It is the difference minimum value information between information. FIG. 21 (G-1) shows an addition averaging process for FIGS. 21 (E-1), (E-2), (E-3), and (E-4) in order to simplify the subsequent processing. This is the result of combining each difference minimum value information into one.

  The minimum difference information is a result of extracting unnecessary components from each parallax image. When the reduction process is performed based on the difference minimum value information, the amount of reduction of the unnecessary component is reduced as compared with the case of using the difference maximum value information, but the influence of the subject parallax component is also reduced. For this reason, the subject parallax component can be easily reduced by applying each processing flow of the first to fourth embodiments. Since the subsequent processing flows can be applied to the flows of the first to fourth embodiments, description thereof is omitted here. Thus, even when the number of parallaxes increases, the “parallax composite image” and the “first unnecessary component image” can be calculated. Since the subsequent processing flow and basic handling are the same as in the first to fourth embodiments, their details are omitted.

  According to the present embodiment, an unnecessary component formed by unnecessary light (ghost or flare) can be determined from a difference image based on a plurality of parallax images obtained by one imaging. That is, it is possible to determine an unnecessary component included in a captured image without performing multiple imaging. Moreover, according to the present Example, the unnecessary component reduced image which reduced the unnecessary component from the parallax composite image can be obtained. In this embodiment, an example of a gray scale image is shown for the sake of simplicity of explanation, but a color image can be similarly applied. In this case, the above-described processing is performed independently for each color channel, and the respective colors are finally combined into one image.

  Next, a sixth embodiment of the present invention will be described. Ren. "Plenoptic Camera" has been proposed in "Light Field Photographic with a Hand-Held Plenoptic Camera" (Stanford Tech Report CTSR 2005-2) of Ng et al. By using the method “Light Field Photography” in “Plenoptic Camera”, it is possible to capture information on the position and angle of the light beam from the object side.

  FIG. 22 is a diagram illustrating an imaging system of the imaging apparatus according to the present embodiment, and illustrates a configuration of an imaging system of “Plenoptic Camera”. The optical system 301 includes a main lens (photographing lens) 301b and an aperture stop 301a. A microlens array 301c is disposed at the imaging position of the optical system 301, and an imaging element 302 is disposed behind (on the image side) of the microlens array 301c. The microlens array 301c serves as a separator (separating means) so that a ray group passing through a certain point in the subject space such as the point A and a ray passing through a point near the point A are not mixed on the image sensor 302. It has a function. As can be seen from FIG. 22, the upper line, chief ray and underline from point A are received by different pixels. For this reason, a group of rays passing through the point A can be obtained separately for each angle of rays.

  Also, “Full Resolution Light Field Rendering” (Adobe Technical Report January 2008) by Todor George et al. Is known. This document proposes an imaging system shown in FIG. 23 and FIG. 24 as a method of acquiring light position and angle information (Light Field).

  In the configuration of the imaging system shown in FIG. 23, the microlens array 301c is arranged behind (image side) the imaging position of the main lens 301b, and the light ray group passing through the point A is re-imaged on the imaging element 302. Thus, the light beam group can be obtained separately for each light beam angle. In the configuration of the imaging system shown in FIG. 24, the microlens array 301c is arranged in front of the imaging position of the main lens 301b (on the object side), and a light beam passing through the point A is imaged on the imaging device 302. By doing so, the light beam group can be obtained separately for each angle of the light beam. Both configurations are the same in that the light beam passing through the pupil of the optical system 301 is divided according to the passing region (passing position) in the pupil. In these configurations, as shown in FIG. 25, the image sensor 302 includes one microlens ML (different from the microlens array 301c) and one light receiving unit G1 in pairs via the color filter CF. A conventional imaging device can be used.

  When the optical system 301 shown in FIG. 22 is used, an image as shown in FIG. 26A is obtained. FIG. 26B shows an enlarged view of one of many circles arranged in FIG. One circle corresponds to the stop STP, and the inside thereof is divided by a plurality of pixels Pj (j = 1, 2, 3,...). Thereby, the intensity distribution of the pupil is obtained within one circle. Further, when the optical system 301 shown in FIGS. 23 and 24 is used, a parallax image as shown in FIG. 27 is obtained. In the image shown in FIG. 26A, a plurality of parallax images as shown in FIG. 27 can also be obtained by rearranging a plurality of pixels Pj in each circle (aperture STP) side by side.

  As described in the first to fifth embodiments, unnecessary light such as ghost passes through the pupil with a bias in the pupil. For this reason, an unnecessary component can be determined by using the image processing method demonstrated in Examples 1-5 in the imaging device which divides | segments and images a pupil like a present Example.

  As another example, a parallax image can be obtained even when the same subject is imaged using a plurality of cameras as shown in FIG. For this reason, the image processing method described in the first to fifth embodiments can be used in such a plurality of cameras. C1, C2, and C3 are actually separate imaging devices, but can be regarded as an integrated imaging device that divides a large pupil into three images. In addition, as shown in FIG. 29, pupil division can be performed by providing a plurality of optical systems OSj (j = 1, 2, 3,...) In one imaging apparatus. Also in the imaging apparatus of the present embodiment, the second unnecessary component obtained by removing or reducing the subject parallax component from the first unnecessary component can be determined. In addition, an unnecessary component reduced image in which the second unnecessary component is reduced can be obtained from the parallax composite image.

  In each of the above embodiments, the imaging apparatus that executes the image processing method has been described. However, the image processing method described in each embodiment can also be executed by an image processing program installed in a computer device (image processing apparatus). . This case will be described in Example 7 to be described later as an imaging system.

  Next, with reference to FIG. 30, the imaging system in Example 7 of this invention is demonstrated. FIG. 30 is a configuration diagram of an imaging system 700 in the present embodiment. As illustrated in FIG. 30, the imaging system 700 includes an imaging device 701, an image processing device 702, and a display device 703.

  The imaging device 701 corresponds to the imaging device 200 of each of the above-described embodiments. The image processing apparatus 702 is a computer device that executes the image processing method of each embodiment. The image processing apparatus 702 has a communication unit (not shown). The communication unit receives a captured image (parallax image) from the imaging device 701. The connection between the image processing apparatus 702 and the imaging apparatus 701 via the communication unit may use either a wired or wireless connection method.

  The display image calculated based on the parallax image by the image processing device 702 or the display image obtained by adding a part of the mask image to the parallax image is stored in a storage unit (not shown) provided in the image processing device 702. Or displayed on the display device 703. The display device 703 is, for example, a liquid crystal display or a projector. The user can perform work while confirming an image in the middle of image processing via the display device 703. The user may specify the first unnecessary component region and the second unnecessary component region during this work. Accordingly, the user can check information regarding whether or not an unnecessary component is included in the captured image via the display device 703.

  As described above, in each embodiment, the image processing apparatus (image processing unit 204) includes the first determination unit (unnecessary component detection unit 204a) and the second determination unit (extraction processing unit 204b). The first determination unit determines a first unnecessary component based on difference information (for example, a difference image or a difference composite image) of a plurality of parallax images. The second determining means determines the second unnecessary component by reducing the parallax component (subject parallax component) from the first unnecessary component.

  Preferably, the second determining means determines the second unnecessary component based on the first unnecessary component and a mask (first unnecessary component region) created based on the first unnecessary component. More preferably, the second determining means determines the second unnecessary component by creating a mask by changing the luminance value of the first unnecessary component and performing a masking process using the mask on the first unnecessary component. To do. More preferably, the second determining unit changes the luminance value by performing at least one of a smoothing process, a contrast changing process (contrast increasing process), and a threshold process on the first unnecessary component (S120). , S121, S122). More preferably, the second determining means performs binarization processing on the first unnecessary component after performing at least one of smoothing processing, contrast changing processing, and threshold processing (S123). More preferably, the second determination unit performs an enlargement process on the first unnecessary component after the binarization process (S124). More preferably, the second determination unit creates a mask based on the first unnecessary component after the enlargement process (S125).

  Preferably, the second determination unit generates an unnecessary component reduced image (first unnecessary component reduced image) based on the parallax image and the first unnecessary component (S210). The second determining means determines an edge component based on the difference between the blurred image obtained by adding blur to the unnecessary component reduced image and the unnecessary component reduced image (S211 and S212). Then, the second determination unit reduces the edge component from the first unnecessary component (S213).

  Preferably, the second determination unit generates a first unnecessary component image based on the first unnecessary component (S304). The second determining unit determines an edge component based on the difference between the blurred image obtained by adding blur to the first unnecessary component image and the first unnecessary component image (S311 and S312). Then, the second determination unit reduces the edge component from the first unnecessary component (S313).

  Preferably, the image processing apparatus includes a reduction unit (unnecessary component reduction unit 204c) that reduces a second unnecessary component from an image based on a plurality of parallax images. More preferably, the image based on the plurality of parallax images is a parallax composite image obtained by combining the plurality of parallax images.

  Preferably, the imaging apparatus includes display means (display unit 205, rear display 20) for displaying information on a mask created based on the first unnecessary component in order to determine the second unnecessary component.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  According to each embodiment, the first unnecessary component of the photographed image is determined without performing a plurality of imaging, and the second unnecessary component in which the subject parallax component is reduced from the first unnecessary component can be determined. An apparatus, an image processing method, an image processing program, and a storage medium can be provided.

  As mentioned above, although the preferable Example of this invention was described, this invention is not limited to these Examples, A various deformation | transformation and change are possible within the range of the summary.

204 Image processing unit (image processing apparatus)
204a Unnecessary component detection unit (first determination means)
204b Extraction processing unit (second determining means)

Claims (17)

First determining means for determining a first unnecessary component based on difference information of a plurality of parallax images;
Have a, a second determining means for determining a second required component by reducing the parallax component from the first unwanted components,
The image processing apparatus according to claim 2, wherein the second determining unit determines the second unnecessary component based on the first unnecessary component and a mask created based on the first unnecessary component . The second determining means includes
Changing the luminance value of the first unnecessary component to create the mask;
Wherein by performing the masking process using the mask with respect to the first unwanted components, the image processing apparatus according to claim 1, characterized in that determining the second required component. Said second determination means, smoothing process on the first unnecessary components, contrast changing process, and, by performing at least one thresholded, claim 2, characterized in that for changing the luminance value An image processing apparatus according to 1. The second determination unit performs binarization processing on the first unnecessary component after performing at least one of the smoothing processing, the contrast change processing, and the threshold processing. The image processing apparatus according to claim 3 . The image processing apparatus according to claim 4 , wherein the second determination unit performs an enlargement process on the first unnecessary component after the binarization process. The image processing apparatus according to claim 5 , wherein the second determination unit creates the mask based on the first unnecessary component after the enlargement process. The second determining means includes
Generating an unnecessary component reduced image based on the parallax image and the first unnecessary component;
Determining an edge component based on a difference between a blur image obtained by adding blur to the unnecessary component reduced image and the unnecessary component reduced image;
The image processing apparatus according to any one of claims 2 to 6, characterized in that to reduce the edge component from the first required component. The second determining means includes
Generating a first unnecessary component image based on the first unnecessary component;
Determining an edge component based on a difference between a blur image obtained by adding blur to the first unnecessary component image and the first unnecessary component image;
The image processing apparatus according to any one of claims 2 to 6, characterized in that to reduce the edge component from the first required component. The image processing apparatus according to any one of claims 1 to 8, further comprising a reduction means for reducing the second unnecessary component from the image based on the plurality of parallax images. The image processing apparatus according to claim 9 , wherein the image based on the plurality of parallax images is a parallax composite image obtained by combining the plurality of parallax images. An imaging means for photoelectrically converting an optical image formed via the optical system and outputting a plurality of parallax images;
First determination means for determining a first unnecessary component based on difference information of the plurality of parallax images;
Have a, a second determining means for determining a second required component by reducing the parallax component from the first unwanted components,
The imaging apparatus according to claim 2, wherein the second determining unit determines the second unnecessary component based on the first unnecessary component and a mask created based on the first unnecessary component . The imaging apparatus according to claim 11, characterized in that it comprises further display means for displaying information about the previous Kemah risk. The plurality of parallax images are a plurality of images generated based on light beams that have passed through different regions of the pupil of the optical system,
The imaging means has a plurality of pixels sharing one microlens,
The imaging device according to claim 11 or 12 , wherein the plurality of pixels are configured to receive light beams that have passed through different regions of the pupil of the optical system. Wherein the plurality of parallax images, the light beam passing through the different areas of the pupil of the optical system, wherein the a plurality of images generated led to different pixels from each other in the image pickup device according to claim 11 or 12. The imaging device according to 12 . Determining a first unnecessary component based on difference information of a plurality of parallax images;
Have a, and determining a second required component by reducing the parallax component from the first unwanted components,
2. The image processing method according to claim 1, wherein the second unnecessary component is determined based on the first unnecessary component and a mask created based on the first unnecessary component . An image processing program configured to cause a computer to execute the image processing method according to claim 15 . A storage medium storing the image processing program according to claim 16 .