JP2011205531A - Stereoscopic imaging apparatus, ghost image processing apparatus and ghost image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To perform processing on a ghost image through one time of single-lens stereoscopic imaging.SOLUTION: A ghost image processing method includes the steps of: acquiring a plurality of viewpoint images generated by an imaging device for performing photoelectric conversion on each of light fluxes which have passed through different areas of a single photographing optical system (S102, S104); calculating an image deviation amount caused by a parallax of a subject image among the plurality of acquired viewpoint images (S106); correcting image deviation among the plurality of viewpoint images based on the calculated image deviation amount (S108); and comparing the plurality of viewpoint images after image deviation correction to detect a ghost image in the plurality of viewpoint images (S110).

Description

  The present invention relates to a stereoscopic imaging device, a ghost image processing device, and a ghost image processing method, and in particular, a plurality of viewpoint images (stereoscopic images) generated by photoelectrically converting light beams that have passed through different regions of a single photographing optical system. The present invention relates to a ghost image processing technique for (image).

  When the subject includes a bright light source such as the sun, the light from the light source is reflected in the imaging optical system or on the surface of the image sensor, and an image of the reflected light is formed on the image sensor so that the image is captured. Ghost images may occur.

  Patent Document 1 discloses a configuration in which imaging is performed twice with different F values by changing the aperture diameter of a diaphragm in the photographing optical system, and ghost detection is performed from the difference between the two images.

  Patent Document 2 discloses a configuration in which a focus lens is moved to capture an image at a focus position and a defocus position, and ghost detection is performed from a difference between the focus image and the defocus image.

  Patent Document 3 includes a mechanism for rotating a wavefront conversion element in a plane orthogonal to the optical axis of an optical system in an apparatus that obtains an image with an expanded depth of field by performing image processing on a captured image. In addition, a configuration is disclosed in which ghost removal is performed by performing sum and difference calculation processing between two images acquired when the wavefront conversion element is rotated at rotation angles of 0 degrees and 180 degrees.

JP 2008-228181 A JP 2008-54206 A JP 2005-91865 A

  There is a need for a monocular stereoscopic imaging device that photoelectrically converts light beams that have passed through different areas of a single imaging optical system, and for a stereoscopic image (right-eye image, left-eye image) obtained by a single monocular imaging. There is a demand for reliable ghost image processing.

  However, with the techniques described in the prior art documents related to the conventional imaging device, a ghost image cannot be processed unless imaging is performed a plurality of times under different imaging conditions. In addition, although the prior art document describes that a ghost image is detected from a difference between two images having different shooting conditions, a plurality of viewpoint images (right-eye image, left-eye image) obtained by one monocular three-dimensional imaging are described. Since there is parallax between (images), a problem of false detection occurs when the conventional technique is applied to a plurality of viewpoint images. Therefore, it is difficult to perform the ghost image processing by single monocular three-dimensional imaging by applying the technique described in the prior art document.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide a stereoscopic imaging apparatus, a ghost image processing apparatus, and a ghost image processing method capable of performing processing on a ghost image by one stereoscopic imaging. To do.

  In order to achieve the above object, the present invention is an image acquisition means for acquiring a plurality of viewpoint images generated by an image sensor that photoelectrically converts light beams that have passed through different regions of a single photographing optical system. An image shift amount calculation unit that calculates an image shift amount due to parallax of the subject image between the plurality of viewpoint images, and an image shift correction unit that corrects an image shift between the plurality of viewpoint images based on the image shift amount. And a ghost image detecting means for detecting a ghost image in the plurality of viewpoint images by comparing the plurality of viewpoint images after the image shift correction. To do.

  According to the present invention, the amount of image shift due to the parallax of the subject image between the viewpoint images is calculated, the image shift between the viewpoint images is corrected based on the amount of image shift, and the comparison between the viewpoint images after the image shift correction is performed. Since a ghost image in each viewpoint image is detected, it is possible to detect a ghost image from a plurality of viewpoint images obtained by one stereoscopic imaging without performing a plurality of stereoscopic imaging, False detection can be prevented.

  In one embodiment of the present invention, the ghost image detection unit calculates a difference in pixel values between corresponding pixels of the plurality of viewpoint images after the image shift correction, so that the difference image for each corresponding pixel is calculated. The ghost extraction data is generated. That is, by generating ghost extraction data from a plurality of viewpoint images, it is possible to easily verify the generation of a ghost, remove the ghost image, and the like.

  In one embodiment of the present invention, the plurality of viewpoints after the image shift correction is performed by recognizing a portion where the ghost image exists in each of the plurality of viewpoint images based on the polarity of the difference in the ghost extraction data. Ghost image removing means for removing the ghost image from each of the images, and image deviation amount returning means for returning the image deviation amount between the plurality of viewpoint images from which the ghost image has been removed to the amount before the image deviation correction. It is characterized by that. That is, a ghost image can be removed from a plurality of viewpoint images obtained by one stereoscopic imaging.

  In one embodiment of the present invention, the ghost image removing unit subtracts a value of the ghost extraction data from a pixel value of a portion where the ghost image exists in the plurality of viewpoint images after the image shift correction. To do.

  In one embodiment of the present invention, the ghost image removing means calculates a pixel value of a portion where the ghost image of one of the viewpoint images exists among the plurality of viewpoint images after image shift correction of the other viewpoint image. It is characterized in that the pixel value is replaced with a pixel value where no ghost image exists.

  In one embodiment of the present invention, the imaging device includes a first pixel group and a second pixel group separated in a first pupil division direction, and a third and a third pixel separated in a second pupil division direction. 4 pixel groups, and the image acquisition means includes the first pixel group, the second pixel group, the third pixel group, and the fourth pixel group respectively generated by the first pixel group, the second pixel group, the third pixel group, and the fourth pixel group. A viewpoint image, a second viewpoint image, a third viewpoint image, and a fourth viewpoint image are acquired, and the image shift amount calculating means is a first unit based on a parallax between the first viewpoint image and the second viewpoint image. And calculating a second image shift amount due to a parallax between the third viewpoint image and the fourth viewpoint image, and the image shift correction unit is configured to calculate the first image shift amount. An image shift between the first viewpoint image and the second viewpoint image is corrected based on the amount, and the first viewpoint image is corrected. The image shift between the third viewpoint image and the fourth viewpoint image is corrected based on the image shift amount of the first viewpoint image, and the ghost image detecting means is configured to correct the image shift and the second viewpoint image and the second viewpoint image. A ghost image in the plurality of viewpoint images is detected by comparing the third viewpoint image after the image shift correction and the fourth viewpoint image. . That is, it becomes possible to reliably detect a ghost image by ghost light that is incident not only from one direction (for example, the horizontal direction) but also from the other direction (for example, the vertical direction).

  The present invention also provides an image acquisition means for acquiring a plurality of viewpoint images generated by an image sensor that photoelectrically converts light beams that have passed through different regions of a single photographing optical system, and the acquired plurality of viewpoint images. An image shift amount calculation unit that calculates an image shift amount due to parallax of the subject image between the image shift correction unit, an image shift correction unit that corrects an image shift between the plurality of viewpoint images based on the image shift amount, and after the image shift correction Image synthesizing means for generating a synthesized image that does not include a ghost image by comparing brightness between corresponding pixels of the plurality of viewpoint images and extracting only a darker portion. A ghost image processing apparatus is provided. That is, ghost removal can be performed from a plurality of viewpoint images obtained by a single stereoscopic imaging without performing a stereoscopic imaging a plurality of times, and erroneous detection can be prevented. Further, since ghost extraction is not required, it is possible to speed up the ghost removal process.

  The present invention also provides a stereoscopic imaging apparatus comprising the single imaging optical system, the imaging element, and the ghost image processing apparatus.

  The present invention also provides an image acquisition step of acquiring a plurality of viewpoint images generated by an imaging element that photoelectrically converts light beams that have passed through different regions of a single photographing optical system, and the acquired plurality of viewpoint images. An image shift amount calculating step for calculating an image shift amount due to the parallax of the subject image between, an image shift correction step for correcting an image shift between the plurality of viewpoint images based on the image shift amount, and after the image shift correction And a ghost image detection step of detecting a ghost image in the plurality of viewpoint images by comparing the plurality of viewpoint images.

  In addition, the present invention provides an image acquisition step of acquiring a plurality of viewpoint images generated by an image sensor that photoelectrically converts light beams that have passed through different regions of a single photographing optical system, and the acquired viewpoints An image shift amount calculating step for calculating an image shift amount due to parallax of a subject image between images, an image shift correction step for correcting an image shift between the plurality of viewpoint images based on the image shift amount, and an image shift correction. An image composition step of comparing the brightness between corresponding pixels between the plurality of later viewpoint images and extracting only the darker portion to generate a composite image that does not include a ghost image. A ghost image processing method characterized by the above.

  According to the present invention, it is possible to perform processing on a ghost image such as ghost image detection and ghost image removal by one stereoscopic imaging.

Block diagram of stereoscopic imaging device The figure which shows the structural example of a phase difference image sensor A diagram showing the main and sub-pixels one by one 3 is an enlarged view of the main part of FIG. The flowchart which shows the flow of an example of the ghost image process in 1st Embodiment. (A) is a schematic diagram showing a state in which main pixel data and subpixel data having a ghost image are superimposed and displayed, (B) is a diagram showing the main pixel data, and (C) is a diagram showing the subpixel data. Schematic diagram showing the incidence of abnormal light in monocular stereoscopic imaging (A) And (B) is explanatory drawing used for description of the image shift by parallax, and image shift correction Diagram showing an example of ghost extraction data The flowchart which shows the flow of an example of the ghost image process in 2nd Embodiment. The flowchart which shows the flow of an example of the ghost image process in 3rd Embodiment. The flowchart which shows the flow of an example of the ghost image process in 4th Embodiment. Explanatory drawing used for explanation of ghost image processing in the fourth embodiment Plane perspective view showing the pixel arrangement of the image sensor in the fifth embodiment

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

<Overall configuration of imaging device>
FIG. 1 is a block diagram showing an embodiment of a stereoscopic imaging apparatus 10 according to the present invention.

  The stereoscopic imaging apparatus 10 records captured images on a memory card 54 (hereinafter also referred to as “media”), and the operation of the entire apparatus is centrally controlled by a central processing unit (CPU) 40.

  The stereoscopic imaging device 10 is provided with operation units 38 such as a shutter button, a mode dial, a playback button, a MENU / OK key, a cross key, and a BACK key. A signal from the operation unit 38 is input to the CPU 40, and the CPU 40 controls each circuit of the stereoscopic imaging device 10 based on the input signal. For example, lens driving control, aperture driving control, photographing operation control, image processing control, image processing Data recording / reproduction control, display control of the liquid crystal monitor 30 for stereoscopic display, and the like are performed.

  The shutter button is an operation button for inputting an instruction to start shooting, and includes a two-stage stroke type switch having an S1 switch that is turned on when half-pressed and an S2 switch that is turned on when fully pressed. The mode dial is a selection means for selecting a 2D shooting mode, a 3D shooting mode, an auto shooting mode, a manual shooting mode, a scene position such as a person, a landscape, a night view, a macro mode, a moving image mode, and a parallax priority shooting mode according to the present invention. is there.

  The playback button is a button for switching to a playback mode in which a still image or a moving image of a stereoscopic image (3D image) or a planar image (2D image) that has been recorded is displayed on the liquid crystal monitor 30. The MENU / OK key is an operation key having both a function as a menu button for instructing to display a menu on the screen of the liquid crystal monitor 30 and a function as an OK button for instructing confirmation and execution of the selection contents. It is. The cross key is an operation unit for inputting instructions in four directions, up, down, left, and right, and functions as a button (cursor moving operation means) for selecting an item from the menu screen or instructing selection of various setting items from each menu. To do. The up / down key of the cross key functions as a zoom switch for shooting or a playback zoom switch in playback mode, and the left / right key functions as a frame advance (forward / reverse feed) button in playback mode. . The BACK key is used to delete a desired object such as a selection item, cancel an instruction content, or return to the previous operation state.

  In the photographing mode, the image light indicating the subject is imaged on the light receiving surface of the phase difference CCD 16 which is a solid-state image sensor through the photographing optical system 12 including the focus lens and the zoom lens, and the diaphragm 14. The photographing optical system 12 is driven by a lens driving unit 36 controlled by the CPU 40, and performs focus control, zoom control, and the like. The diaphragm 14 is composed of, for example, five diaphragm blades, and is driven by the diaphragm driving unit 34 controlled by the CPU 40. For example, the diaphragm 14 is controlled in six steps from 1 to an aperture value from an aperture value F1.4 to F11.

  The CPU 40 controls the diaphragm 14 via the diaphragm driving unit 34, and controls the charge accumulation time (shutter speed) in the phase difference CCD 16 and the readout of image signals from the phase difference CCD 16 via the CCD control unit 32. Etc.

<Configuration Example of Monocular 3D Sensor>
FIG. 2 is a diagram illustrating a configuration example of the phase difference CCD 16 (monocular stereoscopic imaging device).

  The phase difference CCD 16 has odd-numbered lines of pixels (main pixels) and even-numbered lines of pixels (sub-pixels) arranged in a matrix, and photoelectric conversion is performed in these main and sub-pixels. The image signals for the two surfaces can be read independently.

  As shown in FIG. 2, among the pixels provided with R (red), G (green), and B (blue) color filters on the odd lines (1, 3, 5,...) Of the phase difference CCD 16, GRGR. .., And BGBG... Pixel array lines are alternately provided. On the other hand, even-numbered lines (2, 4, 6,...) Have the same GRGR. Lines and lines of a pixel array of BGBG... Are alternately provided, and pixels are arranged so as to be shifted in the line direction by a half pitch with respect to pixels of even lines.

  FIG. 3 is a diagram showing each of the photographic optical system 12, the diaphragm 14, and the main pixel PDa and the sub-pixel PDb of the phase difference CCD 16, and FIG. 4 is an enlarged view of the main part of FIG.

  As shown in FIG. 4A, a light beam passing through the exit pupil is incident on a normal CCD pixel (photodiode PD) through the microlens L without being restricted.

  On the other hand, a light shielding member 16A is formed on the main pixel PDa and the sub pixel PDb of the phase difference CCD 16, and the right half or the left half of the light receiving surface of the main pixel PDa and the sub pixel PDb is shielded by the light shielding member 16A. Yes. That is, the light shielding member 16A functions as a pupil division member.

  The phase difference CCD 16 having the above configuration is configured such that the main pixel PDa and the sub-pixel PDb have different regions (right half and left half) in which the light beam is limited by the light shielding member 16A. Without limiting to the light shielding member 16A, the microlens L and the photodiode PD (PDa, PDb) are relatively shifted in the left-right direction, and the light flux incident on the photodiode PD is limited by the shifted direction. Alternatively, by providing one microlens for two pixels (main pixel and subpixel), the light flux incident on each pixel may be limited.

  Returning to FIG. 1, the signal charge accumulated in the phase difference CCD 16 is read out as a voltage signal corresponding to the signal charge based on the readout signal applied from the CCD controller 32. The voltage signal read from the phase difference CCD 16 is applied to the analog signal processing unit 18, where the R, G, B signals for each pixel are sampled and held, and the gain designated by the CPU 40 (corresponding to ISO sensitivity). And then added to the A / D converter 20. The A / D converter 20 converts R, G, and B signals that are sequentially input into digital R, G, and B signals and outputs them to the image input controller 22.

  The digital signal processing unit 24 performs gain control processing including gamma correction processing, gamma correction processing, synchronization processing, YC processing for digital image signals input via the image input controller 22, including offset processing, white balance correction, and sensitivity correction. Then, predetermined signal processing such as sharpness correction is performed.

  The EEPROM 56 is a nonvolatile memory that stores a camera control program, defect information of the phase difference CCD 16, various parameters and tables used for image processing, a program diagram, a plurality of parallax priority program diagrams according to the present invention, and the like. Memory.

  Here, as shown in FIGS. 2B and 2C, the main image data read from the odd-numbered line main pixels of the phase difference CCD 16 is processed as left viewpoint image data and read from the even-line sub-pixels. The sub image data is processed as right viewpoint image data.

  The left viewpoint image data and right viewpoint image data (3D image data) processed by the digital signal processing unit 24 are input to the VRAM 50. The VRAM 50 includes an A area and a B area each storing 3D image data representing a 3D image for one frame. In the VRAM 50, 3D image data representing a 3D image for one frame is rewritten alternately in the A area and the B area. The written 3D image data is read from an area other than the area in which the 3D image data is rewritten in the A area and the B area of the VRAM 50. The 3D image data read from the VRAM 50 is encoded by the video encoder 28 and is output to a stereoscopic display liquid crystal monitor 30 (LCD) provided on the back of the camera, whereby a 3D subject image is displayed on the liquid crystal monitor 30. Displayed on the display screen.

  The liquid crystal monitor 30 is a stereoscopic display unit that can display stereoscopic images (left viewpoint image and right viewpoint image) as directional images having predetermined directivities by a parallax barrier, but is not limited thereto, and is not limited thereto. The left viewpoint image and the right viewpoint image may be viewed separately by using a lens, or by wearing dedicated glasses such as polarized glasses or liquid crystal shutter glasses.

  Further, when the shutter button of the operation unit 38 is pressed (half-pressed) in the first stage, the phase difference CCD 16 starts the AF operation and the AE operation, and the focus in the photographing optical system 12 via the lens driving unit 36. Control is performed so that the lens comes to the in-focus position. The image data output from the A / D converter 20 when the shutter button is half-pressed is taken into the AE detection unit 44.

  The AE detection unit 44 integrates the G signals of the entire screen, or integrates the G signals with different weights in the central and peripheral portions of the screen, and outputs the integrated value to the CPU 40. The CPU 40 calculates the brightness of the subject (shooting EV value) from the integrated value input from the AE detection unit 44, and based on this shooting EV value, calculates the aperture value of the diaphragm 14 and the electronic shutter (shutter speed) of the phase difference CCD 16. It is determined according to a predetermined program diagram, the aperture 14 is controlled via the aperture drive unit 34 based on the determined aperture value, and the phase difference CCD 16 via the CCD control unit 32 based on the determined shutter speed. To control the charge accumulation time.

  The AF processing unit 42 is a part that performs contrast AF processing or phase AF processing. When performing contrast AF processing, by extracting a high frequency component of image data in a predetermined focus area from at least one of left viewpoint image data and right viewpoint image data, and integrating the high frequency component An AF evaluation value indicating the in-focus state is calculated. AF control is performed by controlling the focus lens in the photographic optical system 12 so that the AF evaluation value is maximized. When performing the phase difference AF process, the phase difference between the image data corresponding to the main pixel and the sub pixel in the predetermined focus area in the left viewpoint image data and the right viewpoint image data is detected, and this phase difference is detected. The defocus amount is obtained based on the information indicating. AF control is performed by controlling the focus lens in the photographing optical system 12 so that the defocus amount becomes zero.

  When the AE operation and the AF operation are completed and the shutter button is pressed in the second stage (full press), the left corresponding to the main pixel and the sub-pixel output from the A / D converter 20 in response to the press. Two pieces of image data of the viewpoint image (main image) and the right viewpoint image (sub-image) are input from the image input controller 22 to the memory (SDRAM) 48 and temporarily stored.

  The two pieces of image data temporarily stored in the memory 48 are appropriately read out by the digital signal processing unit 24, where predetermined signals including generation processing (YC processing) of luminance data and color difference data of the image data are performed. Processing is performed. The YC processed image data (YC data) is stored in the memory 48 again. Subsequently, the two pieces of YC data are respectively output to the compression / decompression processing unit 26, and after a predetermined compression process such as JPEG (joint photographic experts group) is executed, the YC data is stored in the memory 48 again.

  From the two YC data (compressed data) stored in the memory 48, a multi-picture file (MP file: a file in a format in which a plurality of images are connected) is generated. The PM file is read by the media controller 52. And recorded on the memory card 54.

  Hereinafter, the stereoscopic imaging apparatus according to the present invention will be described in various embodiments.

<First Embodiment>
FIG. 5 is a flowchart showing a flow of an example of ghost image processing in the first embodiment. This process is a ghost image extraction process for extracting a ghost image from a plurality of viewpoint images constituting a stereoscopic image, and is performed by the CPU 40 of FIG. The CPU 40 constitutes an image shift amount calculation unit, an image shift correction unit, and a ghost image detection unit in the present invention.

  Prior to the start of this process, the subject is already three-dimensionally imaged by the phase difference CCD 16 (monocular 3D sensor). That is, the light beams that have passed through different areas (right half and left half) of the exit pupil of the photographic optical system 12 are respectively photoelectrically converted by the phase difference CCD 16, and a plurality of viewpoint images (hereinafter referred to as “main pixel data” and “subpixel data”). Is also temporarily stored in the memory 48.

  In steps S <b> 102 and S <b> 104, the CPU 40 acquires main pixel data and subpixel data from the memory 48.

  FIG. 6A schematically shows a state in which main pixel data and sub-pixel data each having a ghost image are superimposed and displayed on the LCD 30. In addition, since it cannot display in three dimensions on this drawing, it represents as a plane image. FIG. 6B shows main pixel data having a ghost image, and FIG. 6C shows sub-pixel data having a ghost image. Due to the structure of pupil division in the stereoscopic imaging device 10, as shown in FIG. 7, abnormal light incident on the main pixel PDa does not enter the subpixel PDb, and abnormal light incident on the subpixel PDb does not enter the main pixel PDa. Is almost. That is, the ghost image appearing in the main pixel data in FIG. 6B hardly appears in the sub-pixel data in FIG. 6C, and the ghost image appearing in the sub-pixel data in FIG. B) hardly appears in the main pixel data.

  In step S <b> 106, the CPU 40 calculates an image shift amount due to parallax (hereinafter also referred to as “parallax amount”).

  Specifically, first, the correspondence between the main pixel in the main pixel data and the sub-pixel in the sub-pixel data is detected (corresponding point detection), and then an image between corresponding pixels (between corresponding points). The amount of deviation (difference between the coordinates of the main pixel in the main pixel data and the coordinates of the subpixel in the subpixel data) is calculated, and the relationship (parallax) between each pixel (main pixel and / or subpixel) and the amount of image deviation A disparity map indicating the distribution) is generated. Note that the parallax map of the present example shows the correspondence between each sub-pixel in the sub-pixel data and the image shift amount. For example, in the main pixel data shown in FIG. 8 (A) and the sub-pixel data shown in FIG. 8 (B), the image of the tree is shifted by D, so that the image shift in each pixel constituting the tree image. Associate quantity D. In FIGS. 8A and 8B, the human image has almost no image shift, but actually a small image shift corresponding to the subject distance occurs in the human image.

  In step S108, the CPU 40 corrects the image shift caused by the parallax of the subject image between the main pixel data and the sub-pixel data based on the parallax map. In this example, by moving the subpixel in the subpixel data, the coordinates of the main pixel in the main pixel data are matched with the coordinates of the subpixel in the subpixel data corresponding to the main pixel. For example, in the subpixel data shown in FIG. 8B, the tree image is moved leftward by D.

  In step S110, the CPU 40 extracts a ghost image by calculating a difference in pixel value between the main pixel and the sub-pixel after the image shift correction. For example, when the main pixel data shown in FIG. 6B and the sub-pixel data shown in FIG. 6C are acquired, ghost extraction data (ghost extraction image) shown in FIG. 9 is generated. The ghost extraction data of this example is composed of an array of pixel value differences (hereinafter referred to as “ghost pixel values”) for each pixel corresponding to main pixel data and sub-pixel data. In the ghost extraction data of this example, the value of the region where the ghost image exists in the main pixel data (ghost pixel value) is positive, and the value of the region where the ghost image exists in the subpixel data (ghost pixel value) is negative. It becomes. The value is zero in a region where neither ghost data exists in the main pixel data nor the sub pixel data. However, since the value actually does not become zero due to the influence of noise or the like, the main pixel and the sub pixel data are generated when generating the ghost extraction data. When the difference between the pixel value and the pixel is equal to or smaller than the threshold value, adjustment (zero adjustment) forcing the value of the ghost extraction data to zero may be performed. The value of the ghost extraction data (ghost pixel value) has a positive / negative polarity as described above, but in FIG. 9, for the convenience of illustration, the ghost pixel value is displayed as a gradation with an absolute value. In FIG. 9, a portion that is not a ghost image (background sky portion) is also extracted because there is a difference in pixel value between the main pixel and the sub-pixel, but actually differs because it differs from the color caused by the ghost light. Are eliminated by filtering.

  In step S112, CPU 40 outputs ghost extraction data. In this example, the data is output to the memory 48, but the output destination is not particularly limited.

  In addition, although the case where the image shift correction is performed on the sub-pixel data has been described as an example, the image shift correction may be performed on the main pixel data. Image shift correction may be performed on both main pixel data and sub-pixel data.

  In this embodiment, an image shift amount due to parallax between main pixel data (first viewpoint image) and sub-pixel data (second viewpoint image) is calculated, and main pixel data and sub-pixels are calculated based on the image shift amount. Since the image shift due to the parallax with the data is corrected and the ghost image in the main pixel data and the sub pixel data is detected by comparing the main pixel data and the sub pixel data after the image shift correction, A ghost image can be detected by monocular stereoscopic imaging, and erroneous detection of the ghost image can be prevented.

Second Embodiment
FIG. 10 is a flowchart showing a flow of an example of ghost image processing in the second embodiment. This process is a ghost image removal process for removing a ghost image from a plurality of viewpoint images constituting a stereoscopic image, and is performed by the CPU 40 of FIG. The CPU 40 constitutes an image deviation amount calculation means, an image deviation correction means, a ghost image detection means, a ghost image removal means, and an image deviation amount return means in the present invention.

  It is assumed that the ghost image extraction process shown in FIG. 5 has already been completed before the start of this process.

  In step S202, CPU 40 acquires ghost extraction data (ghost extraction image). In this example, it is acquired from the memory 48.

  In step S204, the CPU 40 starts main pixel correction. In this example, the main pixel data is already stored in the memory 48, but if it is not on the memory 48, it is obtained from the memory card 54 or the like.

  In step S206, the CPU 40 refers to the ghost extraction data and determines whether or not the value (ghost pixel value) corresponding to the focused main pixel is positive.

  When the value of the ghost extraction data is negative or 0 (when there is no main pixel ghost), in step S208, the CPU 40 refers to the value of the main pixel data (main pixel value) and adopts it as it is.

  When the value of the ghost extraction data is positive (when the main pixel ghost is present), in step S210, the CPU 40 subtracts the value of the ghost extraction data (ghost pixel value) from the main pixel value. That is, the absolute value of the ghost pixel value is subtracted.

  In step S212, the CPU 40 determines whether or not the correction has been completed for all the main pixels. If the correction has not been completed, the process returns to step S206, and if completed, the process proceeds to step S214.

  In step S <b> 214, the CPU 40 outputs the main pixel data from which the ghost component has been removed by the media controller 52 to the memory card 54.

  In step S216, CPU 40 starts subpixel correction. In this example, the sub-pixel data is already stored in the memory 48, but if it is not on the memory 48, it is obtained from the memory card 54 or the like.

  In step S218, the CPU 40 corrects image shift caused by the parallax between the main pixel data and the sub-pixel data based on the parallax map. In this example, it is the same as step S108 in FIG. 5, and image shift correction is performed on the sub-pixel.

  In step S220, the CPU 40 refers to the ghost extraction data and determines whether or not the value (ghost pixel value) corresponding to the focused sub-pixel is negative.

  When the value of the ghost extraction data is negative (when there is a subpixel ghost), in step S222, the CPU 40 adds the value of the ghost extraction data (ghost pixel value) to the subpixel value. That is, the absolute value of the ghost pixel value is subtracted.

  If the value of the ghost extraction data is positive or 0 (in the case of no subpixel ghost), the CPU 40 refers to the value of the subpixel data (subpixel value) and adopts it as it is in step S224.

  In step S226, the CPU 40 determines whether or not the correction has been completed for all the sub-pixels. If not completed, the CPU 40 returns to step S220, and if completed, the process proceeds to step S228.

  In step S228, the CPU 40 recovers the image shift of the subpixel data after the ghost image is removed based on the parallax map. That is, the image shift caused by the parallax is restored as shown in FIG.

  In step S <b> 230, the CPU 40 outputs the subpixel data from which the ghost component has been removed to the memory card 54 by the media controller 52.

  As described above, in the present embodiment, the ghost image exists in each of the first viewpoint image (main pixel data) and the second viewpoint image based on the polarity of the value of the ghost extraction data (ghost pixel value). The ghost image is recognized by subtracting the absolute value of the pixel value of the ghost extracted image from the pixel value of the portion where the ghost image of each of the first viewpoint image and the second viewpoint image after the image shift correction is recognized. And the image shift amount between the first viewpoint image and the second viewpoint image after removal of the ghost image is returned to the amount before the image shift correction based on the parallax map.

<Third Embodiment>
FIG. 11 is a flowchart showing an exemplary flow of ghost image processing in the third embodiment. This process is a ghost image removal process for removing a ghost image from a plurality of viewpoint images constituting a stereoscopic image, and is performed by the CPU 40 of FIG. The CPU 40 constitutes an image deviation amount calculation means, an image deviation correction means, a ghost image detection means, a ghost image removal means, and an image deviation amount return means in the present invention.

  It is assumed that the ghost image extraction process shown in FIG. 5 has already been completed before the start of this process.

  Steps S302 to S304 are the same as steps S202 to S204 in the second embodiment.

  In step S306, the CPU 40 refers to the ghost extraction data and determines whether or not the value (ghost pixel value) corresponding to the focused main pixel is positive.

  When the value of the ghost extraction data is negative or 0 (when there is no main pixel ghost), the CPU 40 refers to the main pixel value and adopts it as the main pixel value as it is in step S308.

  When the value of the ghost extraction data is positive (when the main pixel ghost is present), in step S310, the CPU 40 refers to the subpixel value and adopts the subpixel value as a new main pixel value.

  In step S312, the CPU 40 determines whether or not the correction has been completed for all the main pixels. If the correction has not been completed, the process returns to step S306, and if completed, the process proceeds to step S314.

  Steps S314 to S318 are the same as steps S214 to S218 in the second embodiment.

  In step S320, the CPU 40 refers to the ghost extraction data and determines whether or not the value (ghost pixel value) corresponding to the focused main pixel is negative.

  When the value of the ghost extraction data is negative (when the sub-pixel ghost is present), in step S322, the CPU 40 refers to the main pixel value and adopts the main pixel value as a new sub-pixel value.

  When the value of the ghost extraction data is positive or 0 (when there is no subpixel ghost), the CPU 40 refers to the subpixel value and adopts it as the subpixel value as it is in step S324.

  In step S326, the CPU 40 determines whether or not the correction has been completed for all the sub-pixels. If not completed, the process returns to step S320, and if completed, the process proceeds to step S328.

  Steps S328 to S330 are the same as steps S228 to S230 in the second embodiment.

  As described above, in the present embodiment, a ghost image of one viewpoint image of the first viewpoint image (main pixel data) and the second viewpoint image (subpixel data) after image shift correction is used as the other viewpoint image. The ghost image is removed by replacing the ghost image with an image in which the ghost image does not exist, and the image shift between the first viewpoint image and the second viewpoint image after the ghost image removal is recovered based on the parallax map. .

<Fourth embodiment>
FIG. 12 is a flowchart showing a flow of an example of ghost image processing in the fourth embodiment. This process is a ghost image removal process for removing a ghost image from a plurality of viewpoint images constituting a stereoscopic image, and is performed by the CPU 40 of FIG. The CPU 40 constitutes an image shift amount calculation unit, an image shift correction unit, an image composition unit, and a multi-viewpoint unit according to the present invention. In this process, the ghost image extraction process shown in FIG. 5 is unnecessary.

  Steps S402 to S408 are the same as steps S102 to S108 in FIG.

  In step S410, the CPU 40 compares the brightness (luminance) between corresponding pixels between the main pixel and the sub-pixel after image shift correction. If the main pixel is brighter, the value of the subpixel is referred to in step S412, and the combined pixel value is adopted. If the subpixel is brighter or the brightness is the same, the main pixel is determined in step S414. The pixel value is referred to and used as a synthesized pixel value.

  In step S416, the CPU 40 determines whether or not the adoption (determination) of all composite pixels has been completed. If not completed, the CPU 40 returns to step S410, and if completed, the process proceeds to step S416.

  When the adoption of all the pixels is completed, as shown in FIG. 13, one composite image (hereinafter referred to as “dark part extracted image data”) in which only the dark part is extracted from the main pixel data and the sub-pixel data is obtained. This dark part extracted image data does not include a ghost image because it is an image obtained by extracting only the dark part. That is, one image from which the ghost image has been removed is obtained.

  In step S418, the CPU 40 generates a new stereoscopic image (a plurality of viewpoint images) from the dark part extracted image data based on the image shift amount calculated in step S406. That is, multiple viewpoints are made. In this example, since the dark part extracted image data is an image of the same viewpoint as the main pixel data, this dark part extracted image data is used as new main pixel data, and new sub pixel data is obtained from the dark part extracted image data based on the parallax map. Generate. These new main pixel data and new subpixel data form a new stereoscopic image that does not include a ghost image.

  In step S <b> 420, CPU 40 causes media controller 52 to output a new stereoscopic image to memory card 54.

<Fifth Embodiment>
Next, a fifth embodiment (when there are a plurality of pupil division directions) will be described.

  FIG. 14A shows an arrangement of main pixels + subpixels, FIG. 14B shows an arrangement of only main pixels, and FIG. 14C shows an arrangement of only subpixels.

  The phase difference CCD 16 of the present embodiment has a plurality of pixel group pairs separated in a plurality of pupil division directions (X direction and Y direction in the drawing). Specifically, the phase difference CCD 16 includes a first pixel group pair including a first main pixel group 61a and a first sub-pixel group 61b separated in a first pupil division direction (X direction in the drawing). 61 and a second pixel composed of a second main pixel group 62a and a second sub-pixel group 62b separated in a second pupil division direction (Y direction in the figure) orthogonal to the first pupil division direction A group pair 62.

  Note that the correspondence relationship between the photodiode PD, the microlens L, and the light shielding member 16A in each pixel group pair is basically the same as that described in the first embodiment, and thus the description thereof is omitted here.

  Note that the pupil division direction is the direction connecting the divided areas of the pupil (the left and right direction when the pupil is divided into the left half and the right half, and the up and down direction when the pupil is divided into the upper half and the lower half. Direction). That is, the direction is orthogonal to the virtual boundary line dividing the pupil. However, when the optical axis center does not match between the imaging optical system and the light receiving surface of the image sensor as in a so-called bending optical system, the light receiving surface of the image sensor along the optical path bent in the pupil division direction in the imaging optical system. This is the direction projected on the screen.

  According to the phase difference CCD 16 of the present example, the first pixel group 61 (61a, 61b) generated by the first pixel group pair 61 is used during horizontal shooting (that is, when the x-axis in FIG. 14A is horizontal). A stereoscopic image is composed of the second viewpoint image and the second viewpoint image, and at the time of vertical shooting (that is, when the x-axis is rotated by 90 degrees in FIG. 14A), the second pixel group pair 62 A stereoscopic image can be formed by the third viewpoint image and the fourth viewpoint image generated by (62a, 62b).

  Next, ghost image processing in this embodiment will be described.

  First, the CPU 40 calculates an image shift amount (first image shift amount) due to the parallax between the first viewpoint image and the second viewpoint image, and parallax between the third viewpoint image and the fourth viewpoint image. An image shift amount (second image shift amount) is calculated. Next, the CPU 40 corrects the image shift between the first viewpoint image and the second viewpoint image based on the first image shift amount, and the third viewpoint image based on the second image shift amount. An image shift from the fourth viewpoint image is corrected. Next, the CPU 40 compares the first viewpoint image and the second viewpoint image after the image shift correction, and compares the third viewpoint image and the fourth viewpoint image after the image shift correction. Detect ghost images in a plurality of viewpoint images.

  In the present embodiment, by using images of four viewpoints obtained by the plurality of pixel group pairs 61 and 62, it is possible to reliably detect a ghost image generated not only in the horizontal direction but also in the vertical direction. .

  The present invention is not limited to the examples described in the present specification and the examples illustrated in the drawings, and various design changes and improvements may be made without departing from the spirit of the present invention.

  DESCRIPTION OF SYMBOLS 10 ... Stereo imaging device, 12 ... Imaging optical system, 14 ... Diaphragm, 16 ... Phase difference CCD (imaging element), 16A ... Light shielding member, 16B ... Opening of light shielding member, 24 ... Digital signal processing part, 40 ... CPU, 48 ... Memory, 52 ... Media controller, 54 ... Memory card, 56 ... EEPROM, L ... Microlens, PD ... Photodiode (pixel)

Claims (14)

Image acquisition means for acquiring a plurality of viewpoint images generated by image sensors that photoelectrically convert light beams that have passed through different areas of a single imaging optical system;
An image shift amount calculating means for calculating an image shift amount due to parallax of the subject image between the acquired viewpoint images;
An image shift correction unit that corrects an image shift between the plurality of viewpoint images based on the image shift amount;
A ghost image detecting means for detecting a ghost image in the plurality of viewpoint images by comparing the plurality of viewpoint images after image shift correction;
A ghost image processing apparatus comprising:   The ghost image detection means generates ghost extraction data composed of the difference array for each corresponding pixel by calculating a pixel value difference between the corresponding pixels of the plurality of viewpoint images after image shift correction. The ghost image processing apparatus according to claim 1. Recognizing a portion where the ghost image exists in each of the plurality of viewpoint images based on the polarity of the difference in the ghost extraction data, and extracting the ghost image from each of the plurality of viewpoint images after image shift correction. Ghost image removing means for removing;
An image shift amount returning means for returning an image shift amount between the plurality of viewpoint images from which the ghost image has been removed to an amount before image shift correction;
The ghost image processing apparatus according to claim 2, further comprising:   The ghost image removing unit subtracts a value of the ghost extraction data from a pixel value of a portion where the ghost image exists in the plurality of viewpoint images after image shift correction. Image processing device.   The ghost image removing means uses a pixel value of a portion where the ghost image of one of the viewpoint images exists among the plurality of viewpoint images after image shift correction as a pixel of a portion where the ghost image of the other viewpoint image does not exist. The ghost image processing apparatus according to claim 3, wherein the ghost image processing apparatus is replaced with a value. The imaging device includes a first pixel group and a second pixel group separated in a first pupil division direction, and a third and a fourth pixel group separated in a second pupil division direction. ,
The image acquisition means includes a first viewpoint image, a second viewpoint image generated by the first pixel group, the second pixel group, the third pixel group, and the fourth pixel group, Obtaining a third viewpoint image and a fourth viewpoint image;
The image shift amount calculation means calculates a first image shift amount due to a parallax between the first viewpoint image and the second viewpoint image, and also includes the third viewpoint image and the fourth viewpoint image. A second image shift amount due to the parallax of
The image shift correction unit corrects an image shift between the first viewpoint image and the second viewpoint image based on the first image shift amount, and based on the second image shift amount. Correcting an image shift between the third viewpoint image and the fourth viewpoint image;
The ghost image detection means compares the first viewpoint image and the second viewpoint image after image shift correction, and the third viewpoint image and the fourth viewpoint image after image shift correction. The ghost image processing apparatus according to claim 1, wherein a ghost image in the plurality of viewpoint images is detected by comparing the ghost images. Image acquisition means for acquiring a plurality of viewpoint images generated by image sensors that photoelectrically convert light beams that have passed through different areas of a single imaging optical system;
An image shift amount calculating means for calculating an image shift amount due to parallax of the subject image between the acquired viewpoint images;
An image shift correction unit that corrects an image shift between the plurality of viewpoint images based on the image shift amount;
Image synthesizing means for generating a synthesized image that does not include a ghost image by comparing the brightness between corresponding pixels of the plurality of viewpoint images after image shift correction and extracting only the darker part;
A ghost image processing apparatus comprising:   A stereoscopic imaging apparatus comprising: the single imaging optical system; the imaging element; and the ghost image processing apparatus according to any one of claims 1 to 7. An image acquisition step of acquiring a plurality of viewpoint images generated by an image sensor that photoelectrically converts light beams that have passed through different regions of a single photographing optical system;
An image shift amount calculating step of calculating an image shift amount due to the parallax of the subject image between the acquired viewpoint images;
An image shift correction step of correcting an image shift between the plurality of viewpoint images based on the image shift amount;
A ghost image detection step of detecting a ghost image in the plurality of viewpoint images by comparing the plurality of viewpoint images after image shift correction;
A ghost image processing method comprising:   The ghost image detection step generates ghost extraction data including the array of the differences for each of the corresponding pixels by calculating a pixel value difference between the corresponding pixels of the plurality of viewpoint images after the image shift correction. The ghost image processing method according to claim 9. Recognizing a region where the ghost image exists in each of the plurality of viewpoint images based on the polarity of the difference in the ghost extraction data, the ghost image is obtained from each of the plurality of viewpoint images after image shift correction. A ghost image removing step to be removed;
An image shift amount returning step for returning an image shift amount between the plurality of viewpoint images from which the ghost image has been removed to an amount before image shift correction;
The ghost image processing method according to claim 10, further comprising:   12. The ghost image removing step according to claim 11, wherein the ghost image removing step subtracts a value of the ghost extraction data from a pixel value of a portion where the ghost image exists in the plurality of viewpoint images after the image shift correction. Image processing method.   In the ghost image removing step, the pixel value of the portion where the ghost image of one of the viewpoint images exists among the plurality of viewpoint images after image shift correction is the pixel of the portion where the ghost image of the other viewpoint image does not exist. The ghost image processing method according to claim 11, wherein the ghost image processing method is replaced with a value. An image acquisition step of acquiring a plurality of viewpoint images generated by an image sensor that photoelectrically converts light beams that have passed through different regions of a single photographing optical system;
An image shift amount calculating step of calculating an image shift amount due to the parallax of the subject image between the acquired viewpoint images;
An image shift correction step of correcting an image shift between the plurality of viewpoint images based on the image shift amount;
An image synthesis step of generating a composite image that does not include a ghost image by comparing the brightness between corresponding pixels of the plurality of viewpoint images after image shift correction and extracting only the darker part;
A ghost image processing method comprising: