JP2013236291A - Stereoscopic imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a stereoscopic image with a large depth of field to be captured, which is a focused from a short distance to a long distance.SOLUTION: A stereoscopic imaging apparatus includes: one objective lens 1 for forming an image of a subject in the air; a relay lens system 3 for reforming a formed aerial image 2; pupil division means 33 disposed close to a diaphragm 32 of the relay lens system 3 and dividing a light flux having arrived at the diaphragm 32 into multiple rays; and multiple imaging elements 4a and 4b disposed on each one of image forming surfaces of the relay lens system 3 after division by the pupil division means 33. A stereoscopic image is captured by the multiple imaging elements 4a and 4b. A wave surface modulator 35 is disposed close to the diaphragm 32 of the relay lens system 3, and a stereoscopic image with a large depth of field can be captured by the wave surface modulator 35.

Description

  The present invention relates to a stereoscopic imaging device, and more particularly to a stereoscopic imaging device that enables stereoscopic viewing by capturing a plurality of images having different parallaxes.

  Conventionally, in a field such as an endoscope or a microscope, object light from a subject is captured by one imaging optical system, divided into a plurality of light beams inside the optical system, and a plurality of images with different parallaxes are captured, A stereoscopic imaging apparatus that enables stereoscopic viewing is known (see, for example, Patent Document 1 and Patent Document 2).

  Hereinafter, a conventional stereoscopic imaging apparatus will be described with reference to FIG. In FIG. 15A, light emitted from a point light source on a subject (not shown) enters the objective lens 1. The light beam incident on the objective lens 1 is condensed by the front group lens 11 on the front side of the diaphragm 12 of the objective lens 1 and the rear group lens 13 after the diaphragm, and is formed as an aerial image 2 on the image plane of the objective lens 1. This is conjugate with the point light source on the subject. A relay lens system 3 is arranged with the aerial image 2 as an object plane. The light beam incident on the relay lens system 3 is bent by the front lens group 31 so that the upper light beam, the main light beam, and the lower light beam are substantially parallel, and is incident on the stop 32 of the relay lens system 3. In the vicinity of the stop 32 of the relay lens system 3, pupil dividing means 33 such as a triangular prism or a mirror that divides the light beam in two directions and changes the direction is arranged. Here, the pupil means an image of a diaphragm, and in FIG. 15A, an image formed by the front group lens 31 of the diaphragm 32 of the relay lens system 3 is called a pupil.

  Here, FIG. 15B shows the stop 32 of the relay lens system 3 as viewed from the light incident side. As shown in FIG. 15B, the object light emitted in the right direction from the subject is reversed left and right by the objective lens 1 and reaches the left region 32 a of the pupil 32 of the relay lens system 3. Similarly, object light emitted leftward from the subject is reversed left and right by the objective lens 1 and reaches the right region 32b of the pupil 32 of the relay lens system 3. The light reaching the left region 32a of the pupil of the relay lens system 3 is reflected leftward by the pupil dividing unit 33a, and the light reaching the right region 32b of the pupil of the relay lens system 3 is reflected rightward by the pupil dividing unit 33b. To do.

  That is, when the pupil of the relay lens system 3 is divided into a left region 32a and a right region 32b, a light beam traveling from the object to the right direction in the left region 32a of the pupil, and a light beam traveling from the object to the left in the right region 32b of the pupil. Therefore, it can be seen that light having parallaxes in two different directions gathers in the left region 32a and the right region 32b of the pupil. The light reflected leftward by the pupil dividing means 33a is imaged on the image sensor 4a by the rear group lens 34a of the relay lens system 3. Similarly, the light reflected rightward by the pupil dividing means 33b is imaged on the image sensor 4b by the rear group lens 34b of the relay lens system 3.

  As described above, the conventional stereoscopic imaging apparatus divides the light beam taken in by one objective lens 1 into two different regions using the pupil dividing means 33 provided in the vicinity of the stop 32 of the relay lens system 3, and each of the two different 2's. By forming an image on the two image pickup devices 4a and 4b, two images having different parallaxes in the left-right direction can be picked up.

Japanese Unexamined Patent Publication No. 6-194581 JP-A-8-220448

Here, the depth of field of a general imaging device is expressed by the following equation (1).
Front depth of field Lf = δ · F · L ² / (f ² + δ · F · L)
Rear depth of field Lr = δ · F · L ² / (f ² −δ · F · L)
Depth of field = Lf + Lr (1)
Here, f is the focal length of the optical system, F is the F value (Fno) of the optical system, δ is the allowable confusion circle diameter (allowable blur), and L is the imaging distance.

  As can be seen from the above equation (1), when the imaging distance L is determined, the depth of field is uniquely determined by the Fno of the imaging optical system, the focal length f, and the allowable confusion circle diameter δ. With a general camera, Fno, focal length f, and allowable circle of confusion circle diameter δ are limited, and so-called omnifocal images that are in focus from a short distance to a long distance cannot be taken.

  The same applies to the stereoscopic imaging devices described in Patent Documents 1 and 2, and there is a problem that the depth of field is limited. In particular, along with advances in production equipment such as CCD (Charge Coupled Device) and CMOS (Complementary Metal Oxide Semiconductor) in recent years, the pixels of the image sensor have become higher definition and smaller, and the imaging optical system is brighter and has a permissible circle of confusion. The diameter is small. For this reason, the depth of field of the stereoscopic imaging device as described above is narrowly limited.

  The present invention has been made in view of the above situation, and provides a stereoscopic imaging apparatus capable of imaging a stereoscopic image with a deep depth of field that is in focus from a short distance to a long distance. For the purpose.

  In order to solve the above problems, a first technical means of the present invention includes one objective lens that forms an image of a subject in the air, a relay lens system that re-images the formed aerial image, and the relay. A pupil dividing unit that is arranged in the vicinity of the stop of the lens system and divides the light beam reaching the stop into a plurality of pieces, and a plurality of image pickup devices that are divided by the pupil dividing unit and arranged on each imaging plane of the relay lens system; A stereoscopic imaging apparatus that captures a stereoscopic image using the plurality of imaging elements, wherein a wavefront modulation element is disposed near the stop of the objective lens or the stop of the relay lens system It is.

  According to a second technical means, in the first technical means, the pupil dividing means is a reflecting surface of a prism arranged in the vicinity of the stop of the relay lens system.

  According to a third technical means, in the first or second technical means, the wavefront modulation element is configured as an asymmetric phase plate with respect to an optical axis, and the stereoscopic imaging device stores one predetermined PSF data. And a plurality of image restoration units for restoring each piece of image data captured by the plurality of image pickup devices based on the one PSF data.

  According to a fourth technical means, in the first or second technical means, the wavefront modulation element is configured as a phase plate symmetric with respect to an optical axis, and the stereoscopic imaging apparatus is associated with distance information to a subject in a PSF. A distance-corresponding PSF storage unit that stores data, a distance information calculation unit that calculates distance information from each image data captured by the plurality of image sensors, and PSF data corresponding to the calculated distance information is converted into the distance-corresponding PSF. A plurality of image restoration units that are extracted from the storage unit and restore the image data based on the extracted PSF data are provided.

  According to a fifth technical means, in the second technical means, the reflecting surface of the prism is formed into a curved shape, so that the reflecting surface functions as the wavefront modulation element.

A sixth technical means is the fifth technical means, wherein the reflecting surface shape of the prism is:
Z = αX ³ + βY ³ , where α and β are coefficients, X is a distance in the direction perpendicular to the prism bending from the optical axis, Y is a distance in the prism bending direction from the optical axis,
It is characterized by an aspherical shape defined by

  According to the present invention, when a light beam from one objective lens is divided by a relay lens system to obtain a plurality of captured images having different parallaxes, wavefront modulation arranged in the vicinity of the stop of the objective lens or in the vicinity of the stop of the relay lens system By restoring the image based on the blur caused by the element, it is possible to obtain a captured image having a deep depth of field that is in focus from a short distance to a long distance. For this reason, it is less susceptible to individual differences in lens distortion, optical axis orientation variations due to alignment, etc., compared to stereoscopic viewing of images captured separately by different imaging optical systems, and to the subject you want to see at any time. It is possible to capture a focused stereoscopic image.

It is a figure which shows the principal part structural example of the three-dimensional imaging device by this invention. It is a figure which shows an example of the relationship between the object distance a and the image distance b when the focal distance f of an objective lens is 11 mm. It is a figure which shows an example of an objective lens. It is a figure which shows an example of PSF data when the distance (imaging distance) from an object surface to an optical system changes. It is a figure which shows an example of the captured image corresponding to PSF of FIG. It is a figure which shows the structural example of the relay lens system at the time of giving the function of a wavefront modulation element to a prism. It is a figure which shows an example of the PSF data by an optical system when the function as a wavefront modulation element shown by Formula (4) is given to the reflective surface of a prism. It is a figure which shows an example of the captured image corresponding to PSF of FIG. It is a figure for demonstrating the principle of the omnifocal camera using a phase plate. It is a figure which shows an example of the decompression | restoration image which removed the blurring from the captured image of FIG. 8 using the PSF data of FIG. It is a block diagram which shows the principal part structural example of the three-dimensional imaging device which concerns on one Embodiment of this invention. It is a block diagram which shows the principal part structural example of the three-dimensional imaging device which concerns on other embodiment of this invention. It is a figure which shows an example of the image imaged with two image pick-up elements with which a three-dimensional imaging device is provided. It is a schematic diagram for demonstrating an example of the image restoration method by this invention. It is a figure for demonstrating the conventional stereoscopic imaging device.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments according to a stereoscopic imaging device of the invention will be described with reference to the accompanying drawings. The stereoscopic imaging apparatus according to the present invention can be particularly effectively applied to a 3D microscope, a 3D endoscope, a high pixel 3D camera, and the like having a small F value and a narrow depth of field.

FIG. 1 is a diagram illustrating a configuration example of a main part of a stereoscopic imaging apparatus according to the present invention. The basic configuration is the same as the configuration shown in FIG. 15 except that the wavefront modulation element 35 is provided in the relay lens system 3. In FIG. 1A, as described above, light emitted from a point light source on a subject (not shown) enters the objective lens 1. The light beam incident on the objective lens 1 is condensed by the front group lens 11 on the front side of the diaphragm 12 of the objective lens 1 and the rear group lens 13 after the diaphragm, and is formed as an aerial image 2 on the image plane of the objective lens 1. In this imaging, the subject and the aerial image 2 are in a conjugate relationship, and the following equation (2) is established when the angle of view is a paraxial.
1 / a + 1 / b = 1 / f Equation (2)
Where a is the distance from the subject to the lens (object distance), b is the distance from the lens to the image plane (image distance), and f is the focal length of the objective lens.

  FIG. 2 shows an example of the relationship between the object distance a and the image distance b when the focal length f of the objective lens 1 is 11 mm. From this, it can be seen that the image distance b is reduced to 11.4 mm to 11.0 mm with respect to the object distance a of 300 mm to 10000 mm. That is, the aerial image 2 can be in the range of 11.4 mm to 11.0 mm from the objective lens 1.

  FIG. 3 shows an example of the objective lens 1. In the figure, 11 is a front group lens of the objective lens 1, 12 is a stop, and 13 is a rear group lens of the objective lens 1. As described above, both the front lens group 11 and the rear lens group 13 are composed of a plurality of lenses in order to satisfy specifications such as aberration, brightness, and field angle of the optical system. Further, a principal lens at each angle of view incident on the image sensor is made parallel by a cemented lens disposed at the end of the rear group lens 13. As a result, the image size can be kept unchanged regardless of the focal position of the objective lens 1.

  Returning to FIG. 1A, the relay lens system 3 is arranged with the aerial image 2 as the object plane. The light beam incident on the relay lens system 3 is bent by the front lens group 31 so that the upper light beam, the main light beam, and the lower light beam are substantially parallel, and is incident on the stop 32 of the relay lens system 3. In the vicinity of the stop 32 of the relay lens system 3, pupil dividing means 33 such as a triangular prism or a mirror that divides the light beam in two directions and changes the direction is arranged. Here, the pupil means an image of a stop, and an image formed by the front group lens 31 of the stop 32 of the relay lens system 3 is called a pupil. The pupil dividing means 33 is not particularly limited to the reflection type element as described above, and for example, a transmission type element such as a fly-eye lens, a diffraction grating, or a hologram may be used. Hereinafter, a case where a prism (hereinafter also referred to as a prism 33) is used as the pupil dividing unit 33 will be described as an example.

  FIG. 1B shows the diaphragm 32 of the relay lens system 3 viewed from the light incident side. As shown in FIG. 1B, the object light emitted from the subject in the right direction is reversed left and right by the objective lens 1 and reaches the left region 32a of the pupil 32 of the relay lens system 3. Similarly, object light emitted leftward from the subject is reversed left and right by the objective lens 1 and reaches the right region 32b of the pupil 32 of the relay lens system 3. The light reaching the left region 32a of the pupil of the relay lens system 3 is reflected leftward by the pupil dividing unit 33a, and the light reaching the right region 32b of the pupil of the relay lens system 3 is reflected rightward by the pupil dividing unit 33b. To do.

  That is, when the pupil of the relay lens system 3 is divided into the left region 32a and the right region 32b by the pupil dividing means 33, the light beam directed from the object in the right direction in the left region 32a of the pupil and the object in the right region 32b of the pupil. Since the light beams traveling from left to right are collected, light having parallax in two different directions is collected in the left region 32a and the right region 32b of the pupil. The light reflected leftward by the pupil dividing means 33a is imaged on the image sensor 4a by the rear group lens 34a of the relay lens system 3. Similarly, the light reflected rightward by the pupil dividing means 33b is imaged on the image sensor 4b by the rear group lens 34b of the relay lens system 3. These image sensors 4a and 4b are constituted by a CCD or a CMOS.

  As described above, the stereoscopic imaging apparatus according to the present invention divides the light beam captured by one objective lens 1 into two different regions using the pupil dividing means 33 provided near the stop 32 of the relay lens system 3, By imaging on two different image pickup devices 4a and 4b, two images having different parallaxes in the left-right direction can be picked up. Further, in the stereoscopic imaging apparatus according to the present invention, the wavefront modulation element 35 is disposed in the vicinity of the stop 32 of the relay lens system 3. The light beam modulated by the wavefront modulation element 35 forms an image on the pixels of the image pickup elements 4a and 4b such as a CCD and a CMOS. The imaging elements 4a and 4b convert light blurred and imaged by the wavefront modulation element 35 on each element surface into an image signal (image data), and output the converted image signal to an image restoration unit (described later). The image restoration unit performs a filtering process on the image signal including the blur, thereby restoring the two stereoscopic images from which the blur is removed.

  As the wavefront modulation element 35, for example, an optical axis symmetric phase plate or an optical axis asymmetric phase plate called a tertiary phase plate can be used. In addition, the reflecting surface of the prism 33 may be curved so that the reflecting surface functions as a wavefront modulation element. Here, the PSF (Point Spread Function) used for image restoration according to the present invention will be briefly described. PSF is a point spread function that represents how a point light source on the object plane spreads on the image plane when it passes through the optical system. In other words, this PSF shows the response of the optical system to the point light source, and can be said to show the degree of blurring of the image.

  FIG. 4 is a diagram illustrating an example of PSF data when the distance from the object plane to the optical system (hereinafter referred to as an imaging distance) changes. The example of FIG. 4B shows PSF data at the best focus, the example of FIG. 4A shows PSF data when the imaging distance is short, and the example of FIG. 4C is when the imaging distance is far. The PSF data at the time of is shown. From this, it can be seen that in a general optical system, the PSF size is reduced to the smallest at the best focus, and the PSF size is increased when the best focus is removed.

  FIG. 5 is a diagram illustrating an example of a captured image corresponding to the PSF of FIG. The example in FIG. 5B shows a captured image when the imaging distance is the best focus, the example in FIG. 5A shows a captured image when the imaging distance is short, and the example in FIG. 5C is the imaging distance. The captured image when is far is shown. Since the blur of the PSF and the captured image varies greatly depending on the imaging distance, it can be said that the depth of field is largely limited in a general optical system. Here, the depth of field of a general imaging apparatus is as shown by the above-described equation (1). That is, when the imaging distance is determined, the depth of field is uniquely determined by the Fno (F value) of the imaging optical system, the focal length, and the allowable confusion circle diameter. A general camera cannot capture a so-called omnifocal image that is in focus from a short distance to a long distance because of restrictions on Fno, focal length, and allowable circle of confusion.

  Therefore, in the present invention, the wavefront modulation element 35 is provided in the optical system, and the image including the blurred image is subjected to predetermined image processing to perform image restoration, thereby focusing from a short distance to a long distance. It is configured to obtain a stereoscopic image having a deep depth of field. As the wavefront modulation element 35, it is desirable to use a third-order phase plate that is asymmetric with respect to the optical axis. Since this third-order phase plate has a small change in PSF due to defocusing, it is only necessary to prepare one PSF data predetermined for image restoration. The principle of the third-order phase plate is described in, for example, “Shinichi Komatsu,“ Wavefront Coding Technology for Expanding the Depth of Field of an Imaging System—Physical Mechanisms and Recent Trends ””, Journal of the Institute of Image Information and Television Engineers, 2009, Vol. 63, No. 3, pp 279-283 ", etc., and will not be described here.

Further, as the wavefront modulation element 35, an optical axis symmetric phase plate may be used. However, since the optical axis symmetric phase plate has a large PSF change due to defocusing, the PSF is changed based on the distance information obtained by the two image sensors 4a and 4b, and an image is restored by the changed PSF. As a result, it is possible to perform image restoration that is in focus from a short distance to a long distance in the same manner as the third-order phase plate. As a specific example of the optical axis symmetrical phase plate, it is configured by different surfaces on the incident side and the emission side, the first surface on the incident side has a concave spherical shape on the subject side, and the second surface on the emission side is An even-order aspherical shape defined by the equation (3) is given.
Z = α (−ρ ² + ρ ¹⁶ ), where α is a coefficient, ρ is a radial distance from the optical axis (3)

  Here, as described above, the function as the wavefront modulation element 35 may be realized by the two reflecting surfaces of the prism 33. Accordingly, it is possible to provide a function as a wavefront modulation element without adding a new component to the stereoscopic imaging apparatus.

FIG. 6 is a diagram showing a configuration example of the relay lens system 3 when the prism 33 has the function of a wavefront modulation element. The front group lens 31 and the rear group lens 34 are composed of bonded lenses for achromatic purposes, and the number of lenses can be further increased to improve performance such as aberration reduction. A prism 33 is disposed in the vicinity of the stop, and the prism 33 has an aspherical reflecting surface. For example, when functioning as a tertiary phase plate, the shape of the reflecting surface of the prism 33 can be expressed by the following formula (4).
Z = αX ³ + βY ³ Formula (4)
Here, α and β are coefficients, X is a distance in a direction perpendicular to the bending of the prism from the optical axis, and Y is a distance in the bending direction of the prism from the optical axis.

  The prism 33 can be produced by, for example, a method of producing a transparent plastic material by injection molding using a mold, or a method of producing by polishing glass.

  FIG. 7 is a diagram showing an example of PSF data by the optical system when the reflecting surface of the prism 33 is provided with a function as a wavefront modulation element expressed by the equation (4). The example in FIG. 7A shows PSF data when the imaging distance is short, the example in FIG. 7B shows PSF data at the best focus, and the example in FIG. 7C is when the imaging distance is far. The PSF data is shown. Unlike a general optical system, the original PSF size is large and the image is blurred, but it can be seen that the change in blur is small regardless of the distance. In other words, the degree of blur does not change depending on the depth of field, and if the PSF is known, it is possible to restore the captured image by image processing and generate an omnifocal image.

  FIG. 8 is a diagram illustrating an example of a captured image corresponding to the PSF of FIG. The example in FIG. 8B shows a captured image at the best focus, the example in FIG. 8A shows a captured image when the imaging distance is short, and the example in FIG. 8C is when the imaging distance is far. The captured image at the time of is shown. 7 and 8, it can be seen that there is little change in the PSF and the blur of the captured image depending on the imaging distance.

  The principle of a so-called omnifocal camera using such a wavefront modulation element will be briefly described. An omnifocal camera takes a blurred image (intermediate image) by inserting a special wavefront modulation element in the optical lens. If this blur can be made constant regardless of defocusing, a single restoration is possible. An image can be restored with a function (deconvolution filter). That is, from the relationship between the spatial domain and the frequency domain shown in FIG. 9, f (x, y) indicating the original image and h (x, y) indicating the PSF are Fourier-transformed to obtain F (u, v), H ( u, v). Then, G (u, v) (= H (u, v) F (u, v)) indicating the Fourier transform of the degraded image is subjected to filter processing using an inverse filter, a Wiener filter, or the like, and the inverse Fourier transform is performed. By returning to the space area with, an image from which blur is removed is generated.

  FIG. 10 is a diagram illustrating an example of a restored image obtained by removing the blur from the captured image of FIG. 8 using the PSF data of FIG. The example in FIG. 10B shows a restored image at the best focus, the example in FIG. 10A shows a restored image when the imaging distance is short, and the example in FIG. 10C is when the imaging distance is far. The restoration image at the time of is shown. Thus, it is understood that the blur is removed regardless of the imaging distance, and it is possible to obtain a focused captured image from a short distance to a long distance.

  FIG. 11 is a block diagram illustrating a configuration example of a main part of a stereoscopic imaging apparatus according to an embodiment of the present invention. The stereoscopic imaging device includes a first imaging device 4a, a second imaging device 4b, a first image restoration unit 5a, a second image restoration unit 5b, and a PSF storage unit 6. That is, when the wavefront modulation element arranged in the optical system is an optical axis asymmetric third-order phase plate, the stereoscopic imaging apparatus includes a PSF storage unit 6 that stores one predetermined PSF data, and the one PSF. A plurality of image restoration units 5a and 5b for restoring each image data picked up by the plurality of image pickup devices 4a and 4b based on the data. Thereby, it is possible to restore an image based on one PSF in an optical system including the objective lens 1 to the relay lens system 3 and restore an image for stereoscopic viewing that is in focus from a short distance to a long distance.

  Although FIG. 1 shows the case where the wavefront modulation element 35 is disposed in the vicinity of the diaphragm (pupil) 32 of the relay lens system 3, it may be disposed in the vicinity of the diaphragm (pupil) 12 of the objective lens 1. Good. Here, in the relay lens system, since the magnification is close to the same magnification, the depth of field and the depth of focus do not change much, but in an optical system such as a general microscope, as shown in FIG. Since the image is reduced and formed by the objective lens, the depth of focus becomes smaller than the depth of field. For this reason, it is considered that the wavefront modulation element 35 is preferably disposed in the vicinity of the diaphragm 32 of the relay lens system 3 rather than in the vicinity of the diaphragm 12 of the objective lens 1.

  In addition, when the objective lens can be exchanged with a microscope or the like, and the wavefront modulation element is provided on the objective lens side, the optical accuracy changes with the exchange of the objective lens. For this reason, by arranging the wavefront modulation element 35 in the relay lens system 3, there is also an advantage that a change in optical accuracy due to replacement of the objective lens can be reduced.

  On the other hand, when the enlarged image is formed by the objective lens, the relationship between the depth of field and the depth of focus is opposite to that in the case of the reduced image, so that the wavefront modulation element 35 is located near the stop 32 of the relay lens system 3. In addition, it is considered that it is preferable to dispose the objective lens 1 in the vicinity of the diaphragm 12.

  In FIG. 11, image data captured by the first image sensor 4 a is an intermediate image 1, and image data captured by the second image sensor 4 b is an intermediate image 2. That is, these intermediate images 1 and 2 are images including a blur (blurred image). The intermediate image 1 is output to the first image restoration unit 5a, and the intermediate image 2 is similarly output to the second image restoration unit 5b. The image restoration units 5a and 5b restore the intermediate images 1 and 2 including the blur using one predetermined PSF data (PSF storage unit 6), the first image restoration unit 5a outputs the restored image 1, The second image restoration unit 5 b outputs the restored image 2. As the PSF data, for example, a PSF having a frequently used shooting distance, a PSF having a hyperfocal distance (the shortest distance at which infinity enters the depth of field), a PSF having infinity, and the like may be used. .

  There are a plurality of methods for restoring image data, and an example is image restoration using a Wiener filter. In this method, PSF data is deconvolved (deconvolved) in the frequency domain to restore a blurred image. For example, when an image is taken with a digital camera, deterioration such as camera shake and defocusing occurs. Deterioration such as blurring and blurring is expressed by the aforementioned PSF (point spread function). That is, the deteriorated image is expressed as a convolution integral between the original image and the PSF. When the PSF representing degradation is known, the original image can be restored from the degraded image and the PSF using a Wiener filter.

Specifically, a deteriorated image due to blurring or blurring is modeled as the following equation.
g (x, y) = h (x, y) * f (x, y) (5)
Here, g (x, y) represents image data obtained by imaging, that is, a deteriorated image that is an observation image. H (x, y) represents PSF data representing degradation, f (x, y) represents an original image without degradation, and * represents convolution integration.

It is known that the degraded image model represented by the above formula (5) is represented as follows in the frequency domain. That is, the degraded image model of Expression (5) can be expressed by the following Expression (6) by Fourier transform.
G (u, v) = H (u, v) F (u, v) (6)
Here, G (u, v), H (u, v), and F (u, v) are the degraded image g (x, y), PSFh (x, y), and original image f (x, y). ) Respectively.

It is often used to estimate (restore) the Fourier transform of the original image by performing a filtering process on the Fourier transform of the degraded image. Image restoration using filter processing is generally expressed as the following equation (7).
F (u, v) = K (u, v) G (u, v) (7)
Here, F (u, v) is a Fourier transform of the estimated original image, that is, a restored image (restored image). K (u, v) is a filter for image restoration, and is simply referred to as an image restoration filter hereinafter. G (u, v) is the Fourier transform of the degraded image.

  Conventionally, an inverse filter is known as the simplest image restoration filter in the field of image restoration in which an original image is restored from a deteriorated image. This inverse filter is an image restoration filter of K (u, v) = 1 / H (u, v). Here, H (u, v) is a Fourier transform of PSF h (x, y).

  However, since this inverse filter is very unstable with respect to noise, it is practically rarely used. For this reason, many image restoration filters in which the inverse filter is stabilized are used in the field of image restoration. Many conventional image restoration filters have parameters for adjusting the stability of the filter, and the Wiener filter is one example and is widely used.

The Wiener filter is represented by the following formula (8).
Here, K _w (u, v) is a Wiener filter. H ′ (u, v) represents a complex conjugate of the Fourier transform H (u, v) of PSF, and d represents a parameter representing the stability of the filter.

The image restoration by the Wiener filter of the above formula (8) is represented as the following formula (9) by substituting K _w (u, v) of the formula (8) into the formula (7).
Here, F (u, v) is a Fourier transform of the estimated original image, that is, a restored image. G (u, v) is the Fourier transform of the degraded image.

  The PSF data is calculated in advance by predetermined optical design software or the like and stored in the PSF storage unit 6.

  FIG. 12 is a block diagram illustrating a configuration example of a main part of a stereoscopic imaging apparatus according to another embodiment of the present invention. The stereoscopic imaging device includes a first imaging device 4a, a second imaging device 4b, a first image restoration unit 7a, a second image restoration unit 7b, a distance information calculation unit 8, a third image restoration unit 9a, 4 image restoration units 9b and a distance-corresponding PSF storage unit 10. In this embodiment, the wavefront modulation element disposed in the optical system is an optical axis symmetric phase plate. In this case, since the PSF changes greatly due to defocusing, the PSF is changed according to the distance information. Is done. That is, the stereoscopic imaging apparatus calculates distance information from the distance-corresponding PSF storage unit 10 that stores PSF data in association with distance information to the subject and each image data captured by the two imaging elements 4a and 4b. An information calculation unit 8 and PSF data corresponding to the calculated distance information are extracted from the distance-corresponding PSF storage unit 10, and a plurality of image restoration units 9a and 9b are provided for restoring each image data based on the extracted PSF data. .

  In FIG. 12, the image data captured by the first image sensor 4 a is an intermediate image 1, and the image data captured by the second image sensor 4 b is an intermediate image 2. That is, these intermediate images 1 and 2 are images including a blur (blurred image). The intermediate image 1 is output to the first image restoration unit 7a and the third image restoration unit 9a, and the intermediate image 2 is output to the second image restoration unit 7b and the fourth image restoration unit 9b. The first image restoration unit 7a and the second image restoration unit 7b restore the intermediate images 1 and 2 including the blur with one predetermined PSF data. As this PSF data, for example, a PSF with a frequently used shooting distance, a PSF with a hyperfocal distance (the shortest distance at which infinity enters the depth of field), a PSF with infinity, or the like may be used.

  As described above, the captured image (intermediate image 1) from the first image sensor 4a is restored by the first image restoration unit 7a, and the captured image (intermediate image 2) from the second image sensor 4b is the second. Is restored by the image restoration unit 7b. The restoration by the image restoration units 7a and 7b is not a complete image restoration but a temporary image restoration. The two restored images restored in this way are output to the distance information calculation unit 8.

  The distance information calculation unit 8 generates disparity information (disparity map) that is a difference in coordinate values between image signals representing a common subject input from the first image sensor 4a and the second image sensor 4b. The process performed by the distance information calculation unit 8 will be described more specifically.

  The distance information calculation unit 8 calculates parallax information between the image sensors for each pixel based on the image signals respectively input from the image restoration units 7a and 7b. The parallax is a difference between pixel positions representing a common subject represented by image signals input from two different image sensors. Since the parallax information corresponds to the distance, it can be grasped as distance information.

  In order to calculate the parallax information, the distance information calculation unit 8 performs, for example, block matching. In block matching, the distance information calculation unit 8 calculates an evaluation value representing the similarity between pixel values of an image block that is a part of one image signal and an image block that is a part of the other image signal, and is most similar. An image block for which an evaluation value indicating this is calculated is searched from the other image signal. The evaluation value is, for example, a sum of absolute differences (SAD). The distance information calculation unit 8 calculates the parallax between corresponding parts of the subject that both image signals represent in common. Here, an example of an image captured by the two image sensors 4a and 4b will be described below.

  FIG. 13 is a diagram illustrating an example of images captured by the two imaging elements 4a and 4b included in the stereoscopic imaging device. FIG. 13A shows an image captured by the first image sensor 4a. This image is an image including a vehicle as a subject. FIG. 13B shows an image captured by the second image sensor 4b. The image of FIG. 13B is an image including a vehicle common to the vehicle of FIG. Here, the horizontal coordinate of the center part of the license plate N1 of the vehicle included in the image of FIG. 13A is H1, and the vertical coordinate is V. The part corresponding to this part, that is, the horizontal coordinate of the central part of the license plate N2 of the vehicle included in the image of FIG. 13B is H2, and the vertical coordinate is V. In this case, the horizontal coordinate difference (H1-H2) between the images is parallax.

As described above, the parallax represents the distance Z from the stereoscopic imaging device to the subject. Between the parallax (H1-H2) and the distance Z, for example, there is a relationship represented by Expression (10).
Here, f is the focal length of the imaging unit, B is the baseline length, and p is the pixel pitch of the imaging element. The parallax takes different values depending on the distance from the stereoscopic imaging device to the subject. The parallax is 0 when the subject is at infinity, and increases as the distance from the stereoscopic imaging device to the subject decreases.

  From the pixel value of the parallax map obtained by the distance information calculation unit 8 as described above, the distance to the subject imaged in the image is calculated. This distance information is output to the third image restoration unit 9a and the fourth image restoration unit 9b. That is, the intermediate image 1 from the first image sensor 4a and the distance information from the distance information calculation unit 8 are input to the third image restoration unit 9a. Similarly, the intermediate image 2 from the second image sensor 4b and the distance information from the distance information calculation unit 8 are input to the fourth image restoration unit 9b. Then, the image restoration units 9a and 9b extract the PSF data corresponding to the distance information from the distance information calculation unit 8 from the distance correspondence PSF storage unit 10, and restore the intermediate images 1 and 2 based on the extracted PSF data. The restored images 1 and 2 after restoration are output.

  The PSF data is calculated in advance by a predetermined optical design software or the like. For example, the distance-corresponding PSF is changed by changing the step for each imaging distance, such as every 20 mm step for 300 to 400 mm, and every 50 mm step for 400 to 500 mm. It is stored in the storage unit 10. As can be seen from equation (1) of the depth of field described above, the depth of field is deep on the long distance side and the depth of field is shallow on the short distance side, so rough steps on the long distance side, There are fine steps. The distance-corresponding PSF storage unit 10 can also store PSF data for each color such as RGB or for each angle of view.

  FIG. 14 is a schematic diagram for explaining an example of an image restoration method according to the present invention. As described above, the distance information calculation unit 8 calculates distance information representing the distance from the stereoscopic imaging device to the subject from the intermediate image 1 and the intermediate image 2. Then, the third image restoration unit 9a extracts PSF data corresponding to the distance information from the distance information calculation unit 8 from the distance correspondence PSF storage unit 10, and restores the intermediate image 1 based on the extracted PSF data. The restored image 1 is output. Although not shown in the figure, the PSF data corresponding to the distance information from the distance information calculation unit 8 is extracted from the distance-corresponding PSF storage unit 10 for the fourth image restoration unit 9b as well, and the extracted PSF is extracted. The intermediate image 2 is restored based on the data, and the restored image 2 is output. In this example, the description of the first image restoration unit 7a and the second image restoration unit 7b is omitted.

  The stereoscopic images (restored images 1 and 2) generated as described above may be displayed on a display unit (not shown) or may be recorded in a memory. In this way, by restoring an image using PSF, it is possible to perform focused 3D imaging from a short distance to a long distance.

DESCRIPTION OF SYMBOLS 1 ... Objective lens, 2 ... Aerial image, 3 ... Relay lens system, 4 ... Image sensor, 5, 7, 9 ... Image restoration part, 6 ... PSF memory | storage part, 8 ... Distance information calculation part, 10 ... Distance correspondence PSF memory | storage 11, 31, front group lens, 12, 32, stop, 13, 34, rear group lens, 33, pupil dividing means, 35, wavefront modulation element.

Claims (6)

One objective lens that forms an image of the subject in the air, a relay lens system that re-images the imaged aerial image, and a light beam that reaches the stop arranged in the vicinity of the stop of the relay lens system is divided into a plurality of parts Stereoscopic imaging for capturing a stereoscopic image by the plurality of imaging elements, and a plurality of imaging elements that are divided by the pupil dividing means and arranged on each imaging plane of the relay lens system A device,
A stereoscopic imaging apparatus, wherein a wavefront modulation element is disposed in the vicinity of the stop of the objective lens or in the vicinity of the stop of the relay lens system.   The stereoscopic imaging apparatus according to claim 1, wherein the pupil dividing unit is a reflecting surface of a prism disposed in the vicinity of the stop of the relay lens system.   The wavefront modulation element is configured as an asymmetric phase plate with respect to an optical axis, and the stereoscopic imaging device includes a PSF storage unit that stores a predetermined piece of PSF data, and the plurality of pieces based on the one piece of PSF data. The stereoscopic imaging apparatus according to claim 1, further comprising: a plurality of image restoration units that restore each image data imaged by the imaging element.   The wavefront modulation element is configured as a phase plate symmetric with respect to the optical axis, and the stereoscopic imaging device includes a distance-corresponding PSF storage unit that stores PSF data in association with distance information to a subject, and the plurality of imaging elements. A distance information calculation unit that calculates distance information from each captured image data, PSF data corresponding to the calculated distance information is extracted from the distance-corresponding PSF storage unit, and each image is based on the extracted PSF data The stereoscopic imaging apparatus according to claim 1, further comprising a plurality of image restoration units that restore data.   The stereoscopic imaging apparatus according to claim 2, wherein the reflecting surface of the prism has a curved shape, so that the reflecting surface functions as the wavefront modulation element. The reflecting surface shape of the prism is
Z = αX ³ + βY ³ , where α and β are coefficients, X is a distance in the direction perpendicular to the prism bending from the optical axis, Y is a distance in the prism bending direction from the optical axis,
The stereoscopic imaging device according to claim 5, wherein the stereoscopic imaging device has an aspheric shape defined by: