JPH11337313A - Distance measuring device and method thereof, and image restoration device and method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable a distance measuring device to perform stable and high- accuracy distance measuring without inconvenience by separating a light passing through a telecentric optical system to a plurality of lights and respectively taking in images at a focusing position being different from each other from separated lights to calculate a distance to an object. SOLUTION: The distance measuring device is provided with an opening mask 1 as a light-passing means, a lens system 2 converging lights entered from the opening mask 1, CCDs 3, 4, 5 which optical configurations are different from each other, a calculation part. The opening mask 1 having pin holes 6, 7 with coding opening shape (pupil-shape) for easy analysis of blurring quantity and the lens system 2 having a convex lens 8, a concave lens 9 and a concave lens 10 constitute the telecentric optical system where image magnification among 3 images obtained by CCDs 3, 4, 5 are respectively equal. Then, the calculation part compares and analyzes blurring contained the obtained images to determine a focusing position from the blurring quantity.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a distance measuring device and method for measuring the distance of an object.

[0002] The present invention also relates to an image restoration apparatus and method for restoring a focused image based on the distance measurement of such an object.

[0003]

2. Description of the Related Art Hitherto, many methods have been proposed for measuring a distance by using a relationship between a blur caused by a positional relationship between a lens and an image plane and a distance to a subject. A method used in autofocus cameras and the like. 2. Lens focus method. 3. A method that uses the relationship between the distance and the blur amount of the image instead of directly obtaining the in-focus position. 4. A method of measuring a three-dimensional shape using visible light. There is.

The method used in an autofocus camera or the like uses a dedicated optical system instead of a normal image pickup device for pupil division. In the lens focus method, an ordinary image sensor is used.In this case, an image at the best focus is obtained from an image obtained while driving the focus position of the lens, and a distance from the focus position at that time is obtained. Ask.

[0005] Of the methods using the relationship of the image blur amount to the distance instead of directly obtaining the focus position, the method proposed by Nayer of Columbia University uses a telecentric optical system and a double focus camera.
Furthermore, a method of measuring a three-dimensional shape using visible light is as follows.
It was proposed by Yamada of Osaka City University, etc.
This is an extension of the coded aperture method conventionally used in X-ray imaging technology.

[0006]

However, in the method used in an autofocus camera or the like, it is necessary to use a dedicated optical system instead of a normal image pickup device for pupil division. Alternatively, only the distance to several points can be obtained. The lens focus method requires sequential lens drive, and it is difficult to measure a moving object or perform real-time measurement using the lens drive method.

According to the method proposed by Nayer, modeling of lens aberrations, precise alignment of the CCD (charge coupled device) and elimination of noise from the obtained image, because a lens with a simple circular aperture is used Is required. In addition, since the size of the blur is also limited to a relatively small one, it is difficult to measure a long-distance object and secure depth accuracy.

The method proposed by Yamada et al. Does not optically extend the conventional coded aperture method at all, and it is difficult to obtain sufficient performance considering the area and noise performance of the existing CCD. is there. In particular, it is very difficult to measure a distant object, and only a minute object on the order of millimeters can be measured.

A first object of the present invention is to provide a distance measuring apparatus and a method capable of measuring a distance of an object stably and with high accuracy without causing the above-mentioned inconvenience.

A second object of the present invention is to provide an image restoration apparatus and method for restoring a perfectly focused image without causing the above-mentioned inconvenience.

[0011]

According to a first aspect of the present invention, there is provided a distance measuring apparatus having a pupil shape for allowing light to pass therethrough so that the blur amount can be easily analyzed, and the light passing means. A lens system that converges the light that has passed through, and an image capturing unit that separates the light converged by the lens system into two or more lights, and captures images at different focusing positions from these separated lights. And a distance calculating means for calculating the distance of the object using these images, wherein a telecentric optical system is constituted by the light passing means and the lens system.

In measuring the distance to the object, it is necessary to compare and analyze the blur contained in the obtained image, and to obtain the in-focus position from the amount. In an ordinary camera used for capturing an image, the light passage means often has a circular or nearly circular aperture in order to obtain a smooth and natural blur. However, with such a shape, the blur amount cannot be accurately estimated. In particular, the loss of information in the high frequency region of the spatial frequency lowers the distance resolution and makes it difficult to restore the original image after distance measurement. Therefore, the light passing means is structured so that the blur amount can be easily analyzed. As a result, information loss due to blurring can be minimized, and highly accurate distance measurement can be performed. In addition, sufficient performance can be easily obtained even in consideration of the area and noise performance of the existing CCD, and measurement of a long-distance object becomes easy. In the present specification, the pupil shape means a contractible aperture, that is, not only an aperture but also an aperture having an arbitrary shape without contractility.

Further, since there is no need to use a dedicated optical system for pupil division, there is no inconvenience that only a distance to one or several points on the screen can be obtained.

The image extracting means separates the light having passed through the light passing means into two or more lights, each having substantially the same light amount, and extracts from the separated lights images relating to objects having different depths. This enables analysis independent of the type of surface teletexture, and facilitates measurement of a moving object and real-time measurement without the need for sequential lens driving.

Further, since the telecentric optical system is constituted by the light passing means and the lens system, the image magnifications between a plurality of images taken out by the image taking out means become equal. In a normal optical system, the size of the image on the image differs depending on the position of the image plane because the light passing means is arranged near the principal point. On the other hand, in the telecentric optical system, by arranging the light passing means on the front focal plane, all the light beams passing through the light passing means and passing through the center of the light passing means are perpendicularly incident on the image plane. For this reason, a change in the position of the image plane causes only blurring of the image, and the image magnification does not change. Therefore, modeling of lens aberration, precise alignment of the CCD, and removal of noise from the obtained image are not necessary. Further, the size of the blur is not limited to a comparatively small one, and measurement of a long-distance object and securing of depth accuracy are facilitated.

As a result, by forming a telecentric optical system with such a light passing means and a light separating means, it is possible to measure the distance of an object stably and with high accuracy without causing the above-mentioned inconvenience.

In the distance measuring apparatus according to a second aspect of the present invention, the light passing means has two pinholes.

Since the light passing means has two pinholes, the analysis can be further facilitated and the calculation time can be reduced.

According to a third aspect of the present invention, in the distance measuring apparatus, the image capturing means may be arranged such that all surfaces are redeposited to remove spectral characteristics and the converged light is applied to the first surface. A first prism to be incident, a second prism having all surfaces re-evaporated to remove spectral characteristics and a first surface joined to a second surface of the first prism, A third prism having a first surface bonded to a second surface of the second prism and a second prism of the first prism, wherein all surfaces are redeposited to remove characteristics;
A first solid-state imaging device on which light reflected by the first surface and the first surface is incident, and a second solid-state image sensor on which light reflected by the second surface and the first surface of the second prism is incident. , And a third solid-state imaging device to which light passing through the first, second, and third prisms is incident, and the optical characteristics of the first, second, and third solid-state imaging devices. It is characterized in that the target arrangements are different from each other.

By means of such an image extracting means, it is possible to easily measure a moving object and real-time measurement without requiring sequential lens driving when measuring an object.
Further, such an image taking means is a normal 3-CCD
It can be obtained simply by processing the spectral prism block of the color camera so that the optical arrangement of each CCD is different from each other.

According to a fourth aspect of the present invention, in the distance measuring apparatus according to the fourth aspect, the distance calculating means obtains a focus position using an analysis method based on generation, and calculates a focus position based on the focus position and a focal length of the lens system. And calculating the distance of the object.

The distance calculation means obtains the in-focus position by using an analysis method based on the generation. That is, the function values for various in-focus positions are calculated in advance, compared with the values obtained from the image, and the in-focus position when the difference between them is the smallest is set as the estimated value. It is difficult to determine the focus position directly from the image,
By using the analysis method based on the generation, the in-focus position can be easily estimated. Using the focal position and the focal length of the lens system obtained in this way,
The distance of the object can be calculated based on the Gaussian imaging formula. This will be described later.

In the distance measuring method according to a fifth aspect of the present invention, a light passing means having a pupil shape for allowing light to pass so that the blur amount can be easily analyzed, and the light passing through the light passing means is converged. The light that has passed through the telecentric optical system constituted by the lens system is separated into two or more lights, images of different focusing positions are captured from the separated lights, and the distance of the object is determined using these images. Is calculated.

By using the telecentric optical system in this way, the distance measurement of the object can be performed stably and with high accuracy without causing the above-mentioned inconvenience.

According to a sixth aspect of the present invention, in the distance measuring method, the light passing means has two pinholes.

Since the light passing means has two pinholes, the analysis can be further facilitated and the calculation time can be reduced.

According to a seventh aspect of the present invention, in the distance measuring method, the light passing through the telecentric optical system is separated into three lights, and the separated lights are separated into first and second lights having different optical arrangements. The light is incident on the second and third solid-state imaging devices, respectively.

In this case, it is not necessary to sequentially drive the lens when measuring the object, and the measurement of the moving object and the real-time measurement can be easily performed.

In the distance measuring method according to the present invention, when calculating the distance of the object, a focus position is obtained by using an analysis method based on generation, and the focus position and the focus of the lens system are determined. The distance of the target is calculated based on the distance.

As described above, the focus position can be easily estimated by using the analysis method based on the generation, and therefore, the distance to the object can be easily calculated.

According to the ninth aspect of the present invention, there is provided an image restoring device having a pupil shape for allowing light to pass therethrough so as to facilitate analysis of a blur amount, and converging light passing through the light passing means. A lens system that separates the light converged by the lens system into two or more lights, and from these separated lights, an image capturing unit that captures images at different in-focus positions, and using these images. It is characterized by comprising a distance calculating means for calculating a distance of an object and a focused image restoring means for restoring a focused image, wherein a telecentric optical system is constituted by the light passing means and the lens system.

As a result, highly accurate distance measurement can be performed, and as a result, a perfectly focused image can be stably restored without blurring.

In the image restoration apparatus according to a tenth aspect of the present invention, the light passing means has two pinholes.

Since the light passing means has two pinholes, the analysis can be further facilitated and the calculation time can be reduced.

In the image restoring apparatus according to the present invention, the image capturing means may be arranged so that all surfaces are redeposited to remove spectral characteristics and the converged light is applied to the first surface. The first prism to be incident, and the first prism is redeposited on all surfaces to remove spectral characteristics.
And a second prism having a first surface joined to a second surface of the prism, and all surfaces being redeposited to remove spectral characteristics, and a first surface being attached to a second surface of the second prism. A third solid-state image sensor to which light reflected by the second surface and the first surface of the first prism is incident, and a third prism of the second prism A second solid-state imaging device on which light reflected by the second surface and the first surface is incident, and a third solid-state imaging device on which light passing through the first, second, and third prisms is incident With
The optical arrangement of the first, second and third solid-state imaging devices is different from each other.

With such an image extracting means, it is possible to easily measure a moving object and real-time measurement without requiring sequential lens driving when measuring an object.
Further, such an image taking means is a normal 3-CCD
It can be obtained simply by processing the spectral prism block of the color camera so that the optical arrangement of each CCD is different from each other.

In the image restoration apparatus according to a twelfth aspect of the present invention, the distance calculation means obtains a focus position using an analysis method based on generation, and calculates the focus position based on the focus position and the focal length of the lens system. And calculating the distance of the object.

As described above, the focus position can be easily estimated by using the analysis method based on the generation, and therefore, the distance to the object can be easily calculated.

According to a thirteenth aspect of the present invention, in the image restoration apparatus according to the thirteenth aspect, the in-focus image restoration means obtains a spatial frequency spectrum of an original image containing no blur from the in-focus position, and performs an inverse Fourier transform on the spectrum. In order to restore the in-focus image.

The in-focus image restoration means obtains the spatial frequency spectrum of the original image which does not include blur from the in-focus position, and restores the original image by inverse Fourier transform. Therefore, since the images are compounded in a signal processing manner instead of simply integrating the in-focus regions, even if any of the images taken out by the image taking means is out of focus, all the images are taken out of focus. A focused image can be obtained.

According to the image restoring method of the present invention, the light passing means having a pupil shape for passing the light so as to facilitate the analysis of the blur amount and the light passing through the light passing means are converged. The light that has passed through the telecentric optical system constituted by the lens system is separated into two or more lights,
From the separated light, images at different in-focus positions are fetched, respectively, and the distance of the target is calculated using these images, and the focused image is restored.

As a result, highly accurate distance measurement can be performed, and as a result, a perfectly focused image without blurring can be stably restored.

According to a fifteenth aspect of the present invention, in the image restoration method, the light passing means has two pinholes.

Since the light passing means has two pinholes, the analysis can be further facilitated and the calculation time can be reduced.

In the image restoring method according to the present invention, the light passing through the telecentric optical system is separated into three lights, and the separated lights are separated into first and second lights whose optical arrangements are different from each other. The light is incident on the second and third solid-state imaging devices, respectively.

In this case, it is not necessary to sequentially drive the lens at the time of measuring the object, and the measurement of the moving object and the real-time measurement can be easily performed.

According to the image restoration method of the present invention, when calculating the distance to the object, a focus position is obtained by using an analysis method based on generation, and the focus position and the focus of the lens system are determined. The distance of the target is calculated based on the distance.

By using the analysis method based on the generation as described above, the in-focus position can be easily estimated, and therefore, the distance to the object can be easily calculated.

In the image restoring method of the present invention, when restoring the original image, a spatial frequency spectrum of a focused image that does not include blur is obtained from the focused position, and this is inversely Fourier transformed. And restoring the in-focus image.

Therefore, even if none of the extracted images is in focus, an all-focus image can be obtained.

[0051]

DESCRIPTION OF THE PREFERRED EMBODIMENTS A distance measuring apparatus and method and an image restoring apparatus and method according to the present invention will be described in detail with reference to the drawings. FIG. 1 is a diagram showing a distance measuring device according to the present invention. The distance measuring device includes an aperture mask 1 as a light passing means, a lens system 2 for converging light incident from the aperture mask 1, CCDs 3, 4, and 5 having optical arrangements different from each other, And a calculation unit as a distance calculation means.

The aperture mask 1 has a coded aperture shape (pupil shape) structured so as to facilitate the analysis of the blur amount. Specifically, it has two pinholes 6 and 7. The lens system 2 includes a convex lens 8, a concave lens 9, and a convex lens 10.
Having.

An arithmetic unit (not shown) estimates the in-focus position based on the images taken from the CCDs 7, 8, and 9. This operation will be described later in detail.

The aperture mask 1 and the lens system 2 are CCD
With respect to the three images obtained by 7, 8, and 9, a telecentric optical system in which the image magnification therebetween is equal is configured. In the case of a normal optical system, as shown in FIG. 2A, in order to dispose the stop 11 near the principal point of the lens system 12, C
The magnification changes according to the position change of the CDs 13 and 14.
On the other hand, in the case of the telecentric optical system, FIG.
By arranging the stop 15 at the front focal plane of the lens system 16 as shown in FIG. For this reason, a change in the position of the CCDs 17 and 18, ie, the image plane, causes only blurring of the image, and does not change the image magnification. Thus, when the distance to the target is constant, the blur reduction can be expressed as a position-invariant kernel folding operation, and the analysis is simplified.

In order to measure a distance using such a distance measuring device, blurs included in the obtained images are compared with each other.
It is necessary to analyze and obtain the in-focus position from the amount. In an ordinary camera used for capturing an image, the pupil shape often has a circular or nearly circular aperture in order to obtain a smooth and natural blur. However, in such a pupil shape, the gain of the high-frequency component simply decreases monotonously with the occurrence of the blur, and the magnitude of the blur has to be estimated from the quantitative analysis. In addition, the loss of high frequency components makes it difficult to restore the original image. Therefore, it is necessary to structure the pupil shape so that the analysis is easy, and to develop an analysis method suitable for the pupil shape.

The pupil shape is required to satisfy the following conditions. 1. The size of the blur is easily obtained from the structure of the frequency characteristics. 2. The gain in the high-frequency region does not drop overall regardless of the size of the blur. 3. Have an opening area that allows a sufficient amount of light to pass through. What is important here is that since images having different amounts of blur are obtained from a plurality of arranged imaging planes, the zero point may be locally included in the gain characteristic of the structured pupil shape. is there. Rather, by structuring gain characteristics such as periodically providing zeros, it is possible to use a technique such as pattern matching for blur amount analysis, and to greatly improve the accuracy of distance measurement.

FIG. 3 is a diagram showing a part of a multi-focus camera used for a distance measuring device. This multi-focus camera includes a coded aperture mask (not shown), a lens 19 corresponding to the lens system 2 in FIG.
1, 22 and CCDs corresponding to CCDs 3, 4, and 5 in FIG.
23, 24, and 25. These prisms 20, 2
All surfaces 1, 2 have been redeposited to remove spectral properties. In this case, the first of the prisms 20 is arranged such that the optical arrangements of the CCDs 23, 24 and 25 are different from each other.
The optical path length from the surface 26 to the CCD 23 is 1 mm compared to the optical path length from the first surface 26 of the prism 20 to the CCD 24.
The distance from the first surface 26 of the prism 20 to CC
The optical path length up to D25 is made 1 mm longer than the optical path length from the first surface 26 of the prism 20 to the CCD 24.

The prism 21 is provided on the second surface 27 of the prism 20.
And the second surface of the prism 21
The first surface 30 of the prism 22 is joined to the surface 29 and the CCD 2
The incident light on the lenses 3, 24, and 25 are respectively set to be 1/3 of the incident light on the first surface 26 of the prism 20 from the lens 19.

By shifting the CCDs 23, 24 and 25 in the direction perpendicular to the image plane in this manner, three images at different focus positions can be simultaneously captured. In addition, such a multi-focus camera is provided with a normal 3-C
It can be obtained simply by processing the spectral prism block of the CD type color camera so that the optical arrangement of each CCD is different from each other.

Until now, no method has been proposed in which a multicentric camera is integrated with a telecentric optical system and a coded aperture (pupil shape). However, high performance as described later can be obtained only by using these three at the same time.

Next, it is easy to understand how to analyze the three images obtained by the multi-focus camera to measure the distance to each point on the object (these points are represented by an xy coordinate system). This will be described with reference to FIGS. 2B and 4. In FIG. 4, the light passing means is indicated by an opening mask 33 having two pinholes 31 and 32,
The lens system is shown by a convex lens 34, and the image taking out means is CC
D35, 36 and 37 are shown. In order to form a telecentric optical system, the aperture mask 33 must be
At the front focal plane.

In the telecentric optical system shown in FIG. 2B, the stop 15 is stopped down, and an image free from blur, which is obtained when only the center light beam passes therethrough, is obtained as s (x,
y). This image s (x, y) is the same regardless of the position of the imaging surface. Further, the pupil shape of the aperture 15 is a
Assuming that (x, y) and the obtained image is i (x, y), these relationships are expressed by the following equations.

(Equation 1)

Thus, the fact that the change in image due to focusing is simply expressed by convolution of the blur kernel with respect to the original image is the mathematical basis for using a telecentric optical system instead of a normal optical system. Here, the blur kernel is figure similar to the pupil shape, the ratio k _m
(M = 1, 2, 3) indicates that the distance from the lens 34 to the target point P is u, the focal length of the lens 34 is f, the in-focus position of the lens 34 is v, and the distance from the lens to each CCD is If the position is referred to as w _m, the ratio k _m can be expressed as follows from FIG.

(Equation 2)

Here, Gaussian imaging formula

(Equation 3) Clearing the distance u using the ratio k _m is

(Equation 4) It is represented by From [Equation 4], the ratio k _m is found to be a difference in position w _m of the focus position v and the CCD is a linear value. Obviously, when the in-focus position v is obtained, the distance u is obtained from [Equation 3]. Focus state, i.e., v = k _m = 0 next when the w _m, this time, the blur kernel is a delta function. Of _{course, i m (x, y)} = s (x, y), i.e., an image having no blurring is observed.

The blur forming process represented by [Equation 1] is a property that is established irrespective of the pupil shape. Here, the pupil shape a
(X, y) Setting the observed image i _m (x,
The focus position v and the perfectly focused image s (x, y) can be obtained from y).

The blur kernel caused by the pupil shape shown in FIG.

(Equation 5) Represented by Here, δ (x) is the Dirac δ function. Substituting this into [Equation 1] gives

(Equation 6) Becomes When this is one-dimensional Fourier transformed in the x-axis direction,

(Equation 7) become that way. That is, the gain of the frequency characteristic of the blur by the distance measuring device having two pinholes is represented by the absolute value of the cos function, and the period is determined by the blur amount.

Since [Equation 7] is a real function,
The phase component is either 0 or π. From these, it can be seen that the distance measuring device having two pinholes satisfies the above conditions 1 and 2. Since a plurality of blurred images are obtained, by appropriately weighting the spectral distributions obtained therefrom, the existence of a zero point does not matter in both the distance measurement and the fully focused image restoration described later.

In the case of a normal circular aperture, a function of a low-pass filter for reducing the high frequency component of the frequency characteristic as the amount of blur increases is provided. By structuring the pupil shape, the frequency characteristic of the blur can be reduced. It turns out that it can be structured.

In such a distance measuring device, a plurality of images relating to different blur amounts can be obtained at the same time. These one-dimensional Fourier transforms for x are represented by I
_m (s, y)

(Equation 8) Becomes Here, the position of each image plane is defined as w _m, and S (s,
Let y) be the spatial frequency spectrum of perfect focus (without blur). This is unknown but common to all image planes. Therefore, this term is eliminated by dividing [Equation 8] with each other. For example, _Im (s,
If taking I _n (s with taking y) in the molecule, y) and (n a, and any of integers different from 1 to 3 and m.) in the denominator,

(Equation 9) It is sufficient to find v that minimizes. However, the first term is a value obtained from the Fourier transform of the input image, which is a complex value, and it is difficult to directly obtain the in-focus position v using the value. Therefore, an analysis method based on generation is used. FIG. 5 shows the flow of this processing. First, the input images I ₁ ,
An image to be processed is extracted from one scan line of each of I ₂ and I ₃ , and Blackman window processing is performed. Next, a first-order Fourier transform for x is performed, and the obtained spectrum is divided for each spatial frequency s between adjacent image planes.

In this case, since there are three image planes, all combinations in which the denominator and numerator are exchanged, that is, r ₁₂ r ₂₁
Find the sum of r ₂₃ r ₃₂ . [Equation 9] includes division, but actually solves the problem related to the zero point of the frequency response by calculating the weighted sum. The weighting utilizes the absolute value of I _n (s, y) used for the denominator. If the sum of the weighting coefficients does not reach the threshold value, it is determined that distance measurement is impossible because the region does not include a sufficient texture. Naturally, the distance u is obtained from the in-focus position v obtained by the analysis method based on generation and [Equation 3].

FIG. 6 shows the sum of the differences when different focus positions v ₁ and v ₂ are substituted into the second term of [Equation 9]. As shown in the figure, a very steep valley-like evaluation value is obtained around v ₁ = v ₂ , and the remaining position is an even residual, so that distance calculation can be performed with high accuracy. It can be seen that it is.

According to the present embodiment, the use of the multi-focus camera enables real-time measurement without having a mechanical drive part. Further, since the aperture mask has a structure that allows light to pass so as to facilitate the analysis of the blur amount, it is possible to efficiently reduce the blur caused by the pupil shape. Further, a telecentric optical system is constituted by the aperture mask and the lens system, that is, by arranging the aperture mask on the front focal plane of the lens system, the distance measurement of the object can be performed stably and accurately without causing the above-described inconvenience. Can be.

Next, an image restoration apparatus using a distance measuring device will be described. In this case, in addition to the components (including the calculation unit) of the distance measurement device as shown in FIG. 1, the image restoration device includes an image restoration unit (FIG. 1) as a focused image restoration unit for restoring a focused image. (Not shown).

In order to actually input an image, the opening area needs to be a finite value. However, in the description so far, the diameter of the pinhole is ignored, and the effect of blurring due to the diameter of each pinhole is not included. Thus, in the actual calculation, taking this effect into account, the spectral response of the blurred image represented by [Equation 7] is multiplied by the sinc function, and

(Equation 10) And Where β is a value determined by the diameter of the pinhole, which is significantly smaller than α,
The period of the sinc function is smaller than the period of the cos function. When β = 0, it is an ideal pinhole case. The value of β used in the calculation was determined from the size of the pupil shape together with α. Since the second term of [Equation 9] is made into a table in advance to increase the speed, this calculation may be performed at the time of template generation.

In order to restore a perfectly focused image, using the obtained distance v,

[Equation 11] , And perform its inverse Fourier transform. Naturally, S (s, y) is common for all m, so [Equation 1
By obtaining the average of m using the absolute value of the denominator of [1] as a weight, a perfectly focused image can be correctly restored even when a zero point is included in the frequency response of the blur.

Next, a distance map obtained by actual measurement and a perfectly focused image which does not include blur will be described. FIG.
Reference numeral 9 denotes three images obtained by the distance measuring device. From these, it can be seen that images with different amounts of blur have been obtained. It can be seen that none of these images are perfectly in focus for the background text.
FIG. 10 is a perfectly focused image restored as described above.

FIG. 11 shows the three objects obtained by the calculation.
It is a three-dimensional shape and a perfectly focused image. According to this, it can be seen that the three-dimensional shape of the plane portion of the background, the clock in front, and the distance between them are measured. In the distance map, a portion that is missing in white is a portion that does not include any clue patterns or the like in all objects.

FIGS. 12 and 13 show an enlarged image of a part of the input image captured by the central CCD and an enlarged image of a part of the perfectly focused image, respectively.
In Photo F, both the clock in front and the characters in the background can be clearly seen. Since the image is restored by signal processing instead of simply integrating the in-focus areas, a completely in-focus image can be obtained even when none of the input images is in focus. In an experiment measured for an object having a similar text texture on a plane, the standard deviation of the depth value was about 1.2 mm for an object about 1.1 mm away. The calculation time is DEC Alpha 400M
Using Hz, it took about 30 seconds to calculate the distance and about 20 seconds to restore a perfectly focused image.

According to the present embodiment, the information of the image is hardly lost, and it is possible to stably restore a perfectly focused image without blurring.

Next, a case where the image restoration apparatus according to the present invention is applied to a range image sensor will be described. Such an image restoration apparatus aims to obtain a three-dimensional shape of an object and an all-focus image that does not include blur by photographing the object in the same manner as a normal television camera. As shown in FIG. 14A, an image 4 on a focal plane 39 taken by a normal camera 38 that performs focusing in the direction of arrow A.
0 indicates that information on the depth of the object 41 is included in the image of the object 41 as a blur 42.
It is difficult to separate from zero.

On the other hand, the distance image sensor 43 to which the image restoration device according to the present invention is applied can separate and extract the omnifocal image 45 and the distance image 46 of the object 44 as shown in FIG. 14B. Originally, the distance image 46 has been used in the field of surveying / measurement, industrial automation, etc.
Simultaneously acquiring the omnifocal image 45 is particularly expected to be applied in a multimedia field such as a CG synthesized image 47 obtained by synthesizing a computer graphics (CG) image and a real image, and a stereoscopic television.

In the case of such a distance image sensor 43, by performing the measurement without the mechanical driving part, the time required for the measurement is about 1/30 second, which is equivalent to that of the normal camera 38. This enables measurement of the moving object. Also,
By performing measurement without projecting a laser beam or the like at all, measurement is possible even in a general living environment or an outdoor environment. These are the performances required for use in the multimedia field.

The present invention is not limited to the above embodiment, and many modifications and variations are possible. For example,
In the above embodiment, only the case where the light passing means has two pinholes has been described. However, the light passing means may have another arbitrary structure so that the blur amount can be easily analyzed.

Further, as the lens system, only the case where the convex lens, the concave lens and the convex lens are arranged in order and the case where the lens system is constituted by a single convex lens have been described, but any other lens system may be used.

Although the description has been given of the case where the incident light is divided into three, the present invention can be applied to the case where the incident light is separated into a plurality of other lights. CC described in the above embodiment
Other than the optical arrangement of D can be adopted.

Furthermore, the case where the image restoring device according to the present invention is applied to a range image sensor has been described, but this is applied to an industrial range finder, a stereoscopic television camera,
The present invention can be applied to an industrial field for performing three-dimensional image measurement such as a security monitoring system and an image media field. The industrial range finder measures the position and orientation of a component or the like and recognizes the type, and is used for gripping a component by a robot. According to the stereoscopic television camera, a binocular parallax image required for stereoscopic vision can be generated by processing the obtained distance image by a computer. According to the security monitoring system, it is possible to recognize not only the image of a person but also the position and movement of the person in space.

[Brief description of the drawings]

FIG. 1 is a diagram showing a distance measuring device according to the present invention.

FIG. 2 is a diagram showing a normal optical system and a telecentric optical system.

FIG. 3 is a diagram showing a part of a multi-focus camera used in the distance measuring device according to the present invention.

FIG. 4 shows a telecentric optical system with two pinholes.

FIG. 5 is a diagram showing a flow of a distance estimation process.

FIG. 6 is a diagram showing the sum of residuals of frequency characteristics for different focusing positions v ₁ and v ₂ .

FIG. 7 is a first image obtained by a distance measurement device.

FIG. 8 is a second image obtained by the distance measurement device.

FIG. 9 is a third image obtained by the distance measurement device.

FIG. 10 is a restored perfect in-focus image.

FIG. 11 shows a three-dimensional shape and a perfectly focused image of a target obtained by calculation.

FIG. 12 is an enlarged view of a part of an image captured by a central CCD among input images.

FIG. 13 is an enlarged view of a part of a perfectly focused image.

FIG. 14 is a diagram for explaining a range image sensor to which a normal camera and an image restoration device according to the present invention are applied.

[Explanation of symbols]

1,33 aperture mask 2,12,16 lens system 3,4,5,23,24,25,35,36,37 C
CD 6,7,31,32 Pinhole 8,10,34 Convex lens 9 Concave lens 19 Lens 20,21,22 Prism 26,28,30 First surface 27,29 Second surface 38 Normal camera 39 Focusing surface 40 Image 41, 44 Object 42 Blur 43 Distance image sensor 45 All-in-focus image 46 Distance image 47 CG composite image f Focal length of lens 34 I ₁ , I ₂ , I ₃ Input image P Target point u From lens 34 to target point P Distance v Focusing position of lens 34 w ₁ , w ₂ , w ₃ Position from lens 34 to each CCD

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] May 20, 1999

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Claim 1

[Correction method] Change

[Correction contents]

[Procedure amendment 2]

[Document name to be amended] Statement

[Correction target item name] Claim 2

[Correction method] Change

[Correction contents]

[Procedure amendment 3]

[Document name to be amended] Statement

[Correction target item name] Claim 5

[Correction method] Change

[Correction contents]

[Procedure amendment 4]

[Document name to be amended] Statement

[Correction target item name] Claim 9

[Correction method] Change

[Correction contents]

[Procedure amendment 5]

[Document name to be amended] Statement

[Correction target item name] Claim 14

[Correction method] Change

[Correction contents]

[Procedure amendment 6]

[Document name to be amended] Statement

[Correction target item name] 0011

[Correction method] Change

[Correction contents]

[0011]

According to a first aspect of the present invention, there is provided a distance measuring apparatus, comprising: a light passing means structured to facilitate analysis of a blur amount; and a light passing through the light passing means. A lens system that converges, and the light converged by this lens system is separated into two or more lights, and from these separated lights,
Image capturing means for capturing images at different in-focus positions, and distance calculating means for calculating a distance to a target using these images, wherein a telecentric optical system is configured by the light passing means and the lens system. It is a feature.

[Procedure amendment 7]

[Document name to be amended] Statement

[Correction target item name] 0016

[Correction method] Change

[Correction contents]

As a result, by forming a telecentric optical system with such a light passing means and a lens system, it is possible to measure the distance of an object stably and accurately without causing the above-mentioned inconvenience.

[Procedure amendment 8]

[Document name to be amended] Statement

[Correction target item name] 0023

[Correction method] Change

[Correction contents]

The distance measuring method according to a fifth aspect of the present invention comprises a light passing means structured so that the blur amount can be easily analyzed, and a lens system for converging light passing through the light passing means. The light that has passed through the telecentric optical system is separated into two or more lights, images of different focusing positions are captured from the separated light, and the distance of the object is calculated using these images. It is a feature.

[Procedure amendment 9]

[Document name to be amended] Statement

[Correction target item name] 0031

[Correction method] Change

[Correction contents]

According to a ninth aspect of the present invention, there is provided an image restoring apparatus, comprising: a light passing means structured to facilitate analysis of a blur amount; a lens system for converging light passing through the light passing means; An image capturing unit that separates the light converged by the lens system into two or more lights, and captures images at different focusing positions from the separated lights, and calculates a target distance using the images. And a focused image restoring means for restoring a focused image, wherein a telecentric optical system is constituted by the light passing means and the lens system.

[Procedure amendment 10]

[Document name to be amended] Statement

[Correction target item name] 0041

[Correction method] Change

[Correction contents]

According to a fourteenth aspect of the present invention, there is provided an image restoring method comprising a light transmitting means structured to facilitate analysis of a blur amount, and a lens system for converging light passing through the light transmitting means. The light that has passed through the telecentric optical system is separated into two or more lights, images of different focus positions are captured from the separated lights, and the distance to the object is calculated using these images. It is characterized in that a focused image is restored.

[Procedure amendment 11]

[Document name to be amended] Statement

[Correction target item name] 0043

[Correction method] Change

[Correction contents]

According to a fifteenth aspect of the present invention, in the image restoration method according to the fifteenth aspect, the light passing means has two pinholes.

Claims (18)

[Claims] 1. A light passing means having a pupil shape for passing light so as to facilitate analysis of a blur amount, a lens system for converging light passing through the light passing means, and a lens system converged by the lens system. Image capturing means for separating the light into two or more lights and capturing images at different focusing positions from the separated lights, and distance calculating means for calculating the distance to the object using these images. A distance measuring device, wherein a telecentric optical system is constituted by the light passing means and a lens system. 2. The distance measuring device according to claim 1, wherein said light passing means has two pinholes. 3. The image capturing means comprises: a first prism on which all surfaces are redeposited to remove spectral characteristics, and wherein the converged light is incident on a first surface; A second prism having a first surface bonded to a second surface of the first prism while all surfaces are redeposited for removal, and a redevaporation of all surfaces for removal of spectral characteristics. And a third prism having a first surface joined to a second surface of the second prism, and light reflected by the second surface and the first surface of the first prism are incident thereon. A first solid-state image pickup device, a second solid-state image pickup device into which light reflected by the second surface and the first surface of the second prism is incident, and the first, second and third solid-state image pickup devices. And a third solid-state imaging device on which light passing through the prism is incident. First, claim 1, the optical arrangement of the second and third solid-state imaging device is characterized in that so as to differ from each other
Or the distance measuring device according to 2. 4. The apparatus according to claim 1, wherein the distance calculating means calculates an in-focus position using an analysis method based on generation, and calculates the distance to the object based on the in-focus position and the focal length of the lens system. The distance measuring device according to any one of claims 1 to 3, wherein: 5. A light-transmitting means having a pupil shape for transmitting light so as to facilitate analysis of a blur amount, and a light passing through a telecentric optical system constituted by a lens system for converging light passing through the light transmitting means. A distance measuring method, comprising separating light into two or more lights, capturing images at different focusing positions from the separated lights, and calculating a distance to a target using the images. 6. The distance measuring method according to claim 5, wherein said light passing means has two pinholes. 7. The light that has passed through the telecentric optical system is separated into three lights, and these separated lights are respectively incident on first, second, and third solid-state imaging devices having different optical arrangements. The distance measuring method according to claim 5 or 6, wherein: 8. When calculating the distance to the object, a focus position is obtained using an analysis method based on generation, and the distance to the object is calculated based on the focus position and the focal length of the lens system. The distance measuring method according to any one of claims 5 to 7, wherein: 9. A light passing means having a pupil shape for allowing light to pass therethrough so as to facilitate analysis of a blur amount, a lens system for converging light passing through the light passing means, and a lens system converged by the lens system. Image capturing means for separating light into two or more lights, and capturing images at different focus positions from the separated lights, and distance calculating means for calculating a distance to a target using these images; An image restoration apparatus comprising: a focused image restoration unit for restoring a focused image; and a telecentric optical system is configured by the light passing unit and a lens system. 10. The image restoration apparatus according to claim 9, wherein said light passing means has two pinholes. 11. The image capturing means comprises: a first prism on which all surfaces are redeposited to remove spectral characteristics, and wherein the converged light is incident on a first surface; A second prism having a first surface bonded to a second surface of the first prism while all surfaces are redeposited for removal, and a redevaporation of all surfaces for removal of spectral characteristics. And a third prism having a first surface joined to a second surface of the second prism, and light reflected by the second surface and the first surface of the first prism are incident thereon. A first solid-state image pickup device, a second solid-state image pickup device into which light reflected by the second surface and the first surface of the second prism is incident, and the first, second and third solid-state image pickup devices. And a third solid-state imaging device on which light passing through the prism is incident. Et first claim 9 in which the optical arrangement of the second and third solid-state imaging device is characterized in that so as to differ from each other
Or the image restoration device according to 10. 12. The apparatus according to claim 1, wherein the distance calculating means obtains a focus position using an analysis method based on generation, and calculates a distance to the object based on the focus position and a focal length of the lens system. The image restoration device according to any one of claims 9 to 11, wherein: 13. The in-focus image restoring means obtains a spatial frequency spectrum of an original image which does not include blur from the in-focus position, and restores the in-focus image by performing Fourier inverse transform on the spectrum. 13. The image restoration device according to claim 12, wherein: 14. A light-passing means having a pupil shape for allowing light to pass therethrough so as to facilitate the analysis of a blur amount, and a light passing through a telecentric optical system constituted by a lens system for converging light passing through the light-passing means. It separates light into two or more lights, captures images at different in-focus positions from these separated lights, calculates the target distance using these images, and restores the focused image. Characteristic image restoration method. 15. The image restoration method according to claim 14, wherein said light passing means has two pinholes. 16. The light that has passed through the telecentric optical system is separated into three lights, and these separated lights are respectively incident on first, second, and third solid-state imaging devices having different optical arrangements. The image restoration method according to claim 14 or 15, wherein: 17. When calculating the distance to the object, a focus position is obtained using an analysis method based on generation, and the distance to the object is calculated based on the focus position and the focal length of the lens system. The image restoration method according to any one of claims 14 to 16, wherein: 18. When restoring the original image, a spatial frequency spectrum of a focused image that does not include blur is obtained from the focus position, and the Fourier inverse transform is performed on the spatial frequency spectrum to restore the focused image. The image restoration method according to claim 17, wherein