JP2002218505A - Three-dimensional image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a three-dimensional image pickup device that can execute distance data calculation with higher precision on the basis of a pattern image photographed by a plurality of cameras. SOLUTION: A light emitting element and a light receiving element are formed on a semiconductor chip adjacent to each other, the light emitting element on the semiconductor chip projects a pattern for measuring a three- dimensional shape and the light receiving element on the chip picks up the image of the pattern on the optical axis. When the field angle of a projector and the field angle of a camera are coincident within the precision of a pixel size and even when there exists forward/backward discontinuity in the measurement object, a video image of a shade causing mis-recognition as a code is not caused in the image photographed by the camera. Thus, the re-coding accuracy of the three-dimensional image pickup device adopting a re-coding method can be enhanced.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

[0001] 1. Field of the Invention [0002] The present invention relates to a three-dimensional image pickup apparatus for generating three-dimensional image information from luminance information and distance information. In particular, a three-dimensional image based on a triangulation method in which pattern projection images obtained by irradiating pattern light on a measurement target are imaged from different directions by a plurality of imaging means and distance information is obtained based on a change in the pattern. The present invention relates to an imaging device.

[0002]

2. Description of the Related Art A technique for acquiring a three-dimensional shape includes an active technique (Active vision) and a passive technique (Passive v).
ision). The active method uses (1) a method that emits laser light or ultrasonic waves to measure the amount of light reflected from an object and the arrival time to extract depth information, and (2) a special pattern light source such as slit light. A pattern projection method for estimating a target shape from image information such as a geometric deformation of a target surface pattern, and (3) a method of obtaining three-dimensional information by forming contour lines by moiré fringes by optical processing. is there. On the other hand, the passive method is based on
There are monocular stereoscopic vision in which three-dimensional information is estimated from one image using knowledge about illumination and shadow information, and binocular stereoscopic vision in which depth information of each pixel is estimated based on the principle of triangulation.

In general, the active method has higher measurement accuracy, but in many cases, the measurable range is small due to limitations of light emitting means. On the other hand, the passive method is versatile and has few restrictions on objects, but generally has low measurement accuracy. The present invention relates to a pattern projection method, which is a three-dimensional measuring device of the active method.

In the pattern projection method, a reference pattern light such as a linear vertical stripe pattern is projected onto a target object, and photographing is performed from a direction different from the direction in which the reference pattern light is projected. . The photographed pattern is deformed by the shape of the object, and becomes a deformed pattern of, for example, a curved stripe pattern. By associating the observed deformation pattern with the projected pattern, three-dimensional measurement of the object can be performed.

[0005] In the pattern projection method, the problem is how to easily associate the deformed pattern with the projected projection pattern with less erroneous correspondence. For example, when there are large discontinuous irregularities on the surface of the object, the photographed deformation pattern is interrupted on the way, and it is difficult to associate with the projection pattern. Therefore, various well-known pattern projection methods (such as a spatial pattern coding method and a color coding method) have been proposed.

As an example, the spatial pattern coding method uses, for example, a plurality of vertical stripes of a plurality of gradations as a projection pattern. The arrangement of the gradations makes it easy to find the corresponding projection pattern even in the discontinuously cut portion of the deformation pattern.

However, even with these conventional methods, it is often difficult to associate a correct code depending on the object to be photographed, such as when the luminance (reflection coefficient) of the surface of the object changes greatly depending on the location.

Therefore, the present inventors first disclosed in JP-A-2000-2000.
In 65542, a three-dimensional image capturing apparatus capable of dramatically improving the accuracy of associating a photographed deformation pattern with a projected projection pattern and obtaining a three-dimensional image with high accuracy has been proposed.

[0009] Japanese Patent Application Laid-Open No. 2000-65542 discloses a camera that shoots from a direction different from the projection direction of a projection pattern, a second camera or a third camera, and a light beam that is the same as the optical axis of the projector of the projection pattern. A first camera is further provided on the axis. The first camera photographs a pattern of the target object photographed in the same optical axis direction as the projection pattern. This pattern taken by the first camera,
That is, a pattern photographed from the same direction as the projection pattern has a distribution of luminance (reflection coefficient) on the surface of the target object,
The resulting image has changed luminance.

At the same time, the luminance distribution of the pattern photographed by the first camera also corresponds to the luminance distribution of the deformation pattern photographed by the second camera on an optical axis different from that of the projector.

Next, the projection pattern is re-coded according to the pattern captured by the first camera. Thereafter, instead of the projection pattern, a pattern captured by the first camera, that is, a re-coded pattern is used as a comparison target of the deformed pattern.

According to this configuration, even when the luminance of the surface of the target object, that is, the reflection coefficient is largely changed and distributed, the pattern taken by the first camera including the influence of the distribution is used as a reference. Therefore, the accuracy of associating the projection pattern with the deformation pattern is dramatically improved.

[0013]

However, even in the above-mentioned prior art, there is a case where accuracy of associating the projection pattern with the deformation pattern is insufficient. This is the case, for example, when the distance between the target object and the background before and after is large, when the unevenness of the target object in the front-back direction is large, or when a plurality of objects overlap in the front-back direction. In such a case, the projection light from the projector is viewed from the front of the object,
It does not reach the back. The part behind this is shaded.

At this time, the light projector for projecting the projection light and the first camera arranged on the same optical axis generally have different angles of view even though they have the same optical axis. For this reason, in an image captured by the first camera, a fine shadow image is generated at a discontinuous portion before and after the object, that is, at an edge portion. The shadow image generally has a thin black belt-like pattern. This is recognized as a new error code during recoding. It is recognized and coded as a new pattern, usually at the darkest level. For this reason, there is an obstacle in performing recoding corresponding to the projection pattern.

Here, the shadow image becomes a problem when its width is larger than the resolution dimension of the first camera. More specifically, in the image projected on the surface of the image sensor of the first camera, the width of the shadow image is
It is considered that the image is generated as an image when the size is larger than the size of the pixel of the image sensor.

SUMMARY OF THE INVENTION The present invention has been made in view of the above-mentioned problems of the prior art. In a three-dimensional image pickup apparatus which obtains distance data by projecting a pattern, the present invention is applied to a case where a back-and-forth direction of a target object is generated. It is another object of the present invention to provide a three-dimensional image capturing apparatus capable of accurately associating a projection pattern with a deformation pattern. In particular, in the three-dimensional image capturing apparatus to which the above-described re-encoding method is applied, even when the front-back direction of the target object is generated, a three-dimensional image can be acquired with extremely high accuracy.
It is an object to provide a two-dimensional image capturing device.

[0017]

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned object, and a first aspect of the present invention is to provide an image obtained by photographing a pattern image obtained by projecting a coded pattern onto a measurement target. A three-dimensional image capturing apparatus that obtains distance information of a measurement target based on a light projector that projects a coded pattern; a first camera that captures a projection pattern by the light projector from an optical axis direction of the light projector; A second camera that captures the projection pattern from a direction different from the optical axis direction of the light projector, and assigns a new code to a region where the amount of change in the capture pattern by the first camera with respect to the projection pattern is equal to or greater than a predetermined value; Using the allocated code, first distance information is generated from a shooting pattern of the second camera, and the first distance information and the first camera are used to generate first distance information. It has a re-coding process and distance information obtaining means for obtaining distance information based on the obtained luminance information, and the projector and the first camera are configured to have substantially the same angle of view. In a three-dimensional image pickup device.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, a difference in the angle of view between the light projector and the first camera is equal to or smaller than the resolution size of the first camera. .

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is constituted by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. It is characterized in that it is constituted by a light receiving element.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is constituted by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. In addition to the configuration including the light receiving element, the surface light emitting element and the light receiving element are formed on the same semiconductor substrate.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is constituted by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. A semiconductor substrate on which the light-emitting element of the light projector is formed, and a semiconductor substrate on which the light-receiving element of the first camera is formed, which are adhered to each other to form one semiconductor substrate. It is characterized by being formed in.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is constituted by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. A light-receiving element, wherein the surface light-emitting element and the light-receiving element are adjacent to each other to form a set of light-emitting and light-receiving elements, and the light-emitting and light-receiving element sets are arranged in an array. And wherein the projector and the first camera are formed.

Further, a second aspect of the present invention relates to a three-dimensional image pickup apparatus for acquiring distance information of a measurement target based on an image of a pattern image obtained by projecting a coded pattern onto the measurement target. A projector that projects the coded pattern, a first camera that captures a projection pattern by the projector from the optical axis direction of the projector, and a second camera that captures the projection pattern from a direction different from the optical axis direction of the projector. A new code is assigned to an area in which the amount of change of the photographing pattern by the first camera with respect to the projection pattern is equal to or greater than a predetermined value,
A re-code for generating first distance information from a pattern captured by a second camera using the allocated code, and obtaining distance information based on the first distance information and luminance information obtained by the first camera. A three-dimensional image pickup device, characterized in that the three-dimensional image pickup device has a configuration in which the light projector and the first camera are integrated.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is formed by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. In addition to the configuration including the light receiving element, the surface light emitting element and the light receiving element are formed on the same semiconductor substrate.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is constituted by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. A semiconductor substrate on which the light-emitting element of the light projector is formed, and a semiconductor substrate on which the light-receiving element of the first camera is formed, which are adhered to each other to form one semiconductor substrate. It is characterized by being formed in.

Further, in one embodiment of the three-dimensional image pickup device of the present invention, the light projector is constituted by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed on the semiconductor substrate. A light-receiving element, wherein the surface light-emitting element and the light-receiving element are adjacent to each other to form a set of light-emitting and light-receiving elements, and the light-emitting and light-receiving element sets are arranged in an array. And wherein the projector and the first camera are formed.

[0027]

In the three-dimensional image pickup apparatus according to the present invention, the light projector and the first camera are formed so as to have the same optical axis and the same angle of view. For this reason, even when a shadow in the front-rear direction of the target object occurs, the ray vector of the projection light projected on a certain point on the target object and the photographing ray vector of the first camera for photographing the above point Are equal to each other.

For this reason, even if there is a discontinuity in the front and back of the target object, a fine shadow image is not generated in the image taken by the first camera. Therefore, it is not recognized as a new error code at the time of recoding. Therefore, recoding corresponding to the projection pattern can be performed accurately. As described above, a high-precision three-dimensional image capturing apparatus capable of performing high-precision recoding even when there is a step in the front-back direction of the target object is obtained.

[0029]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a perspective view of a three-dimensional image pickup apparatus according to an embodiment of the present invention.

First, the principle of acquiring distance data using the re-coding process will be described. FIG. 1 is a block diagram illustrating a configuration example of a three-dimensional image capturing apparatus that performs acquisition of distance data using recoding processing. FIG. 2 shows the positional relationship between the light source and the image sensor.

In the configuration shown in FIG. 2, the three-dimensional shape measuring apparatus includes three cameras 101 to 103 and a projector 104. The illustrated distances I1, I2, and I3 are made equal so that the distance relationship between the cameras is uniform. First camera 10
1 and the light projector 104 are arranged so that their optical axes coincide with each other by using a half mirror 105 as a beam splitter. The second camera 102 and the third camera 103
Are arranged on both sides of the camera 101 and the projector 104 so that their optical axes are different from each other. The distance between the central optical axis and the optical axes on both sides is the base length L.

The light projector 104 includes a light source 106, a mask pattern 107, an intensity pattern 108, and a prism 10.
9 is provided. Here, as the light source 106, a light source in an invisible region using infrared light or ultraviolet light can be used. In this case, each camera is configured as shown in FIG. That is, the incoming light 310 is split in two directions by the prism 301, one of which is in the invisible region (infrared or ultraviolet).
An imaging device (for example, a CCD) through the transmission filter 302
Camera 303, and the other light passes through an invisible region (infrared and ultraviolet) cutoff filter 304 and enters an imaging device 305.

The light source 106 shown in FIG. 2 is not limited to the visible region or the invisible region, but may be a light source having a wavelength band capable of capturing an image. In this case, the first camera 101 uses a progressive scan type CCD camera, and the configuration of the second camera 102 and the third camera 103 is not particularly limited. However, considering the correspondence with the first camera 101, a CCD camera having the same configuration is desirable. The pattern is projected from the light source 106, and three cameras (101 to 103) shoot simultaneously. Each camera has filters 302 and 304 (see FIG. 3).
The light that has passed through is obtained by the imaging devices 303 and 305, whereby the images are collectively acquired.

The configuration of the three-dimensional shape measuring apparatus will be described with reference to FIG. As shown in the drawing, the second camera 102 stores luminance information obtained by photographing in a luminance value memory 121 and stores a photographing pattern in a pattern image memory 122. Third
The camera 103 similarly stores the luminance information in the luminance value memory 1
23, and stores the photographing pattern in the pattern image memory 12.
4 is stored. The first camera 101 stores the luminance information in the luminance value memory 125 and stores the photographing pattern in the pattern image memory 126. The projector 104 divides each slit into cells on a square lattice to divide the slit into cells on a square lattice in order to refer to a coded pattern created in advance.
27.

Using the stored photographing pattern and luminance information, a three-dimensional image is obtained as follows. The following operation is performed by the second camera 102 and the first camera 101
, The third camera 103 and the first camera 10
Here, the combination of the second camera 102 and the first camera 101 will be described as an example.

In FIG. 1, the area dividing unit 128
The region division of the photographing pattern photographed by the camera 101 is performed. Then, an area where the intensity difference between adjacent slit patterns is equal to or less than the threshold is extracted as an area 1 where light from the projector does not reach, and an area where the intensity difference between the slit patterns is equal to or more than the threshold is extracted as an area 2. I do. The re-coding unit 129 re-codes the extracted area 2 using the photographing pattern stored in the pattern image memory 126 and the projection pattern stored in the frame memory 127.

FIG. 4 is a flowchart when recoding is performed. First, each slit pattern is divided in the vertical direction for each slit width (step S11) to generate a square cell. An average value of the intensity is obtained for each of the generated cells, and the average value is set as the intensity of each cell (step S12).
In order from the center of the image, the intensity between each corresponding cell of the projection pattern and the photographing pattern is compared, and the intensity between the cells is equal to or greater than the threshold because the pattern has changed due to factors such as the reflectance of the object and the distance to the object. It is determined whether they are different (step S13). If the difference does not exceed the threshold value, the re-encoding ends for all the photographed cells (step S17).

If the difference is not less than the threshold value, it is determined whether or not the cell has a new strength (step S14). If the cell has a new strength, a new code is generated and assigned (step S15). If the cell is not of a new strength, the cell is coded using the arrangement of the slit patterns of other parts (step S16). This ends the re-coding (step S17).

FIG. 5 shows an example of coding of the slit pattern. FIG. 5A shows a projection pattern coded by the arrangement of the slits.
(Strong), 2 (medium), and 1 (weak) are assigned. In FIG. 6B, a code 0 is newly assigned because the intensity has changed in the third cell from the left and a new code has appeared. In the same figure (c), 3 from the left
Since an existing code appears in the second cell from the top, the code is recoded as a new code from the cell arrangement, such as [232] in the vertical arrangement and [131] in the horizontal arrangement. This recoding is equivalent to projecting a complicated pattern such as a two-dimensional pattern onto a portion where the shape of the object is rich in change, and projecting a simple pattern onto a portion where the shape changes little. This process is repeated, and re-encoding is performed by assigning a unique code to all cells.

FIG. 6 shows a plate 606 disposed in front of a wall 605 by using cameras 601 to 603 and a projector 604.
Shows an example of projecting a coded pattern. The coded pattern is the slit pattern shown in FIG. At this time, in the images obtained by the cameras 601 and 602, as shown in FIGS. 8 and 9, regions 801 and 901 which are shadows of the plate 606 are generated. In this example, a slit pattern as shown in FIG. 10 is obtained as a newly coded pattern on the surface of the plate 606.

Next, returning to FIG. The code decoding section 130 of the second camera 102 extracts the projection pattern from the pattern image memory 122 and divides it into cells in the same manner as described above. Then, the code of each cell is detected using the code re-coded by the re-coding unit 129 first,
Based on the detected code, the slit angle θ from the light source
Is calculated. FIG. 11 is a view showing a method of calculating a distance in the spatial coding. The slit angle θ of the cell to which each pixel belongs, the x coordinate on the image taken by the second camera, the focal length F which is a camera parameter, and the base line From the length L, the distance Z is calculated by the following equation (1).

[0042]

Z = (F × L) / (x + F × tan θ) (1)

The distance Z is calculated by the third camera 103
The same applies to the code decoding unit 131 on the side.

The distance for the above-mentioned area 1 is calculated as follows. In the area 1, pattern detection cannot be performed based on the projected pattern.
The parallax is detected using the luminance information read out from 21, 123, and 125, and the distance is calculated based on the parallax. Since the distance has been calculated by the above-described operation for the region other than the region 1, the minimum value of the distance of the region 1 is obtained, and the pixels that can be associated are also limited. Using these restrictions, correspondence between pixels is detected to detect parallax d, and a distance Z is calculated by the following equation (2) using a pixel size λ as a camera parameter.

[0045]

## EQU2 ## Z = (L × F) / (λ × d) (2)

The distance information obtained by the combination of the first camera 101 and the third camera 103 by the above-described method cannot detect the distance information of the shadow area 801 of the plate shown in FIG. On the other hand, the distance information obtained by the combination of the first camera 101 and the second camera 102 cannot detect the distance information of the shadow area 901 of the plate shown in FIG. That is, these shadow areas are areas that are blocked by the plate and light does not reach, and cannot be measured by this method. Therefore, distance information is obtained for a region other than the shadow. That is,
In the distance information integration unit 133 in FIG.
An image of the first camera 101 (FIG. 1) is obtained from the distance information calculated by the set of the first camera 101 and the second camera 102 and the distance information calculated by the first camera 101 and the third camera 103.
The distance information for all the pixels in 2), that is, the pixels other than the shadow is acquired. The distance information obtained by the above operation is stored in a three-dimensional image memory in association with, for example, the luminance image of the first camera 101, thereby capturing a three-dimensional image.

In one embodiment of the three-dimensional image pickup apparatus of the present invention, the light projector 104 of FIG. 1 and the first camera 101, which is the first camera optically coaxial with the light projector, have the same optical system. And a light-emitting element and a light-receiving element on the same semiconductor substrate.

The light emitter 104 emits light from the semiconductor chip,
The first camera 103 shoots with the same semiconductor chip. On this semiconductor chip, a light receiving element having the same address as a light emitting element having a certain address is formed adjacent to one pixel. Further, the projector 104 and the first camera 103
Are housed in the same housing using the same optical system. With this configuration, in this embodiment, the angle of view of the projector and the angle of view of the first camera are within the accuracy of one pixel.
Matches.

FIG. 13 is a sectional structural view of a pixel of a first embodiment of a semiconductor chip constituting a camera as an image sensor arranged coaxially with the light projector.

In this embodiment, a light emitting element array of a light projector and a light receiving element array of a camera as an image pickup element arranged coaxially with the light projector are formed on one semiconductor chip. Further, a pair of the light emitting element and the light receiving element is defined as one pixel, and a plurality of the pixels are arranged in a two-dimensional array. FIG. 13 is a sectional structural view showing one pixel formed on a semiconductor chip.

The structure shown in FIG. 13 will be described. First, a light emitting element is a known surface emitting laser element (VCSEL: Vert).
ical-Cavity Surface-Emitt
ingLASAR) 1101. The light receiving element is a known photodiode (PD: Photo Di).
mode) 1102.

A method for manufacturing a semiconductor chip having this configuration will be described. First, AlGaAs is formed on a GaAs substrate 1104.
And GaAs are periodically crystal-grown, and a distributed Bragg reflector (DBR: Distri) is a resonator having a desired wavelength.
butted Bragg Reflector) 110
Form 3 An active layer 1111 is provided thereon, and an upper DBR 1103 is periodically grown thereon. After that, the surface emitting laser element (VCSEL) 1101 and the photodiode (PD) are formed by the mesa etching technique.
Form 1102. After that, surface passivation with the insulating film 1105 and formation of the wiring 1106 are performed.
Complete the device.

FIG. 14 is a circuit diagram of a semiconductor chip having the structure shown in FIG. Surface emitting laser device (VC
SEL) 1101 and photodiode (PD) 110
Two pairs are two-dimensionally arranged. For the photodiode (PD) 1102, a readout circuit using an FET 1201 is provided for each pixel 1200, as in a normal solid-state imaging device.

First, a surface emitting laser element (VCSEL) 1
For 101, the row address pulse [ΦLDRowi
(I = 1, 2,..., M)] and a column address pulse [ΦL
Dcolumnnj (j = 1, 2,..., N)]
A drive current pulse is applied to a desired pixel. As a result, a desired pixel 1200 can be moved from a surface emitting laser element (VC
SEL) 1101 is emitted.

In the light receiving section including the photodiode (PD) 1102, reading, resetting, and light receiving are performed in the same manner as in a known solid-state imaging device. First, reading of each pixel
Row address pulse [ΦPDrowi (i = 1, 2,...,
m)] and a column address pulse [ΦPDcolumnj].
(J = 1, 2,..., N)], a desired pixel is selected, and the charge of the pixel is read. The read charge is taken out through a read line [VPOutj (j = 1, 2,..., N)].

The signal based on this charge is converted to an analog amplifier 12
02 to obtain an output voltage. Next, each pixel is reset at a desired timing. In the reset operation, the reset pulse [ΦPDreseti (i = 1, 2,..., M)] is turned on, and the photodiode (PD) 1102 is connected to the reset voltage [VPDreset].

Each pulse for controlling the above timing is as follows.
Vertical scanner circuit 1203 and horizontal scanner circuit 120
Generated and applied at 4.

As described above, in this embodiment, the light projector is a surface emitting laser element (VCS) in the semiconductor chip.
EL) 1101, and the first camera captures an image with a light receiving unit including a photodiode (PD) 1102 of the same semiconductor chip. As described above, on this semiconductor chip, the light receiving element of the same address as the light emitting element of a certain address is 1
Adjacent to one pixel, the angle of view of the projector and the angle of view of the first camera coincide with the accuracy of the size of the one pixel.

Therefore, the angle of view of the projector and the first camera arranged on the same optical axis are substantially the same, and in the image taken by the first camera, at the discontinuous portion before and after the object, that is, at the edge portion, There is no thin shadow image. The shadow image is problematic when its width is greater than the resolution dimension of the first camera, as described above. In the image projected on the surface of the imaging device of the first camera,
It occurs as an image when the width of the shadow image is larger than the size of the pixel of the image sensor, and causes an error. However, in the configuration of the present invention, the light receiving element of the same address as the light emitting element of a certain address is formed adjacent to one pixel on the semiconductor chip, and the angle of view of the projector and the angle of view of the first camera are determined. Since the angle of view coincides with the precision within the dimension of one pixel, it is possible to prevent a band-like pattern due to a shadow image from being generated, to prevent a new error code from being generated at the time of recoding, and to obtain an accurate image. Accurate distance data can be calculated by the re-coding process.

FIG. 15 is a sectional structural view of a pixel portion showing the configuration of the semiconductor chip of the second embodiment. In FIG. 15, the light emitting element is a surface emitting laser element (VCSEL) 1101 similar to that of the first embodiment. As in the first embodiment, the light emitting device is manufactured by forming an Al layer on a GaAs substrate 1104.
GaAs and GaAs are periodically deposited and grown to form a lower DBR 1103. An active layer 1111 is formed thereon.
And an upper DBR 1103 is formed thereon. After that, a surface emitting laser element (VCSEL) 1101 is separately formed by a mesa etching technique. After that, surface passivation by the insulating film 1105 and wiring 110
6 is completed.

In FIG. 15, the light receiving element is a known photogate (PG: PhotoGa) made of a Si device.
te) 1301 and a known CCD (Charge Co.)
(device) 1302. Photocharges (indicated by black circles in the figure) generated below the photogate (PG) 1301 are transferred using the CCD 1302.

FIG. 15 shows, as a representative example, an insulating film 1304 on a Si wafer and an electrode 1305 made of polysilicon thereon. However, in addition, various known CCD structures such as a buried diffusion channel and multiple polysilicon electrodes can be used. To manufacture the light receiving element, a photo gate (PG) 1301 and a CCD 1302 are formed on a Si substrate by using a well-known silicon semiconductor manufacturing technique.

Next, the light emitting element and the light receiving element are integrated into one chip. First, the back surface of the above-mentioned Si substrate is polished and thinned to form a Si bonded wafer 1303. Next,
After that, the region where the laser element (VCSEL) 1101 is to be disposed is removed by etching to form an opening 1310 on the Si bonded wafer 1303. Then S
The i-bonded wafer 1303 is formed by a known wafer bonding technique using polyimide 1306 as an adhesive.
It is bonded on the aAs substrate 1104. At this time, the opening 13
The position of 10 and the position of the laser element (VCSEL) 1101 are matched. Thus, the semiconductor chip structure shown in FIG. 15 is obtained.

FIG. 16 is a plan view of a semiconductor chip having the structure shown in FIG. In FIG. 16, a laser element (VCSEL) 1101 and a photo gate (PG) 1
One pair of 301 constitutes one pixel. Along vertical columns of photo gates (PG) 1301, vertical CCDs 140
1 is provided, and charges taken out from each photo gate (PG) 1301 are sequentially transferred downward in the figure. At the bottom of the figure, a horizontal CCD 1402 is provided, and a vertical CCD of each column is provided.
The charge received from the CD 1401 is transferred to the left in the figure. At the chip end, a known floating diffusion (FD: (Floating Diffus)
ion) 1403 and an analog amplifier 1404 are used to convert the voltage into an output voltage, and this voltage is read from the output unit 1405.

As described above, also in the present embodiment, light is projected from the above-mentioned semiconductor chip, and an image is taken with the same semiconductor chip. The projector and the first camera use the same optical system and are housed in the same housing. For this reason, the angle of view of the projector and the angle of view of the first camera coincide with an accuracy within the size of the pixel.

Accordingly, the angle of view of the projector and the first camera arranged on the same optical axis are substantially the same, and in the image taken by the first camera, at the discontinuous portion before and after the object, that is, at the edge portion, There is no thin shadow image. The shadow image is problematic when its width is greater than the resolution dimension of the first camera, as described above. In the image projected on the surface of the imaging device of the first camera,
It occurs as an image when the width of the shadow image is larger than the size of the pixel of the image sensor, and causes an error. However, in the configuration of the present invention, the light receiving element of the same address as the light emitting element of a certain address is formed adjacent to one pixel on the semiconductor chip, and the angle of view of the projector and the angle of view of the first camera are determined. Since the angle of view coincides with the precision within the dimension of one pixel, it is possible to prevent a band-like pattern due to a shadow image from being generated, to prevent a new error code from being generated at the time of recoding, and to obtain an accurate image. Accurate distance data can be calculated by the re-coding process.

The present invention has been described in detail with reference to the specific embodiments. However, it is obvious that those skilled in the art can modify or substitute the embodiment without departing from the spirit of the present invention. That is, the present invention has been disclosed by way of example, and should not be construed as limiting. In order to determine the gist of the present invention, the claims described at the beginning should be considered.

[0068]

As described above, according to the present invention, a light emitting element and a light receiving element for each pixel are formed adjacent to each other on a semiconductor chip, and a light emitting pattern for three-dimensional shape measurement is formed on the semiconductor chip. And the optical image is captured by a light-receiving element on the semiconductor chip, so that the angle of view of the projector and the angle of view of the first camera are Match with accuracy within dimensions. Therefore, even if the target object has a discontinuity before and after,
There is no generation of a fine shadow image in an image taken by another camera. As a result, in the three-dimensional image capturing apparatus to which the recoding method is applied, the accuracy of recoding corresponding to the projection pattern can be significantly improved. Therefore,
Even in the case where there is a step in the front-back direction of the target object, a three-dimensional image pickup apparatus with greatly improved accuracy can be obtained.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration example of a three-dimensional image capturing apparatus according to the present invention.

FIG. 2 is a diagram illustrating a camera configuration example of a three-dimensional image capturing apparatus according to the present invention.

FIG. 3 is a diagram illustrating an imaging configuration of the three-dimensional image imaging device of the present invention.

FIG. 4 is a diagram showing a processing flow of the three-dimensional image capturing apparatus of the present invention.

FIG. 5 is a diagram illustrating an example of coding of a projection pattern of the three-dimensional image capturing apparatus of the present invention.

FIG. 6 is a diagram showing an example of a photographing configuration of the three-dimensional image pickup device of the present invention.

FIG. 7 is a diagram showing an example of a projection pattern of the three-dimensional image pickup device of the present invention.

FIG. 8 shows a third camera 1 of the three-dimensional image pickup apparatus of the present invention.
It is a figure which shows the example of the slit pattern image | photographed by 03.

FIG. 9 shows a second camera 1 of the three-dimensional image pickup apparatus of the present invention.
It is a figure which shows the example of the slit pattern image | photographed by 02.

FIG. 10 is a diagram showing an example of a newly coded slit pattern in the three-dimensional image pickup apparatus of the present invention.

FIG. 11 is a diagram illustrating a distance calculation method by a spatial coding method of the three-dimensional image capturing apparatus of the present invention.

FIG. 12 is a diagram illustrating an example of a slit pattern captured by the first camera 101 of the three-dimensional image capturing apparatus according to the present invention.

FIG. 13 is a sectional structural view of a pixel of the first embodiment, showing a configuration of a light projector and an image sensor of the three-dimensional image capturing apparatus of the present invention.

FIG. 14 is a circuit configuration diagram of a first embodiment illustrating a configuration of a light projector and an image sensor of the three-dimensional image capturing apparatus of the present invention.

FIG. 15 is a sectional structural view of a pixel of a second embodiment showing the configuration of the light projector and the image sensor of the three-dimensional image capturing apparatus of the present invention.

FIG. 16 is a plan view showing a structure of a light emitting device and an image sensor of a three-dimensional image capturing apparatus according to a second embodiment of the present invention.

[Explanation of symbols]

 101 First Camera 102 Second Camera 103 Third Camera 104 Projector 105 Half Mirror 106 Light Source 107 Mask Pattern 108 Intensity Pattern 109 Prism 121, 123, 125 Luminance Value Memory 122, 124, 126 Pattern Image Memory 127 Frame Memory 128 Area dividing unit 129 Recoding unit 130, 131 Code decoding unit 133 Distance information integration unit 134 Three-dimensional image memory 301 Prism 303, 305 Imaging device 601, 602, 603 Camera 604 Projector 605 Wall 606 Plate 801, 901 Shadow region 1101 VCSEL (surface emitting laser) 1102 PD (photodiode) 1103 DBR (distributed Bragg reflector) 1104 GaAs substrate 1105, 1304 insulating film 1106 wiring 1200 image 1201 FET 1202,1404 analog amplifier 1203 vertical scanner circuit 1204 horizontal scanning circuit 1301 PG (photo-gate) 1302 CCD 1303 Si bonded wafer 1305 electrodes 1306 polyimide 1401 vertical CCD 1402 horizontal CCD 1403 FD (floating diffusion) 1405 Output section

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Tetsuo Iyoda 430 Sakai Nakai-cho, Ashigara-gun, Kanagawa Green Tech Naka Fuji Xerox Co., Ltd. F-term (reference) 2F065 AA04 AA06 AA07 DD12 FF05 GG06 GG22 HH07 JJ03 JJ05 JJ07 JJ18 JJ26 LL22LL46 PP22 QQ00 QQ24 2H059 AA08 5C061 AA06 AA21 AB04 AB12 AB21 BB03

Claims (10)

[Claims] 1. A three-dimensional image capturing apparatus for acquiring distance information of a measurement target based on an image of a pattern image obtained by projecting a coded pattern onto a measurement target, wherein the coded pattern is projected. A light projector; a first camera for photographing a projection pattern by the light projector from an optical axis direction of the light projector; and a second camera for photographing the projection pattern from a direction different from the optical axis direction of the light projector. A new code is assigned to an area where the amount of change of the pattern taken by the first camera with respect to the pattern is equal to or greater than a predetermined value, and first distance information is generated from the pattern taken by the second camera using the assigned code. Re-encoding processing for acquiring distance information based on the first distance information and the luminance information obtained by the first camera, and the distance information A three-dimensional image pickup apparatus, comprising an acquisition unit, wherein the light projector and the first camera have substantially the same angle of view. 2. The three-dimensional image pickup device according to claim 1, wherein a difference in the angle of view between the light projector and the first camera is equal to or smaller than a resolution size of the first camera. 3. The projector according to claim 1, wherein the light emitter is formed by a surface light emitting element formed on a semiconductor substrate, and the first camera is formed by a light receiving element formed on the semiconductor substrate. The three-dimensional image pickup device according to claim 1. 4. The light emitting device according to claim 1, wherein the light emitting device includes a surface light emitting device formed on a semiconductor substrate, and the first camera includes a light receiving device formed on the semiconductor substrate. 2. The device according to claim 1, wherein the element and the light receiving element are formed on the same semiconductor substrate.
A three-dimensional image capturing apparatus according to claim 1. 5. The projector according to claim 1, wherein the projector is configured by a surface light emitting element formed on a semiconductor substrate, and the first camera is configured by a light receiving element formed on the semiconductor substrate. The semiconductor substrate on which the light emitting element is formed and the semiconductor substrate on which the light receiving element of the first camera is formed are bonded to each other to form a single semiconductor substrate. Three-dimensional image pickup device. 6. The surface light emitting device according to claim 6, wherein the light emitting device is configured by a surface light emitting element formed on a semiconductor substrate, and the first camera is configured by a light receiving element formed on the semiconductor substrate. And the light receiving elements are adjacent to each other to form a set of light emitting and light receiving elements. The light emitting and light receiving element sets are arranged in an array, and the light projector and the first light receiving element are arranged in an array.
The three-dimensional image pickup apparatus according to claim 1, wherein the three-dimensional image pickup apparatus is configured to have a camera. 7. A three-dimensional image pickup apparatus for acquiring distance information of a measurement target based on an image of a pattern image obtained by projecting a coded pattern onto a measurement target, wherein the coded pattern is projected. A light projector; a first camera for photographing a projection pattern by the light projector from an optical axis direction of the light projector; and a second camera for photographing the projection pattern from a direction different from the optical axis direction of the light projector. A new code is assigned to an area where the amount of change of the pattern taken by the first camera with respect to the pattern is equal to or greater than a predetermined value, and first distance information is generated from the pattern taken by the second camera using the assigned code. Re-encoding processing for acquiring distance information based on the first distance information and the luminance information obtained by the first camera, and the distance information A three-dimensional image pickup apparatus, comprising: an acquisition unit, wherein the light projector and the first camera are integrated. 8. The light emitting device according to claim 1, wherein the light emitting device comprises a surface light emitting element formed on a semiconductor substrate, and the first camera comprises a light receiving element formed on the semiconductor substrate. 8. The device according to claim 7, wherein the element and the light receiving element are formed on the same semiconductor substrate.
A three-dimensional image capturing apparatus according to claim 1. 9. The projector according to claim 1, wherein the projector is configured by a surface light emitting element formed on a semiconductor substrate, and the first camera is configured by a light receiving element formed on the semiconductor substrate. The semiconductor substrate on which the light-emitting element is formed and the semiconductor substrate on which the light-receiving element of the first camera is formed are adhered to each other to be formed on one semiconductor base. Three-dimensional image pickup device. 10. The light emitting device according to claim 1, wherein the light emitting device comprises a surface light emitting device formed on a semiconductor substrate, and the first camera comprises a light receiving device formed on the semiconductor substrate. And each of the light receiving elements is adjacent to form a set of light emitting and light receiving elements, and the light emitting and light receiving element sets are arranged in an array to form the light projector and the first camera. 8. The three-dimensional image pickup device according to claim 7, wherein: