JP6580761B1 - Depth acquisition apparatus and method using polarization stereo camera - Google Patents

Abstract

A depth acquisition apparatus that acquires depth information of an object in a three-dimensional space from a polarization image. A depth acquisition apparatus 100 is based on a polarization image captured by a stereo camera 1 including a first polarization camera 11 and a second polarization camera 12 each including a plurality of polarizers that capture an image of a measurement object 5. A luminance information acquisition unit 21 that acquires the luminance of light transmitted through each polarizer in association with each point on the first image plane and each point on the second image plane, and the degree of polarization calculated from the luminance of the light Based on the normal vector calculation unit 22 for calculating the normal vector value at each point on the first image plane and each point on the second image plane, and on each point on the first image plane and on the second image plane A corresponding point calculation unit 24 that calculates a corresponding point by matching normal vector values at each point, a three-dimensional coordinate value calculation unit 25 that calculates a three-dimensional coordinate value of the measurement object 5 based on the corresponding point, Is provided. [Selection] Figure 1

Description

  The present invention relates to an apparatus and a method for acquiring depth information of a subject in a three-dimensional space using a stereo camera including two polarization cameras.

Conventionally, as disclosed in Patent Document 1, two two-dimensional images respectively captured with respect to an object are acquired by a stereo camera including two cameras, and an image in one of the two-dimensional images is acquired. It is well known how to match the correspondence between a point and an image point in the other two-dimensional image using, for example, color or brightness, and obtain the three-dimensional shape (three-dimensional coordinate value) of the object by triangulation. It has been.
However, when matching is performed using captured images (stereo images) using color or brightness between the cameras, the angle between the light source and the camera is different in each camera, so there is a difference in color and brightness distribution between the cameras. It is well known that an error occurs in matching due to the occurrence of an error in the distance calculated based on the parallax between stereo images.

  On the other hand, Non-Patent Document 1 describes research and basic theory on polarization used in the field of computer vision. Non-Patent Document 1 introduces a polarization camera that can measure the polarization state of light in real time, a technique for measuring the shape of an opaque object using the degree of polarization, and the like.

Further, Patent Document 2 discloses a depth map by detecting corresponding points from left and right natural light images, calculating a distance from the imaging surface of the corresponding points from the parallax based on the principle of triangulation, and mapping the distance to the image plane. Generating continuity of the surface of the subject based on the distribution of the normal vector of the subject surface mapped to the image plane calculated based on the polarization image.
Further, in Patent Document 3, a three-dimensional model of an object is registered in advance, and then a normal vector distribution of a subject surface mapped to an image plane calculated based on a polarization image is obtained. It is disclosed that a three-dimensional model of an object is placed in a world coordinate system in which a camera coordinate system is set, and a state that matches the distribution of normal vectors is obtained by moving, rotating, and deforming the model. ing.
Patent Document 4 discloses that the accuracy of a depth image is improved by applying normal information to a depth image of a subject created based on the amount of blur of the subject.
As described above, in Patent Document 2, Patent Document 3, and Patent Document 4, the distribution of normal vectors on a subject surface mapped on an image plane with respect to a three-dimensional model or depth map of an object created in advance is disclosed. It is disclosed that the accuracy of a three-dimensional model or depth map of an object created in advance is improved by using.

  As described above, a technique for creating a distribution of normal vectors on a subject surface mapped on an image plane based on a polarization image is known. Then, as shown in Patent Document 2, Patent Document 3, and Patent Document 4, for example, the distribution of the normal vector is used for a three-dimensional model or depth map of the object created in advance. However, there are known methods for improving the accuracy of a three-dimensional model or depth map of an object. In either case, for example, matching is performed using two colors obtained by capturing an object with a stereo camera or the like using color and brightness. However, it is assumed that the three-dimensional shape (three-dimensional coordinate value) of the object is created in advance by triangulation, and then the adjustment by the polarization image is further performed. It was not something.

JP 2014-186715 A JP 2017-038011 A Japanese Patent Application Laid-Open No. 2018-041278 JP 2018-046340 A

Daisuke Miyazaki, 1 other, "Basic Theory of Polarization and its Applications", Transactions of Information Processing Society of Japan, Computer Vision and Image Media, Vol. 1, no. 1, June 2008, pp. 64-72

  In the above-described conventional technique, two images obtained by capturing an object with, for example, a stereo camera are matched using colors and brightness, and a three-dimensional shape (three-dimensional coordinate value) of the object is created by triangulation. After that, it takes time and effort to perform adjustment using the polarization image.

  The present invention provides an apparatus and method for acquiring depth information (three-dimensional position information) of an object in a three-dimensional space from a polarization image without creating a three-dimensional model or depth map of the object in advance. For the purpose.

  (1) The present invention relates to a first polarization camera including a plurality of polarizers that respectively transmit at least three polarization components, and a second polarization camera that includes a plurality of polarizers that respectively transmit at least three polarization components. The position of each point on the first image plane, which is the image plane of the first polarization camera, and the brightness of the light transmitted through each polarizer in the first polarization camera. A first luminance information acquisition unit that acquires the information in association with information; and luminances of light transmitted through the polarizers in the second polarization camera, respectively, on the second image plane that is an image plane of the second polarization camera. A second luminance information acquisition unit that is acquired in association with position information of a point; and the position of each point on the first image plane of the first polarization camera acquired by the first luminance information acquisition unit. Based on the degree of polarization calculated from the luminance of light transmitted through each polarizer associated with the information, at the position of the point on the surface of the subject projected onto the position of each point on the first image plane The first polarization vector acquired by the first normal vector calculation unit that calculates the first normal vector value and associates it with the position information of each point on the first image plane, and the second luminance information acquisition unit The surface of the subject projected on the position of each point on the second image plane based on the luminance of light transmitted through each polarizer associated with the position information of each point on the second image plane Calculating a second normal vector value at the position of the upper point, and associating the second normal vector value with the position information of each point on the second image plane; and for each point on the first image plane A first normal vector value associated with the position information, and each of the points Corresponding to calculate corresponding points between the first image plane and the second image plane by matching the corresponding second normal vector values associated with each point on the epipolar line on the second image plane Based on the point calculation unit and the corresponding point calculated by the corresponding point calculation unit, a three-dimensional coordinate value in a preset reference coordinate system of a point on the surface of the subject projected on the corresponding point is calculated. A depth acquisition apparatus comprising: a three-dimensional coordinate value calculation unit.

  (2) The first normal vector calculation unit described in (1) is a luminance of light transmitted through each polarizer associated with position information of each point on the first image plane of the first polarization camera. The degree of polarization of diffuse reflected light and / or the degree of polarization of specular reflected light is calculated from the first normal vector value based on the degree of polarization of diffuse reflected light and / or the first method based on the degree of polarization of specular reflected light. A line vector value is calculated, and the second normal vector calculation unit calculates the luminance of light transmitted through each polarizer associated with position information of each point on the second image plane of the second polarization camera. The degree of polarization of diffuse reflected light and / or the degree of polarization of specular reflected light is calculated, and the second normal vector value based on the degree of polarization of diffuse reflected light and / or the second normal based on the degree of polarization of specular reflected light. Vector values may be calculated.

  (3) The corresponding point calculation unit according to (1) or (2), the distribution state of the first normal vector value associated with each point on the first image plane and its vicinity, and each point The first image plane and the second image plane are matched by matching each point on the epipolar line on the second image plane corresponding to and the distribution state of the second normal vector value associated therewith. The corresponding point may be calculated.

  (4) In the depth acquisition device according to any one of (1) to (3), the first normal vector value associated with each point on the first image plane is the same as the first image. Detecting a continuous region including continuous points on the plane, and a point on the first image plane excluding the continuous region, and there is a continuous region in which the normal vector value is the same value in the vicinity of the point, And a plane region detecting unit for detecting an edge point at which the first normal vector value at the point changes discontinuously and rapidly from the normal vector value in the continuous region, and the three-dimensional coordinate value calculating unit includes: Based on the edge point, a three-dimensional coordinate value in the reference coordinate system of each point on the surface area of the subject corresponding to the continuous area where the first normal vector values on the first image plane are the same. It may be calculated.

  (5) The corresponding point calculation unit according to (4), based on a three-dimensional coordinate value in the reference coordinate system of each point on the continuous area, the point on the continuous area on the first image plane and the point Corresponding points on the second image plane may be calculated.

  (6) The depth acquisition device according to any one of (1) to (5) includes a third camera separately from the first polarization camera and the second polarization camera, and an image captured by the third camera. You may make it provide the point cloud data production | generation part which produces | generates point cloud data by matching the data with the three-dimensional coordinate value in the said reference coordinate system.

  (7) The present invention includes a first polarization camera including a plurality of polarizers that respectively transmit at least three directions of polarization components, and a second polarization camera that includes a plurality of polarizers that respectively transmit at least three directions of polarization components. The first image plane, which is the image plane of the first polarization camera, represents the brightness of light transmitted through each polarizer in the first polarization camera by a stereo camera that images the subject and a computer that is communicably connected. A first luminance information acquisition step that is acquired in association with position information of each of the points above, and luminances of light transmitted through the polarizers in the second polarization camera are image planes of the second polarization camera, respectively. In a second luminance information acquisition step that is acquired in association with position information of each point on the two image planes, and in the first luminance information acquisition step Based on the degree of polarization calculated from the brightness of light transmitted through each polarizer associated with the position information of each point on the first image plane of the first polarization camera obtained, the first image plane Calculating a first normal vector value at the position of the point on the surface of the subject projected onto the position of each point on the first normal vector, and associating it with the position information of each point on the first image plane Based on the luminance of the light transmitted through each polarizer associated with the position information of each point on the second image plane of the second polarization camera acquired in the calculation step and the second luminance information acquisition step Calculating a second normal vector value at the position of the point on the surface of the subject projected to the position of each point on the second image plane, and obtaining the position information of each point on the second image plane. A second normal vector calculating step to associate, and A first normal vector value associated with the position information of each point on the first image plane, and a second method associated with each point on the epipolar line on the second image plane corresponding to each point. Based on the corresponding point calculated in the corresponding point calculating step, the corresponding point calculating step of calculating a corresponding point between the first image plane and the second image plane by matching a line vector value The present invention relates to a depth information acquisition method comprising: a three-dimensional coordinate value calculating step of calculating a three-dimensional coordinate value in a preset reference coordinate system of a point on the surface of the subject projected onto the point.

  (8) The present invention relates to a computer program for causing a computer to execute each step described in (7).

  According to the depth acquisition apparatus and method of the present invention, depth information (three-dimensional position information) of an object in a three-dimensional space is obtained from a polarization image without creating a three-dimensional model or depth map of the object in advance. An apparatus and method for obtaining can be provided.

It is a block diagram showing roughly an example of the depth acquisition device concerning the embodiment of the present invention. It is the schematic of the polarizer with which the 1st polarization camera and 2nd polarization camera which concern on embodiment of this invention are provided. It is a figure which shows roughly the coordinate system with which the 1st polarization camera which concerns on embodiment of this invention is provided. It is a schematic block diagram of the memory | storage part which concerns on embodiment of this invention. It is a schematic block diagram of the control part which concerns on embodiment of this invention. It is a figure which shows the flow of a series of processes in embodiment of this invention. It is a figure which shows the flow of a series of processes in embodiment of this invention.

Hereinafter, an example of an embodiment of the present invention will be described.
FIG. 1 is a diagram illustrating a functional configuration of a depth acquisition apparatus 100 according to the present embodiment. In addition, in this embodiment, the depth acquisition based on the diffuse reflected light which is the light scattered by the pigment particle which comprises the measuring object 5 used as a to-be-photographed object is illustrated.

  As shown in FIG. 1, the depth acquisition apparatus 100 is based on a stereo camera 1 that captures an object 5 to be measured at an arbitrary frame rate (for example, 30 fps) and image data captured by the stereo camera 1. A control unit 2 that performs predetermined information processing and a storage unit 3 are included.

<Stereo camera 1>
The stereo camera 1 includes a first polarization camera 11 and a second polarization camera 12.
As shown in FIG. 2, the first polarizing camera 11 and the second polarizing camera 12 include pattern polarizers 111 and 121, respectively.
For example, as shown in the groove of FIG. 2, the pattern polarizers 111 and 121 are two-dimensionally formed by combining four types of polarizers having polarization main axis angles of 0 degrees, 45 degrees, 90 degrees, and 135 degrees. Is a collection of polarizers arranged in Each polarizer exhibits a polarization characteristic of transmitting a polarization component in a direction perpendicular to the groove direction and not transmitting (reflecting) a polarization component in a direction parallel to the groove direction. The pattern polarizer shown in FIG. 2 is disclosed in, for example, Japanese Patent Application Laid-Open Nos. 2009-58533, 2017-38011, and 2018-46340.
The imaging elements 112 and 122 (not shown) are a group of pixels (pixels) arranged in a two-dimensional shape that receive light transmitted through the pattern polarizers 111 and 121. The image sensor in this embodiment illustrates the case of the image sensor concerning a gray scale. That is, four types of pattern polarizers having different angles are arranged on each pixel, and the luminance of light corresponding to the four principal axis angles can be acquired at the same time.
By detecting the transmitted polarization component with a photodetection layer (not shown) included in the imaging elements 112 and 122 (not shown), four types of polarization luminances are acquired simultaneously. When four types of luminance information are acquired by four image sensors, four types of luminance information can be associated with pixels corresponding to the image sensors. Further, four types of luminance information may be associated with a pixel in which these are grouped.
Note that the acquisition of polarization luminance is not limited to an image sensor according to a gray scale. For example, an image sensor (R, G, B) relating to a color image exemplified in JP 2009-58533 A, JP 2017-38011 A, JP 2018-46340 A, or the like may be applied. . In this case, since four types of polarization luminance can be simultaneously acquired for each of R, G, and B, for example, the polarization luminance by any combination of R, G, and B may be applied. For example, a color space color based on RGB converted to a gray scale represented only by luminous intensity (luminance storage conversion) may be applied. Alternatively, polarization luminance based on the R polarization image, polarization polarization G, or polarization luminance based on the B polarization image may be applied.

<Coordinate system of first polarization camera 11 and second polarization camera 12>
Here, a coordinate system included in the first polarization camera 11 and the second polarization camera 12 will be briefly described.
The first polarization camera 11 has a unique coordinate system, that is, a first camera coordinate system, a first normalized image coordinate system, and a first digital image coordinate system. Similarly, the second polarization camera 12 includes a unique coordinate system, that is, a second camera coordinate system, a second normalized image coordinate system, and a second digital image coordinate system. Hereinafter, the coordinate system of the first polarization camera 11 will be described. FIG. 3 is a diagram illustrating a coordinate system (a first camera coordinate system and a first image coordinate system) included in the first polarization camera 11. In FIG. 3, the first polarization camera 11 and the second polarization camera 12 are shared, and the index is omitted.
In this embodiment, it is assumed that a corrected image in which the influence of lens distortion is canceled is obtained in the first polarization camera 11 and the second polarization camera 12, and the internal parameters are known.

<First camera coordinate system>
The first camera coordinate system is a three-dimensional coordinate system viewed from the first polarization camera 11 as a three-dimensional coordinate representing a space. The first camera coordinate system generally has the center of the camera lens as the origin, the optical axis direction of the first polarization camera 11 as the Z axis, the upward direction of the first polarization camera 11 as the Y axis, and the first polarization camera 11. Is a three-dimensional coordinate system (right-handed system) with the horizontal direction as the X-axis.
Hereinafter, unless otherwise specified, coordinate values obtained by the first polarizing camera 11 are represented by (X ₁ , Y ₁ , Z ₁ ).

<First normalized image coordinate system>
The first normalized image coordinate system, a point on the optical axis as an origin, x-axis, respectively the y-axis, X-axis is taken parallel to the Y axis, the focal length f ₁ 1 and the two-dimensional coordinate system (x, y) is referred to as a first normalized image coordinate system.
Hereinafter, unless otherwise specified, coordinate values in the first normalized image coordinate system are represented by (x ₁ , y ₁ ).
At this time,
x ₁ = X ₁ / Z ₁
y ₁ = Y ₁ / Z ₁
Holds.

<First image coordinate system>
The first image coordinate system is described in units of the number of pixels (pixels).
In the first image coordinate system, the upper left is the origin, the right direction is the first axis (u axis), and the lower direction is the second axis (v axis). The point through which the optical axis passes is called the image center (c _u1 , c _v1 ).
If f ₁ is the focal length of the first polarization camera 11, and δ _u1 and δ _v1 are the horizontal and vertical intervals of the pixels of the image sensor, respectively,
α _u1 = f ₁ / δ _u1 and α _v1 = f ₁ / δ _v1 are focal lengths expressed in units of pixels.
At this time, between the first normalized image coordinates (x ₁ , y ₁ ) and the first image coordinates (u ₁ , v ₁ ),
u ₁ = α _u1 x ₁ + c _u1 where α _u1 = f ₁ / δ _u1
v ₁ = α _v1 y ₁ + c _v1 where α _v1 = f ₁ / δ _v1
Α _u1 α _v1 and c _u1 c _v1 are referred to as camera internal parameters.
The coordinate system included in the first polarization camera 11 has been described above. In the above description, the coordinate system of the second polarization camera 12 is described by replacing “first” with “second” and replacing each index 1 with index 2.

The relationship between the first camera coordinate system of the first polarization camera 11 and the second camera coordinate system of the second polarization camera 12 will be briefly described.
By calibrating the first polarization camera 11 and the second polarization camera 12 in advance, the coordinate value in the first camera coordinate system of the first polarization camera 11 is changed to the coordinate value in the second camera coordinate system of the second polarization camera 12. A coordinate transformation matrix ² T ¹ to be transformed is calculated and stored in advance in the storage unit 3, for example. Similarly, a coordinate conversion matrix ¹ T ² for converting coordinate values in the second camera coordinate system of the second polarization camera 12 into coordinate values in the first camera coordinate system of the first polarization camera 11 is calculated and stored in advance. 3 is stored.
The following relationship holds between them.
¹ T ² · ² T ¹ = E
^{2 T 1 · 1 T 2 =} E

Further, by setting a reference coordinate system (so-called world coordinate system) in advance, each point on the surface of the subject can be represented by reference coordinates in the reference coordinate system. Hereinafter, unless otherwise specified, coordinate values based on the reference coordinate system are represented by (X _W , Y _W , Z _W ).
At this time, by performing calibration in advance, a coordinate transformation matrix ^W T ¹ for converting the coordinate value in the first camera coordinate system to the coordinate value in the reference coordinate system is calculated, and the coordinate value in the reference coordinate system is calculated as the first camera coordinate. leave calculates the coordinates transformation matrix ¹ T ^W for converting the coordinate values in the system, previously stored in the storage unit 3. Similarly, a coordinate conversion matrix ^W T ² that converts coordinate values in the second camera coordinate system into coordinate values in the reference coordinate system is calculated, and coordinates that convert coordinate values in the reference coordinate system into coordinate values in the second camera coordinate system A conversion matrix ^{2 TW} is calculated and stored in the storage unit 3 in advance.

<Control unit 2>
The control unit 2 is a part that controls the entire depth acquisition apparatus 100, and implements various functions in the present embodiment by appropriately reading and executing software (depth acquisition program) stored in the storage unit 3, for example. ing. The control unit 2 may be a CPU. Details of the control unit 2 will be described later.

<Storage unit 3>
FIG. 4 is a schematic block diagram of the storage unit 3. The storage unit 3 is a storage area for various programs and various data for causing the hardware group to function as the control unit 2, and may be a ROM, a RAM, a flash memory, a hard disk drive (HDD), or the like. Further, as will be described later, the storage unit 3 stores a correspondence table (hereinafter referred to as “first”) between positional information of each pixel on the first image plane and normal vector values of the subject surface mapped on the first image plane. (Referred to as “normal vector distribution table 31”), position information of each pixel on the second image plane, and normal vector values of the subject surface mapped on the first image plane (hereinafter referred to as “second normal line”). Vector distribution table 32 "). In addition, a “corresponding point information table 33” and a “three-dimensional coordinate value table 34” described later are stored. Details will be described later.

<Functional configuration of control unit 2>
FIG. 5 is a schematic diagram showing functional blocks provided in the control unit 2. As shown in FIG. 5, the control unit 2 includes a luminance information acquisition unit 21, a normal vector calculation unit 22, a plane area detection unit 23, a corresponding point calculation unit 24, a three-dimensional coordinate value calculation unit 25, Is provided. The luminance information acquisition unit 21 further includes a first luminance information acquisition unit 211 and a second luminance information acquisition unit 212. The normal vector calculation unit 22 further includes a first normal vector calculation unit 221 and a second normal vector calculation unit 222.

<Luminance information acquisition unit 21>
The luminance information acquisition unit 21 includes a first luminance information acquisition unit 211 and a second luminance information acquisition unit 212. The first luminance information acquisition unit 211 sets the luminance of light transmitted through each polarizer in the first polarization camera 11 for each frame imaged by the first polarization camera 11, respectively, on the image plane of the first polarization camera 11. Are acquired in association with position information of each point on the first image plane. On the other hand, the second luminance information acquisition unit 212 determines the luminance of the light transmitted through each polarizer in the second polarization camera 12 for each frame imaged by the first polarization camera 11. Acquired in association with position information of each point on the second image plane which is the image plane.
That is, in the present embodiment, the first luminance information acquisition unit 211 performs, for each frame captured by the first polarization camera 11, four polarization orientations (four ways) in each pixel on the first image plane of the frame. Brightness change with respect to the main axis angle). Similarly, the second luminance information acquisition unit 212 acquires luminance changes for the four polarization directions in each pixel on the second image plane of the frame. As described above, the luminance information is acquired for each frame. Hereinafter, unless otherwise specified, each functional unit calculates depth information and three-dimensional coordinate position information for each frame. The information created by each functional unit is given frame identification information (for example, a time stamp) for identifying a frame.

<Normal Vector Calculation Unit 22>
The first normal vector calculation unit 221 transmits the light transmitted through each polarizer associated with the position information of each point on the first image plane of the first polarization camera 11 acquired by the first luminance information acquisition unit 211. The normal vector value (first normal vector value) at the position of the point on the surface of the measurement object 5 that is the subject projected on the position of each point on the first image plane is calculated based on the brightness of the first image plane. Thus, it is associated with the position information of each point on the first image plane. By doing so, the first normal vector calculation unit 221 includes the position information of each pixel on the first image plane and the normal vector at the point on the surface of the measurement target 5 that is the subject projected on the pixel. A correspondence table (first normal vector distribution table 31) with values is stored in the storage unit 3.
Similarly, the second normal vector calculation unit 222 calculates each polarizer associated with the position information of each pixel on the second image plane of the second polarization camera 12 acquired by the second luminance information acquisition unit 212. Based on the brightness of the transmitted light, a normal vector value (second normal vector value) at the position of a point on the surface of the measurement object 5 to be projected on the position of each pixel on the second image plane. Is calculated and associated with position information of each pixel on the second image plane. By doing so, the second normal vector calculator 222 calculates the position information of each pixel on the second image plane and the normal vector at the point on the surface of the measurement object 5 that is the subject projected on the pixel. A correspondence table (second normal vector distribution table 32) with values is stored in the storage unit 3.

  A method for calculating the normal vector value of the surface of the measurement object 5 serving as a subject based on luminance information observed through a plurality of polarizers is, for example, Non-Patent Document 1, Japanese Patent Application Laid-Open No. 2009-58533. These are disclosed in Patent Document 3 and the like, which are known techniques for those skilled in the art, and these may be applied in the present embodiment. The outline of the first normal vector calculation unit 221 will be briefly described below. Note that the second normal vector calculation unit 222 is the same as the first normal vector calculation unit 221, and a description thereof will be omitted.

<Calculation of luminance maximum value, minimum value, and polarization phase observed in each imaging unit 113>
The luminance I observed for each pixel on the first image plane through the polarizer can be expressed by Equation 1 according to the magnitude of the principal axis angle v (0 degrees ≦ v ≦ 180 degrees) of the polarizer. Are known. Here, I _max and I _min are the maximum value and minimum value of the observed luminance, respectively, and φ is a phase angle that is the magnitude of the principal axis angle when the maximum luminance I _max is observed.

_{I = (I max + I min} ) / 2 + ((I max -I min) / 2) cos (2v-2φ)
(Formula 1)
Then, based on the luminances I ₀ , I ₄₅ , I ₉₀ , I ₁₃₅ acquired when v = 0 °, 45 °, 90 °, and 135 °, their coordinates (I ₀ , 0 °), ( By approximating a curve passing through (I ₄₅ , 45 degrees), (I ₉₀ , 90 degrees), (I ₁₃₅ , 135 degrees) to a cosine function using, for example, the least squares method, I _max , I _min , and The phase angle φ can be obtained.

<Calculation of polarization degree observed at each pixel>
It is known that the degree of polarization ρ observed at each pixel on the first image plane is expressed by Equation 2.
ρ = (I _max −I _min ) / (I _max + I _min ) (Formula 2)
Therefore, by applying I _max and I _min calculated based on Expression 1 to Expression 2, the degree of polarization ρ observed for each pixel can be obtained.

<Normal of the surface of the measurement object 5 as the subject>
The following is known about normals at points on the surface of the measurement object 5 that is a subject projected on each pixel.
That is, when the measurement object 5 that is a subject is photographed by the first polarization camera 11 or the second polarization camera 12, diffuse reflected light that is light scattered by the pigment particles constituting the measurement object 5 is mainly photographed. In this case, it is known that the calculation is based on the azimuth angle representing the angle of the light exit surface and the zenith angle θ which is the radiation angle of the diffusely reflected light on the surface.
Here, the azimuth angle α representing the angle of the light exit surface is equal to the principal axis angle (that is, the phase angle φ) that gives the maximum luminance I _max in (Expression 1).
Further, it is known that the degree of polarization ρ _{d in} the case of diffuse reflection is expressed by (Equation 3) using the zenith angle θ and the refractive index n of the measurement object 5. The refractive index n varies depending on the medium of the measurement object 5, but for most objects, 1.3 ≦ n ≦ 1.8 is satisfied, and n is a value within this range. In this case, for example, when calculation is performed with n = 1.3, an error in the value of θ calculated by (Equation 3) can be ignored.

Therefore, by applying the degree of polarization ρ (= ρ _d ) and n = 1.3 calculated based on Equation 2 to Equation 3, the zenith angle θ can be obtained.

<Normal vector value>
Measurement that becomes a subject projected on each pixel on the first image plane by the azimuth angle α representing the angle of the light exit surface, which is equal to the phase angle calculated in Equation 1, and the zenith angle θ calculated in Equation 3. Normal vector values (p _x , p _y , p _z ) at points on the surface of the object 5 can be calculated by (Equation 4).

As described above, the first normal vector calculation unit 221 determines the position information of each pixel on the first image plane, the subject projected on the pixel, and the frame captured by the first polarization camera 11. A correspondence table (first normal vector distribution table 31) with normal vector values at points on the surface of the measurement object 5 is created and stored in the storage unit 3. Here, the normal vector values p _x , p _y , and p _z stored in the first normal vector distribution table 31 are components (X ₁₎ of the normal vector in the X ₁ axis direction in the first camera coordinate system. axis component), _{Y 1} axially according component _{(Y 1} axis component), and _{Z 1} axial direction according component _{(Z 1} -axis component).

Similar to the first normal vector calculation unit 221, the second normal vector calculation unit 222 performs position information of each pixel on the second image plane with respect to the frame captured by the second polarization camera 12, and A correspondence table (second normal vector distribution table 32) with normal vector values at points on the surface of the measurement object 5 to be projected on the pixels is created and stored in the storage unit 3. Here, each normal vector value p _x , p _y , p _z stored in the second normal vector distribution table 32 is a component (X ₂₎ of the normal vector in the X ₂ axis direction in the second camera coordinate system. axis component), _{Y 2} axially according component _{(Y 2} axis component), and a _{Z 2} axial direction according component _{(Z 2} axial components).

In the above description, when the measurement object 5 that is a subject is photographed by the first polarization camera 11 or the second polarization camera 12, the diffuse reflected light that is the light scattered by the pigment particles constituting the measurement object 5 is mainly used. Although the case where it was photographed was demonstrated, it is not restricted to this.
For example, when the measurement object 5 that is the subject is photographed by the first polarization camera 11 or the second polarization camera 12, light that is regularly reflected on the surface of the measurement object 5 that is the subject (specular reflection light) is mainly photographed. In this case, that is, when the light from the light source appears to be reflected on the glossy surface, the normal vector value calculated based on the specular reflection light may be used.
It is known that the degree of polarization ρ _{s in} the case of specular reflection is expressed by Expression 5 using the zenith angle θ and the refractive index n of the measurement object 5. The refractive index n varies depending on the medium of the measurement object 5, but for most objects, 1.3 ≦ n ≦ 1.8 is satisfied, and n is a value within this range. In this case, for example, when calculating with n = 1.3, as in the case of diffuse reflection, an error in the value of θ calculated by (Equation 5) can be ignored.

In the case of specular reflection, the azimuth angle α representing the angle of the light incident surface is equal to the principal axis angle (that is, an angle different from the phase angle φ by 90 degrees) that gives the minimum luminance I _min in (Expression 1). Then, the measurement object 5 that is a subject to be projected on each pixel is determined by the azimuth angle α (that is, an angle that is 90 degrees different from the phase angle φ) that represents the angle of the light incident surface and the zenith angle θ calculated by Equation 5. A normal vector value (p _x , p _y , p _z ) at a point on the surface of is represented by Equation 4.
Hereinafter, unless otherwise specified, it is assumed that the normal vector value corresponding to each pixel is calculated based on the degree of polarization of diffusely reflected light.

<Plane area detection unit 23>
<Continuous area>
The plane area detection unit 23 searches for a continuous area in which the first normal vector values on the first image plane have the same value, and a set of pixels included in the continuous area (hereinafter referred to as “continuous area set”). , And continuous area identification information indicating whether or not the pixel is included in the continuous area is added as attribute information regarding the pixels constituting the first normal vector distribution table 31.
Thereby, by referring to the first normal vector distribution table 31, it is possible to determine whether or not pixels on the first image plane (hereinafter also referred to as “points” for simplicity) are included in the continuous region. it can.

<Edge point>
The plane area detection unit 23 is a point A on the first image plane excluding continuous areas where the first normal vector values associated with the respective points on the first image plane have the same value, When there is a continuous region where the normal vector value is the same value in the vicinity of A, and the first normal vector value of the point A changes discontinuously and rapidly from the normal vector value in the vicinity, Point A is detected as an edge point. The plane area detection unit 23 adds edge point identification information indicating whether or not the point is an edge point as attribute information regarding the points constituting the first normal vector distribution table 31.
By doing so, it is possible to determine whether or not the point on the first image plane is an edge point by referring to the first normal vector distribution table 31.

<Corresponding point calculation unit 24>
The corresponding point calculation unit 24 includes a first normal vector value associated with the position information of each point on the first image plane and a second method associated with the position information of each point on the second image plane. The line vector values are matched to calculate corresponding points between the first image plane and the second image plane.
As described above, the first normal vector values (p _x , p _y , p _z ) stored in the first normal vector distribution table 31 created for the frame imaged by the first polarization camera 11. Are components in the first camera coordinate system of the first normal vector p ₁ in the X ₁ axis direction (X ₁ axis component), components in the Y ₁ axis direction (Y ₁ axis component), and Z ₁ axis direction according as the component _{(Z 1} -axis component). On the other hand, the second normal vector values (p _x , p _y , p _z ) stored in the second normal vector distribution table 32 created for the frame imaged by the second polarization camera 12 are The component of the two normal vectors p ₂ in the second camera coordinate system in the X ₂ axis direction (X ₂ axis component), the component in the Y ₂ axis direction (Y ₂ axis component), and the component in the Z ₂ axis direction ( is a Z ₂ axial components).
The corresponding point calculation unit 24 relates the values of the second normal vectors p ₂ stored in the second normal vector distribution table 32 in the X ₁ axis direction in the first camera coordinate system in order to match the two. The vector value p ₂ ′ expressed by the component (X _1- axis component), the component related to the Y _1- axis direction (Y _1- axis component), and the component related to the Z _1- axis direction (Z _1- axis component) is converted.
Specifically, the corresponding point calculation unit 24 uses the coordinate transformation matrix ¹ T ² that converts the coordinate value in the second camera coordinate system to the coordinate value in the first camera coordinate system, and uses the second normal vector distribution table 32. Each second normal vector value p ₂ stored in ( ₂ ) is converted into p ₂ ′ based on (Equation 7).
p ₂ ′ = ¹ T ² · p ₂ (Formula 7)
Hereinafter, the converted vector value is referred to as a second normal vector value (converted). The corresponding point calculation unit 24 may store the second normal vector value (converted) in the second normal vector distribution table 32.

For the point A included in the continuous region in which the first normal vector value associated with each point on the first image plane is the same value, the corresponding point calculation unit 24 performs only the normal vector value matching. The corresponding point B on the second image plane cannot be searched. For this reason, the corresponding point calculation unit 24 first selects the second image plane corresponding to the point A on the first image plane excluding the continuous region where the first normal vector values on the first image plane are the same value. The corresponding point B is searched.
Thereafter, the corresponding point calculation unit 24 uses, for example, the edge point information stored in the first normal vector distribution table 31 so that the first normal vector values on the first image plane are the same value. The corresponding points on the second image plane of the points included in are searched. A detailed description of the search for corresponding points on the second image plane of points included in the continuous area will be described later.

<Search for the corresponding point B on the second image plane corresponding to the point A on the first image plane excluding the continuous region>
The corresponding point calculation unit 24 determines that the first normal vector value associated with the point A on the first image plane excluding the continuous region where the first normal vector value on the first image plane is the same value is a point. The point B is located on the epipolar line on the second image plane corresponding to A, and the second normal vector value (converted) associated with the point B matches the normal vector value of the point A. Then, the point B (= corresponding point candidate of the point A) that exists discretely on the epipolar line is searched.
When there are a plurality of corresponding point candidates B, for example, the corresponding point calculation unit 24 matches the distribution of normal vector values in the vicinity of each corresponding point candidate B with the distribution of normal vector values in the vicinity of point A ( Corresponding point candidate B (most similar) is determined to be a corresponding point of point A. Here, the vicinity of the point A means a predetermined number of points surrounding the point A.
In this way, the corresponding point calculation unit 24 calculates the corresponding point B on the second image plane with respect to the point A on the first image plane excluding the continuous region in which the first normal vector values have the same value, and stores it. The attribute information of the point A (the coordinate value of the first image coordinate system, the first normal vector value, etc.) and the attribute information of the corresponding point B (the coordinate value of the second image coordinate system, the second normal vector value, etc.) ) Is stored in association with the frame identification information. Hereinafter, this is referred to as “corresponding point information table 33”.

<Three-dimensional coordinate value calculation unit 25>
The three-dimensional coordinate value calculation unit 25 applies points on the first image plane from the first polarization camera to all points A (and corresponding points B) stored in the corresponding point information table 33 based on triangulation. The distance to the point A ′ on the surface of the subject projected onto A and the three-dimensional coordinate values (X _1A , Y _1A , Z _1A ) in the first camera coordinate system are calculated. Similarly, the three-dimensional coordinate value calculation unit 25 calculates a three-dimensional coordinate value (X _2A , Y _2A , Z _2A ) in the second camera coordinate system.
The three-dimensional coordinate value calculation unit 25 uses the three-dimensional coordinate values (X _WA , Y _WA , Z _WA ) of the point A ′ in the reference coordinate system and the three-dimensional coordinate values (X _1A , Y _1A , Z _1A ) can be calculated by converting the conversion matrix ^W T ¹ .
The three-dimensional coordinate value calculation unit 25 uses the three-dimensional coordinate values (X _1A , Y _1A , Z _1A ) in the first camera coordinate system, the second camera coordinate system as attribute information related to each corresponding point in the corresponding point information table 33. 3D coordinate values (X _2A , Y _2A , Z _2A ) and 3D coordinate values (X _WA , Y _WA , Z _WA ) in the reference coordinate system can be stored.

<Calculation of three-dimensional coordinate value of point R on continuous region and search for corresponding point on second image plane>
Next, calculation of the three-dimensional coordinate value of the point R on the continuous area will be described.
The three-dimensional coordinate value calculation unit 25 uses the edge point identification information indicating whether or not the points constituting the first normal vector distribution table 31 are edge points, so that the first normal vector on the first image plane is used. It is possible to calculate a three-dimensional coordinate value in the first camera coordinate system of an arbitrary point R included in a continuous region having the same value.
Specifically, the three-dimensional coordinate value calculation unit 25 selects a straight line having the same normal vector value from arbitrary points R included in the continuous region on the first image plane, An edge point where the normal vector value changes discontinuously along both directions is specified. In specifying the edge point, for example, a point on the straight line that is stored as an edge point in the first normal vector distribution table 31 may be searched.
When the three-dimensional coordinate value calculation unit 25 specifies the edge point P, the three-dimensional coordinate value (X _1P , Y _1P , etc.) in the first camera coordinate system of the edge point P is referred to by referring to the corresponding point information table 33. Z _1P ) is acquired.
The three-dimensional coordinate value calculation unit 25 calculates a plane equation in the first camera coordinate system of a plane including the continuous region. Specifically, assuming that the first normal vector value having the same value is (e _X , e _Y , e _Z ), the plane equation including the continuous region is expressed by Expression 8 where d is a constant.
eX _X + e _Y Y + e _Z Z + d = 0 (Formula 8)
The three-dimensional coordinate value calculation unit 25 substitutes the three-dimensional coordinate values (X _1P , Y _1P , Z _1P ) of the edge point P and calculates a constant d, thereby calculating a plane equation including the continuous region. be able to.

The three-dimensional coordinate value calculation unit 25 can calculate the three-dimensional coordinate value in the first camera coordinate system of an arbitrary point R included in the continuous region, using the calculated plane equation.
Specifically, the three-dimensional coordinate value calculation unit 25 includes a straight line passing through an arbitrary point R included in the continuous area on the first image plane from the origin O ₁ of the first camera coordinate system, and the continuous area. 3-dimensional coordinate values of the intersection R'the plane equation _{(X 1R, Y 1R, Z} 1R) acquires the can calculate the distance to the intersection R'.
For example, the three-dimensional coordinate value calculation unit 25 converts the three-dimensional coordinate values (X _1R , Y _1R , Z _1R ) of the point R ′ in the first camera coordinate system into the coordinate values (X _2R , Y _2R ) in the second camera coordinate system. , Z _2R ) and projecting onto the second image plane, the coordinate value of the point R ′ in the second image coordinate system is acquired. In addition, the three-dimensional coordinate value calculation unit 25 uses the three-dimensional coordinate values (X _WR , Y _WR , Z _WR ) of the point R ′ in the reference coordinate system as the three-dimensional coordinate values of the point R ′ in the first camera coordinate system ( X _1A , Y _1A , Z _1A ) can be calculated by transforming with a transformation matrix ^W T ¹ .
In this way, the three-dimensional coordinate value calculation unit 25 projects all the points R included in the continuous area in the corresponding point information table 33 and the attribute information of the corresponding points and the subject projected on the points R on the first image plane. The three-dimensional coordinate values (X _1R , Y _1R , Z _1R ) in the first camera coordinate system of the point R ′ on the surface of X, the three-dimensional coordinate values (X _2R , Y _2R , Z _2R ) in the second camera coordinate system, And three-dimensional coordinate values (X _WR , Y _WR , Z _WR ) in the reference coordinate system can be stored. A set of these three-dimensional coordinate values is referred to as a “three-dimensional coordinate value table 34”.

In addition, when the two edge points P and Q are specified at both ends on the straight line on the first image plane, the three-dimensional coordinate value calculation unit 25 from the edge point P to the edge point Q on the first image plane. The three-dimensional coordinate value of an arbitrary point R on the straight line can also be calculated using (Equation 9).
Specifically, the distance from the point P to the point Q (for example, the number N _{P-> Q} of points on the straight line including the point P and the point _Q ), and the distance from the point P to the point R (for example, the point P And the number N _{P-> R} of points on the straight line including the point _R ) can be calculated by proportional calculation shown in (Equation 9).
_X1R = ( _{NP-> R} / _{NP-> Q} ). ( _{X1Q- X1P} )
_Y1R = ( _{NP-> R} / _{NP-> Q} ). ( _{Y1Q- Y1P} )
_Z1R = ( _{NP-> R} / _{NP-> Q} ). ( _{Z1Q- Z1P} )
(Formula 9)
As described above, the depth acquisition apparatus 100 acquires depth information (three-dimensional position information) of an object in a three-dimensional space from a polarization image without creating a three-dimensional model or depth map of the object in advance. be able to.

  As described above, the embodiments of the functional units of the depth acquisition apparatus 100 according to the embodiment of the present invention have been described based on the configurations of the stereo camera 1, the control unit 2, and the like.

<Diffusion reflection and specular reflection>
In the present embodiment, in the matching process using the first normal vector value and the second normal vector value, both are calculated based on the degree of polarization of diffusely reflected light. However, the present invention is not limited to this.
For example, a case where large reflected light that is specularly reflected on the surface of the measurement object 5 as a subject is input to the first polarization camera 11 or the second polarization camera 12 depending on the position of the light source, etc. even in the same place. The possibility of occurrence cannot be excluded.
Therefore, when matching the first normal vector value and the second normal vector value by the corresponding point calculation unit 24, the first method calculated based on the polarization degree of the diffusely reflected light is used as the first normal vector value. Two of the first normal vector value (specular reflection) calculated based on the line vector value (diffuse reflection) and the degree of polarization of the specular reflection light may be used.
Similarly, as the second normal vector value, a second normal vector value (diffuse reflection) calculated based on the degree of polarization of diffusely reflected light and a second normal vector value calculated based on the degree of polarization of specularly reflected light. You may use two of (specular reflection).
By doing so, when the two are matched, the second normal vector value (diffuse reflection) is not the same as the case where the second normal vector value (diffuse reflection) matches the first normal vector value (diffuse reflection). When the first normal vector value (specular reflection) matches, the second normal vector value (specular reflection) matches the first normal vector value (diffuse reflection), or the second normal vector value (specular reflection). The case where (reflection) matches the first normal vector value (specular reflection) may be included.

Next, a flow of a series of processes in the depth acquisition apparatus 100 according to the embodiment of the present invention will be described with reference to FIG. 6A and 6B are flowcharts showing a flow of a series of processes in the depth acquisition apparatus 100. FIG. In addition, before starting the following operation | movement, each initial setting, the 1st polarization camera 11, and the 2nd polarization camera 12 have been calibrated, the 1st camera coordinate system of the 1st polarization camera 11, and the 2nd polarization camera 12 The coordinate transformation matrices ² T ¹¹ T ² , ^W T ¹ , ^W T ^2, etc. between the second camera coordinate system and the reference coordinate system are calculated.

  Referring to FIG. 6A, in ST1, for the frame captured by the stereo camera 1, the first luminance information acquisition unit 211 and the second luminance information acquisition unit 212, respectively, each polarizer in the first polarization camera 11 and The brightness of light transmitted through each polarizer in the second polarization camera 12 is acquired in association with the position information of each point on the first image plane and the position information of each point on the second image plane.

  In ST2, the first normal vector calculation unit 221 and the second normal vector calculation unit 222 are associated with the position information of each point on the first image plane of the first polarization camera 11 acquired in step ST1. The first image plane based on the brightness of the light transmitted through each polarizer and the brightness of the light transmitted through each polarizer associated with the position information of each point on the second image plane of the second polarization camera 12. Correspondence table (first normal vector distribution table 31) between the position information of each upper pixel and the normal vector value at the point on the surface of the measurement object 5 to be projected on the pixel, and the second image A correspondence table (second normal vector distribution table 32) between the position information of each pixel on the plane and the normal vector value at the point on the surface of the measurement object 5 to be projected on the pixel is calculated. , And stores it in the 憶部 3.

  In ST3, the plane area detection unit 23 searches for a continuous area in which the first normal vector values on the first image plane have the same value, detects a set of points included in the continuous area, and detects edge points. Is detected.

  In ST4, the corresponding point calculation unit 24 determines that the first normal vector value associated with the point A on the first image plane excluding the continuous area detected in ST3 is on the second image plane corresponding to the point A. And the second normal vector value (converted) associated with the point B coincides with the normal vector value of the point A and exists discretely on the epipolar line. Point B (= point candidate for point A) is searched.

  In ST5, the corresponding point calculation unit 24 checks whether or not there are a plurality of corresponding point candidates for the point A. If there are a plurality of corresponding point candidates (Yes), the process proceeds to ST6. If there is one corresponding point candidate (No), the process moves to ST7.

  In ST 6, the corresponding point calculation unit 24 matches (most similar) the corresponding point whose distribution of normal vector values in the vicinity of each corresponding point candidate B matches the distribution of normal vector values in the vicinity of point A. One candidate B is selected.

  In ST7, the corresponding point calculation unit 24 records one corresponding point candidate B searched in ST5 or one corresponding point candidate B selected in ST6 in the corresponding point information table 33 as a corresponding point of point A.

  In ST8, it is determined whether or not the search for all points A on the first image plane excluding the continuous region has been completed. When the search for all points A on the first image plane excluding the continuous region is completed (in the case of Yes), the process proceeds to ST9. If the search for all points A on the first image plane excluding the continuous region has not been completed (in the case of No), the process proceeds to ST4.

Referring to FIG. 6B, in ST9, the three-dimensional coordinate value calculation unit 25 performs, based on triangulation, all the points A on the first image plane excluding the continuous area stored in the corresponding point information table 33. The distance from the first polarization camera to the point A ′ on the surface of the subject projected onto the point A on the first image plane and the three-dimensional coordinate values (X _1A , Y _1A , Z _1A in the first camera coordinate system) ), The distance from the second polarization camera to the point A ′ on the surface of the subject, the three-dimensional coordinate values (X _2A , Y _2A , Z _2A ) in the second camera coordinate system, and the reference coordinate system of the point A ′ Three-dimensional coordinate values (X _WA , Y _WA , Z _WA ) are calculated and stored in the storage unit 3 in association with frame identification information (for example, a time stamp).

  In ST10, for each continuous area, the plane area acquisition unit 24 calculates a plane equation in the first camera coordinate system of the plane including the continuous area based on the corresponding edge point.

  In ST11, based on the plane equation calculated in ST10, the three-dimensional coordinate value calculation unit 25 calculates the first camera coordinates of the point R ′ on the surface of the subject projected on the arbitrary point R included in the continuous region. The three-dimensional coordinate value in the system is calculated, and the distance from the first polarization camera to the point R ′ on the surface of the first subject is stored in the three-dimensional coordinate value table 34. Similarly, the distance from the second polarization camera to the point R ′ on the surface of the subject, the three-dimensional coordinate value in the second camera coordinate system, and the three-dimensional coordinate value in the reference coordinate system of the point A ′ are calculated. The information is stored in the storage unit 3 in association with frame identification information (for example, a time stamp).

  In ST12, it is determined whether or not the three-dimensional coordinate values of the points R ′ on the surface of the subject projected on all the points R included in the continuous area have been calculated. When the three-dimensional coordinate values of the points R ′ on the surface of the subject projected on all the points R included in the continuous area are calculated (Yes), the process proceeds to ST13. If the calculation of the three-dimensional coordinate values of the points R ′ on the surface of the subject projected on all the points R included in the continuous region has not been completed (in the case of No), the process proceeds to ST11.

  In ST13, it is determined whether or not the calculation of the three-dimensional coordinate values of the points R ′ on the surface of the subject projected on all the points R included in all the continuous regions has been completed. If completed (in the case of Yes), the process proceeds to ST14. If not completed (in the case of No), the process proceeds to ST10.

  In ST14, the created point cloud data is displayed on a display unit such as a smartphone or a tablet based on a three-dimensional display technique known to those skilled in the art.

In ST15, it is determined whether or not imaging of frames at a predetermined frame rate by the stereo camera 1 has been completed. When it is finished (in the case of Yes), the process is finished. If not completed (in the case of No), the process proceeds to ST1.
As described above, the composite point cloud data in each frame is created according to the frame rate, and a three-dimensional video viewed from an arbitrary viewpoint set by the user can be viewed in real time on a smartphone or a tablet, for example. it can.

  As mentioned above, although preferable embodiment of the depth acquisition apparatus 100 of this invention was described, this invention is not restrict | limited to the above-mentioned embodiment, It can change suitably.

<Modification 1>
In the present embodiment, in the matching process using the first normal vector value and the second normal vector value, both are calculated based on the degree of polarization of diffusely reflected light. However, the present invention is not limited to this.
For example, a case where large reflected light that is specularly reflected on the surface of the measurement object 5 as a subject is input to the first polarization camera 11 or the second polarization camera 12 depending on the position of the light source, etc. even in the same place. The possibility of occurrence cannot be excluded.
Therefore, when matching the first normal vector value and the second normal vector value by the corresponding point calculation unit 24, the first method calculated based on the polarization degree of the diffusely reflected light is used as the first normal vector value. Two of the first normal vector value (specular reflection) calculated based on the line vector value (diffuse reflection) and the degree of polarization of the specular reflection light may be used.
Similarly, as the second normal vector value, a second normal vector value (diffuse reflection) calculated based on the degree of polarization of diffusely reflected light and a second normal vector value calculated based on the degree of polarization of specularly reflected light. You may use two of (specular reflection).
By doing so, when the two are matched, the second normal vector value (diffuse reflection) is not the same as the case where the second normal vector value (diffuse reflection) matches the first normal vector value (diffuse reflection). When the first normal vector value (specular reflection) matches, the second normal vector value (specular reflection) matches the first normal vector value (diffuse reflection), or the second normal vector value (specular reflection). The case where (reflection) matches the first normal vector value (specular reflection) may be included.

<Modification 2>
In the above embodiment, the stereo camera 1 may include a third camera (two-dimensional image acquisition unit) separately from the first polarization camera 11 and the second polarization camera 12. By performing calibration with the first polarization camera 11 or the second polarization camera 12 in advance, for example, a point on the surface of the measurement object 5 as a subject is represented by a third camera coordinate system, By projecting onto the third image plane, point cloud data is created by associating the three-dimensional coordinate values in the reference coordinate system with the image data captured by the third camera (two-dimensional image acquisition unit). P.

<Modification 3>
In the depth acquisition apparatus 100 of the present embodiment, for example, a part of the functions of the control unit 2 (for example, the corresponding point calculation unit 24) is distributed to specific computers including virtual computers on the cloud. This is a design matter that can be appropriately performed for the user. Moreover, you may make it give the virtual computer on a cloud a part of function of a control part.

<Modification 4>
In the depth acquisition apparatus 100 of the present embodiment, the frame rate of the stereo camera 1 (the first polarization camera 11 and the second polarization camera 12) is exemplified as 30 fps, and the normal vector value of the subject 5 is calculated for each frame. Although the calculation of the corresponding point and the three-dimensional coordinate value are calculated, the present invention is not limited to this. The frame rate value can be appropriately set according to the high-speed frame rate that the first polarization camera 11 and the second polarization camera 12 can handle.

DESCRIPTION OF SYMBOLS 100 Depth acquisition apparatus 1 Stereo camera 11 1st polarization camera 111 Pattern polarizer 112 Image sensor 12 2nd polarization camera 121 Pattern polarizer 122 Image sensor 2 Control part 21 Luminance information acquisition part 211 1st brightness information acquisition part 212 2nd brightness | luminance Information acquisition unit 22 Normal vector calculation unit 221 First normal vector calculation unit 222 Second normal vector calculation unit 23 Planar region detection unit 24 Corresponding point calculation unit 25 Three-dimensional coordinate value calculation unit 3 Storage unit 31 First normal Vector distribution table 31
32 Second normal vector distribution table 33 Corresponding point information table 34 Three-dimensional coordinate value table 5 Measurement object

Claims (8)

A first polarization camera including a plurality of polarizers that respectively transmit at least three directions of polarization components, and a second polarization camera that includes a plurality of polarizers that respectively transmit at least three directions of polarization components, and image an object A stereo camera
First luminance information acquisition that acquires the luminance of light transmitted through each polarizer in the first polarization camera in association with position information of each point on the first image plane that is an image plane of the first polarization camera. And
Second luminance information acquisition for acquiring the luminance of light transmitted through each polarizer in the second polarization camera in association with the position information of each point on the second image plane, which is the image plane of the second polarization camera. And
The degree of polarization calculated from the luminance of light transmitted through each polarizer associated with the position information of each point on the first image plane of the first polarization camera acquired by the first luminance information acquisition unit. Based on the first normal vector value at the position of the point on the surface of the subject projected to the position of each point on the first image plane, the position of each point on the first image plane is calculated. A first normal vector calculation unit for associating with the information;
Based on the luminance of the light transmitted through each polarizer associated with the position information of each point on the second image plane of the second polarization camera acquired by the second luminance information acquisition unit, the second A second method of calculating a second normal vector value at the position of the point on the surface of the subject projected to the position of each point on the image plane and associating it with the position information of each point on the second image plane A line vector calculation unit;
A first normal vector value associated with the position information of each point on the first image plane, and a second associated with each point on the epipolar line on the second image plane corresponding to each point. A corresponding point calculation unit that calculates a corresponding point between the first image plane and the second image plane by matching a normal vector value;
Three-dimensional coordinate value calculation for calculating a three-dimensional coordinate value in a preset reference coordinate system of a point on the surface of the subject projected onto the corresponding point based on the corresponding point calculated by the corresponding point calculation unit And
A depth acquisition device comprising: The first normal vector calculation unit includes:
The degree of polarization of diffusely reflected light and / or the degree of specularly reflected light is calculated from the luminance of light transmitted through each polarizer associated with the position information of each point on the first image plane of the first polarization camera. Calculating a first normal vector value based on the degree of polarization of the diffusely reflected light and / or a first normal vector value based on the degree of polarization of the specularly reflected light,
The second normal vector calculation unit includes:
The polarization degree of diffuse reflection light and / or the polarization degree of specular reflection light is calculated from the luminance of light transmitted through each polarizer associated with the position information of each point on the second image plane of the second polarization camera. The depth acquisition apparatus according to claim 1, wherein a second normal vector value based on a polarization degree of the diffuse reflected light and / or a second normal vector value based on a polarization degree of the specular reflected light is calculated. The corresponding point calculation unit
Distribution of first normal vector values associated with each point on the first image plane and its vicinity, and each point on the epipolar line on the second image plane corresponding to each point and its vicinity 3. The depth according to claim 1, wherein a matching point between the first image plane and the second image plane is calculated by matching the distribution state of the associated second normal vector value. Acquisition device. Detecting a continuous region including continuous points on the first image plane, wherein the first normal vector values associated with the respective points on the first image plane are the same, and
A point on the first image plane excluding the continuous region, and there is a continuous region in which the normal vector value is the same value in the vicinity of the point, and the first normal vector value at the point is the continuous region A plane region detection unit that detects edge points that change discontinuously and rapidly from the normal vector value in
The three-dimensional coordinate value calculation unit
Based on the edge point, a three-dimensional coordinate value in the reference coordinate system of each point on the surface area of the subject corresponding to the continuous area where the first normal vector values on the first image plane are the same. The depth acquisition apparatus according to any one of claims 1 to 3, wherein the depth acquisition apparatus calculates the depth acquisition apparatus. The corresponding point calculation unit
5. A point on the continuous area of the first image plane and a corresponding point on the second image plane are calculated based on a three-dimensional coordinate value of each point on the continuous area in the reference coordinate system. Depth acquisition device as described. A third camera (two-dimensional image acquisition unit) separately from the first polarization camera and the second polarization camera;
A point cloud data generation unit that generates point cloud data by associating image data captured by the third camera with a three-dimensional coordinate value in the reference coordinate system;
The depth acquisition device according to claim 1, comprising: Stereo camera that captures an object including a first polarization camera including a plurality of polarizers that respectively transmit at least three polarization components and a second polarization camera that includes a plurality of polarizers that respectively transmit at least three polarization components. And a computer that is communicably connected,
First luminance information acquisition that acquires the luminance of light transmitted through each polarizer in the first polarization camera in association with position information of each point on the first image plane that is an image plane of the first polarization camera. Steps,
Second luminance information acquisition for acquiring the luminance of light transmitted through each polarizer in the second polarization camera in association with the position information of each point on the second image plane, which is the image plane of the second polarization camera. Steps,
The degree of polarization calculated from the luminance of light transmitted through each polarizer associated with the position information of each point on the first image plane of the first polarization camera acquired in the first luminance information acquisition step. Based on the first normal vector value at the position of the point on the surface of the subject projected to the position of each point on the first image plane, the position of each point on the first image plane is calculated. A first normal vector calculating step corresponding to the information;
Based on the luminance of the light transmitted through each polarizer associated with the position information of each point on the second image plane of the second polarization camera acquired in the second luminance information acquisition step, the second A second method of calculating a second normal vector value at the position of the point on the surface of the subject projected to the position of each point on the image plane and associating it with the position information of each point on the second image plane A line vector calculating step;
A first normal vector value associated with the position information of each point on the first image plane, and a second associated with each point on the epipolar line on the second image plane corresponding to each point. A corresponding point calculation step of calculating a corresponding point between the first image plane and the second image plane by matching a normal vector value;
Three-dimensional coordinate value calculation for calculating a three-dimensional coordinate value in a preset reference coordinate system of a point on the surface of the subject projected onto the corresponding point based on the corresponding point calculated in the corresponding point calculating step A depth information acquisition method comprising:   The computer program for making a computer perform each step of Claim 7.